Frustrated With AI? Make It Interview You First

Vicki Davis (00:00): Happy Tech Tool Tuesday. This is episode 997. Today, we’re talking about how to overcome the frustrations of working with AI.

Announcer (00:09): This is the 10 Minute Teacher Podcast with your host, Vicki Davis.

Vicki Davis (00:14): Today, we’re talking with Dr. Jie Tao, an associate professor of business analytics at Fairfield University and founding director of the AI and Tech Institute. Jie, you went from teaching business analytics to founding a whole AI institute. What did you see happening that made you think, I need to move in this direction?

Jie Tao (00:35): It’s actually pretty simple and personal for both reasons. The first reason is around COVID time, seven, eight years ago, I made a pledge to myself. I said, you know, I’m doing all the academic work, teaching research and all that. And what’s the point if my own mother wouldn’t read my research article? I made a pledge to myself, I will only work on impactful research that people actually care from this point on. So to give you an example, the second most recent publication I had is actually to detect suicide ideation from social media. So clearly you can see people care about that. We are actually working with 18 clinicians in the U.S. and China, and they’re actually using our research prototype. The research prototype basically raised their outreach rate from 5 per clinician per week to 25. And raise their detection accuracy from low 70s to high 80s, almost 90%. So that’s one of the things I really want to make an impact. The other reason is also pretty personal. I built a tool called Health Buddy. So what it does is basically you subscribe to a keyword and every Monday morning you get an email telling you what’s going on regarding that keyword. Basically, you know, the latest drug trials and experimental treatment, all that stuff. And the reason behind that is also very personal. My dad was diagnosed with bone marrow cancer years ago. And I wanted to do research for him. And even with a doctorate, I’m spending hours every day to read those medical papers. So I need a tool. And I’m building this tool. This is free to public. We are beta testing this with a group of people, subscribing to their, you know, whatever condition they care about. They’re getting those summary emails every week.

Vicki Davis (02:29): Since you brought up the suicidal ideation issues, because as you know, this is just a huge issue, you’ve created these red rules. Okay, what are they? And help us understand, because I think these are really important.

Jie Tao (02:46): Yes, they are very important. The project started in 2010 or 2011, way before I joined the project. Basically, the motivation is simple. So on one social media platform, there is actually one user committed suicide. After he committed suicide, a lot of people would actually go to his page and pay tribute to this particular user. And this group of clinicians find out actually these people paying tribute, a lot of them are actually at risk. That’s why they’re paying tribute to this particular user. So they tried all kinds of different ways. They tried manually reading this. They’re trying to basically curate a dictionary saying, if you mention these words, you have suicidal thoughts. As humans, we can see the problem is that word match is very problematic. For example, what if the word is actually behind a negation, cannot capture that? When I joined the team in 2019, we already have this thing called language model. It’s not the ChatGPT you’re thinking about. It’s actually GPT-2. It is. OpenAI did develop that. But that doesn’t have a chat interface. You have to access it via computer code. First, we’re trying to introduce this human knowledge, this dictionary into GPT. And it didn’t work well. And then finally, the last research decision in that project that we did is we did a very expensive brain surgery. In that little brain of a language model, if you may, it’s basically before it talks back to us, we intercept its brainwaves, and we marry or enforce human knowledge into these brainwaves, fuse them together, and let that drive the decision. That’s how we can raise detection accuracy from even humans in low 70s to an AI in high 80s.

Vicki Davis (04:43): So what are the red rules? Are these like guidelines or this is just an algorithm to detect?

Jie Tao (04:50): So basically, it is an algorithm. It is basically architecture that imposes on the language model as a brain. We force it basically to reason like a clinician. Of course, it’s still very rudimentary, for lack of a better term, compared to a human brain. But for detecting linguistic cues about suicide ideation, it did reach an acceptable or satisfactory level.

Vicki Davis (05:18): So you also have a workshop called Stop Arguing with the Robot. So what is broken in how people are currently using AI?

Jie Tao (05:28): There’s one term I always mention in my own podcast. I call it prompt and pray. So prompt is basically how you communicate with these chatbots, right? You submit a query and it gives you responses. So the query you submit is a prompt. And if the prompt is not carefully designed, you pray. ChatGPT or whatever AI you’re using will give you a decent enough response. To me, that’s very frustrating. And to people I work with, that’s very frustrating because you don’t really know will you get a brilliant answer or something you can never use. So consistency is actually something we’re looking at. And particularly, if we go beyond personal use, if we actually want to use this for, say, workflows or business processes, we can actually live with slightly lower quality as long as the output is consistent. It’s consistently at 70% quality. That’s good. And it’s consistently follow a certain structure. That’s good. What we do is we basically have a prompting framework. It’s regulating the AI to think in a certain way along certain routes, and it will always deliver consistent and robust responses. They may not be the most brilliant responses, but they are reliable.

Vicki Davis (06:54): One thing I teach my students when they really have not a lot of knowledge on a topic is to have the AI tool go back and forth with an interview. Ask me questions. Is that part of something that we should be doing with students?

Jie Tao (07:07): Yes, the interview pattern, based on my experience and the people I trained, their experience is actually the most powerful pattern I would do. Actually, with every session, I would say, I, first, read the information I provided, either through uploading a file or in my prompt. Basically, that’s called priming its context. So the context is memory, right? And then I would say, do not assume anything. If you identify any cognitive gap, ask me a question about it. Whenever I start an interview, I would ask the language model, the chatbot, to start a thing called a scratchpad. So the scratchpad is a structure that basically records every bit of information I provided. And I will directly say, only use the information I provided in a scratchpad. Do not use any from your memory or doing web search. That’s how I basically want the chatbot to behave my way.

Vicki Davis (08:06): And you’ve gotten into some agentic AI research. How would you simply explain agentic AI, where this is heading for us as users of AI?

Jie Tao (08:17): First, there is no universal definition of agentic AI right now. Nobody has that. So the common understanding is LLM or large language model is basically just the brain. The brain cannot directly interact with the real world. It needs tools. Back in the day, you know, before they built in these web search tools into Claude and ChatGPT, it cannot even search the web before you give it that tool. Once the brain decides how to use the tool, for example, you can give it five tools at the same time at any given task, at any given stage of a task, it can decide which tool to use, which tool is the proper way to use, and which tool to use next. That’s what we call basically an agentic process. That’s different from, I don’t know how many people follow this, I think you should, is what we call fully autonomous AI. Something like OpenClaw. OpenClaw is basically you give full autonomy to these agents. You basically give a very brief description of the task. That’s slightly different from what I do. I define what I do as agentic workflow, meaning I still define what are the steps. And within each step, I give limited autonomy to the agent to say, okay, here are the tools. Here are the rules. You figure out how to carry out this step.

Vicki Davis (09:36): A lot of people are hooking up these tiny computers just to be like an agent to do things and are finding it extremely useful. So it is something worth watching and educating yourself about, for sure, because one thing I like about Claude Cowork is I have it help me triage my email. It might draft email, but I’m the one who has to put it in there and type it very granular about the control I would give. But I thought, hey, OpenClaw, that approach may be in our future, right, Jie?

Jie Tao (10:06): We are clearly heading to that truly autonomous, agentic environment. Nobody can make predictions these days. Things are moving so fast right now. I would say we’re not there yet. I’m not ready to give out the levers and the controls to a fully autonomous agent. It’s because two things, without being too technical. One is what we just said, they have a memory of a goldfish. If you let them run too long, they forget about why they’re there. So you can see a lot of these OpenClaw agents got trapped in a circle because they don’t remember why they started that task. And the second is the current architecture behind all these large language models. There is a limitation. There is a glass ceiling. We don’t know when we’re going to hit that glass ceiling, but sooner or later, we will. And that’s when I started researching these things, and I know what the structure is, and I think, you know, we will hit that glass ceiling. We have several promising alternative frameworks on the horizon. So once one of them becomes mainstream, maybe we will see fully autonomous agents.

Vicki Davis (11:11): So if a teacher’s listening right now and they’re talking to their students about artificial intelligence. I noticed you talk about AI as a brain and I actually am really cautious about not anthropomorphizing AI because of how children, how trusting they are if I anthropomorphize it. Are there some tips you have for teachers based on your understanding of, okay, this is how it works and then what we’re seeing with the negative outcomes of interacting with AI?

Jie Tao (11:39): The suggestion is that simple. One is always keep it in a controlled environment, meaning we should have human regulating how it behaves. One thing I recommend, for example, for businesses and K-12 teachers, and my recommendation, if you are basically tuning your tool, whatever you’re building, don’t make it directly student-facing or client-facing the first one. Test it thoroughly. Try to break it. I always tell, you know, my students, try to break it yourself. If you cannot break it, then there’s less chance other people will break it. Because think about this. We just talk about suicide ideation, and that kind of stuff. The chances are if it actually breaks, it could be life and death. My first recommendation is keep it in a controlled environment. The second recommendation I say is not really fully understand the mechanism, under the hood, but fully understand what it can and cannot do. That basic literacy is important. I think a lot of people are thinking either AI is God or garbage. It’s neither. It cannot do everything for you. It’s not God. It’s not garbage. Some of the information are very valuable. I’ll give you an example. Until four months ago, I was proud of myself writing every line of my computer code by hand. Now, I barely write a single line of code. That’s because I fully understand what it can and cannot do. I know when I have to step in and steer it to a different direction. So do not give up the control once you know clearly in your particular context what it can and cannot do. That’s my second piece of advice.

Vicki Davis (13:25): What do you think the biggest mistake teachers make when they first try using AI in their classroom?

Jie Tao (13:31): It’s not Google. It’s not a textbook. Not everything it says is true. It’s basically, the term I always use is overconfident intern. Think about it. Every piece of AI we face right now is basically trained on the whole internet. So whatever you can find, it can find. But it’s still, due to the memory issue and the framework issue, you still have to carefully guide it. My suggestion is be very cautious about your expectation of AI. It can do certain things, but there are certain things it cannot do. It can maybe can do things better or maybe can do things it wasn’t able to do with your guidance. So provide the proper guidance.

Vicki Davis (14:18): We’re talking with Dr. Jie Tao, associate professor of business analytics at Fairfield University and founding director of the AI and Tech Institute. I appreciate you coming on the show. This has been very interesting and enlightening.

Jie Tao (14:33): I wasn’t expecting this, but this is exciting.

Announcer (14:36): Thank you for tuning in to the 10 Minute Teacher Podcast. Join us here every weekday and subscribe to the Classroom Matters newsletter. See you later, educator.