AI Assisted Grading: A Teacher’s 30-Second Checklist

Vicki Davis: Happy Tech Tip Tuesday. Today we’ll be talking about AI assessment. For me, this summertime is the time that I spend training the AI so that it can help me assess. Always have you in the loop. You should be the one giving grades, as well as disclose to your students how you’re using the artificial intelligence and how you are training it. This helps you have both credibility with your students and helps you save time in your modeling how you want artificial intelligence to be used. I hope you enjoyed today’s Tech Tip Tuesday here on the 10 Minute Teacher Podcast.

Announcer: This is the 10 Minute Teacher Podcast with your host, Vicki Davis.

Vicki Davis: Today we’re talking with Steven Swanson. He is a high school engineering teacher at Whittier High School and also an AP Computer Science and AP Physics teacher. He built a tool called ClassLens, an AI grading tool he made because he needed it for his own gradebook. He is unique in the fact that he deleted a feature to automatically return AI-generated grades to students. And that requires that teachers review everything that AI has suggested for feedback. And we’re going to get into that.

Vicki Davis: Steve, ClassLens started because you needed it for your own gradebook. I want you to take me back to the moment in your teaching where you said, “I can’t keep doing this. There’s a better way and I can build it.”

Steven Swanson: The very first day that hit me was — I take a lot of days off because I run an academy. I’m off maybe fifteen days a year for field trips. And I missed one or two days in a row, because sometimes we do our field trips back to back. And I remember I gave one extra assignment than usual to my sophomores. And I did that because they can be a little squirrely and I thought this better keep them a little bit busier. So I gave them that extra one.

Steven Swanson: And when I came back — I said, “You know what, let me grade this.” The night before I went back, actually. And I started grading, and I saw there was like 150 individual things to look at. You might know this too: a lot of teachers, we try to become efficient. We’ll sometimes give credit or no credit. If they turn this in and they did the work — sometimes homework is like that, I just check whether they do it or not. Then they have the high-stakes assignments where you’re checking every little thing.

Steven Swanson: But I don’t feel like that was valuable. They were drawing CAD — computer-aided drafting, for those who are wondering what CAD is. I teach engineering in this class. And I looked at them and I wanted them to be not only accountable, but mainly I wanted them to know what they did wrong if they were doing something wrong. They spent two hours doing it. I wanted them to know.

Steven Swanson: I started it and I said, this is going to take me all night if I want to actually give feedback too. Think about that — writing each individual comment. I have a hundred and fifty-something students. So I was like, this is going to take a long time. So I thought, let me try to create something where maybe I could just take a glance at it and give them a comment on whether they did it right or not. So that’s where it started.

Vicki Davis: And I’ve written lots of things to help me, because when you have something — I mean, I always review it — but when you have something and you have a lot of little boxes to check, if it can at least check the boxes for you, then that tells you where to guide. And it just speeds it up.

Steven Swanson: Yeah. Through all my work I did on it, I did find most recently that making them rubrics — so if you build a very good rubric, and you might know this too, and some of your listeners: AI, the more context you give, the better output they can give you. So if you write a really good rubric, they’ll grade it pretty much spot on to what you’re asking.

Vicki Davis: Recently I had Christian Miller, author of The Honesty Crisis, on my show, and he says — this is a great quote — that when students get feedback from the teacher and aren’t told that AI did it, and it looks like it’s coming from the teacher, he said, quote, “It’s not right. They’re actually being dishonest towards their students, and that’s a violation of trust.” What’s your opinion on what he said?

Steven Swanson: I 100% agree. In fact, I talked to an administrator about it. We were chatting as I was building it. He said, “I can see this as a really good tool. And I see it as a great tool for low-stakes, high-volume assignments, where you just want to give a little bit more back to the kid, but you just don’t have the time.” He suggested — I think this is a good idea — that teachers who use this should put something in their syllabus that says, “Hey, I do use an AI grading tool for assignments, and sometimes the feedback may be given back for assignments.”

Steven Swanson: So definitely, that’s a great idea. And I think that’s always the case. Even the grade itself — even if it’s assisted, even though I got rid of the auto-return feature, it passes in front of me and I’m looking at each grade and I’m looking at each comment. I still think there should be something. I don’t know if it’s per assignment. I don’t know if you need to have every single one say this was written by an AI tool. I don’t know about that — that could be argued either way. But definitely it should be stated somewhere, whether it’s on your website, syllabus, or assignment paper.

Vicki Davis: So you just said something that we’re going to dig into. You turned off the auto-grade — I hate to use the word “grade,” because for me, I feel like as the teacher, every grade passes through me. Now, feedback, I can have it help me write, and then I need to adjust it. But I should look at everything, which is basically what you’ve done. But most AI-assisted grading tools allow a teacher just to hit a button and just send it back. So why were you willing to — take me through the mental process of “okay, I’m going to turn this off, and I know everybody else does it, but this is why.”

Steven Swanson: First off, for some of the listeners, it’s called “human in the loop” now. That’s what it’s often called — does this pass through human inspection or not? I don’t want to say it was a mental thought. It was more of a gut feeling. In the process of building this, you start building feature upon feature upon feature. And when you’re in product development — and this is for any product — what you want to do is just make the easiest environment for the end user, a teacher or whoever it is. And then you think, how can I make their life the easiest, where everything is efficient and everything’s done for you, because automation is key to a lot of things.

Steven Swanson: As you’re pushing the envelope of development, I could get to a point where I could say, “Hey, I could just push a button.” It sucks all the assignments out of whatever you’re using — say that’s Google Classroom — and it pulls the assignments out, grades them all, and then sends them right back to the student. That’s sort of like what you would call a black box. You don’t see it happening. Upon creating it, it almost felt immediately wrong. “Hey, this doesn’t feel right.” And that’s not only the thinking process, it’s 22 years of teaching as well. It just doesn’t feel right.

Vicki Davis: We want help. We want to save time. But we also — I mean, we have a relationship with students. That’s a relationship of trust. Even when I’m not there, when my sub is there and I’m not, I want to know what went on, you know? I don’t like to just hand something over with no supervision. Do you think that’s why maybe it felt icky? Or have you talked to other teachers and how they feel about it?

Steven Swanson: I did talk to another teacher about it. But I think if you think of like a multiple choice — I can push a button, it grades them all, but you have faith that when I say the correct answer is A, then a student chooses A, it will get it right. And if they chose B, they will get it wrong. And you have faith in that, correct?

Steven Swanson: If you had that same faith with an AI — and nobody does right now — that it’s going to do everything right every single time… that faith doesn’t exist right now. It’s building that way. Could they get to that point where you just go, “Wow, everything it does is correct”? Maybe 10 years from now, I don’t know. But it’s not here today.

Vicki Davis: So I want to say this. First of all, ClassLens is not sponsoring this. The reason I accepted your pitch to me — and I have to say that, or else everybody in the world will just send me emails — but you’re a teacher, and in your email, you said something that I really want to know the answer to. You said that there is one question every teacher needs to ask before they let any AI tool grade student work. What is that question?

Steven Swanson: When you’re doing it, you want to know what’s happening to it. Like, where does it go? Where is it stored?

Steven Swanson: I know when I built it — just because I know about FERPA and privacy with student data — I built that from the ground up. That’s where I started. And then I realized, okay, where is this going to go? And then I realized, no, it’s not going to go anywhere. It’s not going to even be stored on my server. It goes to a trusted LLM company. It’s a Google one, made by Google — and because Google Classroom created it, I chose that one.

Steven Swanson: And they also are one of the only companies where they will approve a zero, where they hold none of your data, and it immediately deletes it. And I delete it from my server immediately after grading, and the teacher approves it and it goes back in. And on my end, I don’t need to sell it. I don’t need to collect data. That’s not my forte. And I definitely don’t have the money to pay for that.

Steven Swanson: So no, definitely that was something you should always look at. And for the most part, a lot of districts, before they even approve the use of this — and some districts do require that and others do not, depends on your state — a lot of them will actually look at that for you. They’re going to look and say, okay, they want to know every spot of where this is stored, what is being stored. Is it their name? What is it?

Steven Swanson: The only thing I do store is an obscure number that Google assigns to each student. And I only keep that because I’m planning the next feature, which is if a student wanted to opt out, you would be able to check that box for the student and say, okay, when the grades pass through, it doesn’t do anything for that student. I can foresee that happening in the future where districts might allow opting out, or even a state might say it’s required.

Vicki Davis: If you could change one thing about how the ed tech industry sells AI to schools right now, what would it be?

Steven Swanson: I haven’t thought too much about that, but I do know that it’s being flooded. They’re adding AI tools to everything. I would say that anything — and if this isn’t already true for the company — I think they have to have human in the loop for teacher stuff. I mean, if it involves students and their grades and the grades they’re getting, I think human in the loop has to be every part of it.

Steven Swanson: And that’s why I pretty much reached out to you: you need to have a teacher look at this stuff. You can suggest whatever you want. Here’s a comment it should be. Here’s a grade that I think it should be. But all of those should be editable by the teacher before they click “Okay, I agree to all this,” and then send it to my students. That’s kind of how ClassLens does it. It just lists all the grades, the comments, and you can click on them and see the work and double-check and say — like, every once in a while I’ll see something that’s a little low grade, and let me see, why is that getting such a low grade? Let me look at that assignment and see if I agree with that grade.

Steven Swanson: You definitely want that. Can you have somebody suggest grades? Sure. People have forever used — what? — teaching assistants. Colleges use that, where they have somebody grade a lot of their professors’ work and stuff like that. But because we’re the final say in their grade, I think every single assignment, every comment, should pass through our eyes and be editable so we can change it.

Vicki Davis: Steve, that’s such a great analogy, because I was a TA in college and I graded everything and then wrote my notes and gave it to the professor, but he had the ultimate say. Obviously I wasn’t the professor. So what a great analogy — because we have used people to help us. It’s just that most of us classroom teachers in K-12, most of us don’t qualify for any help. And so we’re so desperate and overworked that we want that help.

Vicki Davis: Let’s shift to this. Pretend that a teacher is listening to this in their car and they’re saying, “Okay, I think I’m going to try to use an AI tool to grade work tomorrow.” Could you give them a 30-second checklist? What should they be looking for as they prepare to do that?

Steven Swanson: One, obviously make sure the Google auth — we talked about that. Make sure that when you sign into Google, or whatever your sign-in is, that there’s no warnings or anything like that. Make sure there is a privacy agreement that they’re not doing anything with the data.

Steven Swanson: And I think the best advice I can give them is: the first time you use it — and hopefully nobody’s using it in a bind where they need their grades turned in 30 minutes — you should be able to provide yourself with enough time when you’re using this for the first time. Grade it. Read every single comment. Read every single one. Even if you go, “Hey, this is just as long as grading” — yes, but you want to make sure it grades correctly. So look at every grade, look at all their work, look at all the comments and go, “How off is this from what I would have given?” Do that. Don’t just trust it right off the bat. And do that for as many assignments as it takes before you start to feel comfortable with it.

Steven Swanson: Like, if you passed this work on to a human, like we just talked about — would you trust them right off the bat? I know I wouldn’t. Even if the district said, “Hey, I’ve got somebody who’ll grade your stuff for you, they sit right next to you” — I know that’s never going to happen — but they’ll grade your stuff. Yeah, I would still want to glance through it and look through it and kind of try to grasp if they were doing it correctly or the way I would have been assigning them, right? So I think they should do the same thing.

Vicki Davis: So Steve, you’ve been really emphatic: human in the loop. Is there anything else about the movement to try to give us AI assistance for assessment that makes you angry or really concerns you?

Steven Swanson: I wouldn’t say angry. This is for science in general — I think they’re building very quickly. They’re almost trying to fly a plane while they build it at the same time. I’m talking about the AI companies themselves. And it’s like an arms race.

Steven Swanson: I take that quote from — what is it — the dinosaur movie, where he says, “Your scientists are too busy thinking whether they can do it rather than if they should.”

Vicki Davis: Jurassic Park.

Steven Swanson: Jurassic Park. There you go. Jurassic Park.

Vicki Davis: I love that one, too.

Steven Swanson: Yeah. And it sometimes feels like that. It’s like, “Hey, we have stuff. Let’s try to figure out — one, just how can we use them right now? Where are we going with this? What is the plan next?” A lot of people don’t know that. And I know it’s uneasy for everybody, and it’s uneasy for me. And I think most of your listeners — and probably you would agree — if I could snap my fingers and just go back before AI and say there was no AI, that life was easier. It’s only going to get more complex as these AI models get more powerful and sharper and more capable.

Steven Swanson: Now you just take an LLM — a large language model, that’s what they’re called — and you give it tools that can do stuff, and then you call them agents, and then, you know, what’s next? So I guess that’s the stuff where — it’s uncertainty, and everybody is uncomfortable with uncertainty. So I wouldn’t say I’m angry about it, but uneasy about it, because it is uncertainty. And we don’t know where this is all heading. I know where it is now, but I think the thing that keeps me up at night is: where does this all go? That’s what’s uneasy for most people.

Vicki Davis: It is. And it’s something we talk about a lot on all my shows, because I believe that nobody’s smart enough to be king of the hill. We’ve got to make a bigger hill. And particularly, we’ve got to make room for classroom teacher voices. And that’s what you’re giving us. I think it’s great to have classroom teachers develop apps, just like classroom teachers having shows, right?

Vicki Davis: So we’ve been talking to Steven Swanson. He’s a high school engineering teacher at Whittier High School and an AP Computer Science and AP Physics teacher. And he created ClassLens. And I can’t finish the show without saying — tell us what movie was filmed at Whittier High School, for our listeners, as we finish.

Steven Swanson: Yeah, so if you’ve got any Back to the Future fans — the high school I work at is actually the home to Hill Valley High, which was the high school in Back to the Future.

Vicki Davis: So you’ve got some Back to the Future going with your AI-assisted grading. So good luck, Steve, and thanks for reaching out. Thanks for coming on the show and sharing your perspective.

Steven Swanson: Yeah, and thanks for having me. I really appreciate it.

Announcer: Thanks for tuning in to the 10 Minute Teacher Podcast. Watch the video version of this podcast and catch my new radio and TV show, Cool Cat Teacher Talk, on YouTube and a radio or TV station near you. Join my Cool Cat Teacher Classroom Matters newsletter at coolcatteacher.com/newsletter. Leave a review if you found this helpful. See you later, educator.