AI Fact Checking Using the Ensemble Method

Generative answers are everywhere. People like them because they are simple and they just “give the answer.” But what if the answer is nuanced. And what if, in the case of the generative tool, it is so eager to give an answer that it makes mistakes that any researcher would understand. Let’s go through an example.

So, first, Generative Adversarial Networks were used from roughly 2017 to 2022 to make deepfake videos. So, one model generates the video, another model evaluates it, and they go back and forth until the video quality becomes so good that it could become undetectable from the real thing. At first I called what I was doing a GAN, but it turns out a GAN is a training architecture, not what I’m doing. The right word is ensembling. Some may call it a “multi-agent debate” or “cross-model verification.” How did I find this out? I am training my AI tools to teach me more about AI and check my word choices and teach me about the models that underlie what I’m doing. It is a helpful method for learning in a domain of knowledge, to be up front with the tools you use about the pursuit of knowledge in that domain. It creates a mini spotlight that the AI tool will use to funnel me more of what I want to learn about.

As I dug into ensemble methods, I found the ROVER Method is one that I’m really using for these transcripts that are more accurate. So, I use Riverside which generates a transcript. Then, I put a video into Adobe Premiere Pro which uses another model to generate a transcript. Finally, I use Auphonic which then uses Whisper to generate a transcript. Then, Claude Cowork has a skill that I built which takes the bio of the guest and the topic to ensure that all acronyms are appropriate for that domain of knowledge and defined in the transcript and I also have fact checking to double check anything said built in. And the transcripts are compared until a final transcript emerges. I’ve found that each model is better at different things. Adobe for timings and finding subtle noises and words. Riverside for getting the names accurate. And Whisper for double-checking hard-to-hear phrases and such. And between them as I run them “against” one another, and have Claude ask me about discrepancies, I can get transcripts that are more accurate, less expensive, and less time consuming than what I’ve ever done before. It is a technique. By running the models against each other, I’m getting better results.

What About Fact Checking

We know about AI “hallucination” or as many prefer to say “confabulation” where AI just makes stuff up. But now, AI can cite things so it is supposed to be better, right?

Well, I went through an example that I’ll be using with students because it really shows the nuances of AI and research studies. Simpler is not always better when it means we think we understand and state error as fact.

Now, do not stop at Step 2, or even at Step 3. I need you to follow this chain of reasoning here so we can answer the question: how accurate are “AI Overviews” and is the question even the right question to ask?

Step 1: Google Search

So, first, I’m working to find hallucination rates for AI currently, so I did a simple Google search.

This is a number I update quite frequently, but I was curious as to the accuracy of these numbers in the generative search box. The first thing that bothers me is that whenever I see numbers presented without citations, an alarm bell goes off. See the words “high rates” and “legal research” and “low rates” – perhaps the citation on the second bullet is there, but sometimes it isn’t.

STEP 2: Claude Fact Checking Skill

So, I took a screenshot of the Google search and went to Claude. Now, granted, if I had pasted in the research links, more accurate information would have happened arguably at this step. But I want to demonstrate how we’re fact checking at a conference or event, that we might take a screenshot, so for now, this test is using screenshots.

So, I went into Claude and pasted the screenshot and asked it to “fact-check these numbers from a Google search.” Then, after it came back with errors, I asked it to update the Google graphic with information on what it found. On the right are Claude’s verdicts, and on the left is the original search. But wait, we need to get the ensemble activated here. We’re not done yet. Gemini may not be so bad, and Claude might not be so good. (Again, I didn’t give links, or item 2 would have been a different answer.) I use ChatGPT Pro and Perplexity Pro as part of this process.

Note: I have programmed my AI tools to help me teach. Everything I create is in the context of teaching someone, even myself, so you can see the lesson for the student emerge organically from that memory file.

STEP 3: Fact Check with ChatGPT Pro set to “high”

So, now I took the graphic from Claude that is above and I pasted it into ChatGPT, again using the screenshots. Its conclusion, “There are errors on both sides.” So, now this third model is finding errors on both sides of the equation, both Gemini and Claude. Here is the summary it found.

ChatGPT, to summarize, found the following:

• Claude got Vectara backwards and found the November 19, 2025 announcement and not the newer announcement.
• ChatGPT’s wording is important here, “I would not call most of Google’s individual numbers hallucinations. The more serious issue is they answer different questions…those percentages cannot meaningfully be placed on one common ruler. Here is what ChatGPT states about these.

A Vectara 1.8% means roughly: When the model is handed the source document and told to summarize only that material, how often does the summary contain something unsupported?
The Stanford legal number means: When an older general-purpose LLM is asked precise questions about federal court cases without necessarily being handed authoritative source material, how often is its response inconsistent with the legal facts?
The Mount Sinai clinical number means: If researchers deliberately planted a nonexistent medical fact in a case, how often will the model fall for the trap and elaborate on it?
AA-Omniscience asks yet another question about whether models guess rather than admit they do not know. Its hallucination rate denominator is specifically incorrect / (incorrect + partial + not attempted).

Oh my, so you mean my fact-check tool can be wrong too? Now, we’re getting past hallucination, and we need to be careful about throwing around this word. We’re talking about accuracy here, and mismatched research outputs can make a big mess.

As ChatGPT said, we don’t have one ruler; we have four rulers here, and they have mixed together things that don’t go together. Humans make mistakes. So do AI models that trained themselves on humans, and for the sake of simplicity, start boiling together ingredients that are basically like mixing chocolate and vinegar and dirt for good measure and expecting a Flambé.

STEP 4: Pasting ChatGPT’s answer back into Claude and asking for a Response

Ok, this is where the apologies start. We all know the drill. We catch AI making mistakes and then it is sickly sorry for what it has done. The two apologies included:

• Vectara Leaderboard, it pulled November 2025 instead of May 11, 2026. This is a good catch and precisely how you see how multiple models can help things.
• ChatGPT caught a logic error because it implied that o1 was not a reasoning model, but it was, so that comparison can’t demonstrate that reasoning models hallucinate more, only that the newer one scored worse than an older one.
• It said it had dismissed suprmind.ai as an “SEO content hub” without looking at the source of the numbers on that page. So, it looked at the source that held the numbers and didn’t realize that it also had a source, revealing a flaw. If something is searchable and findable and holds a number, sometimes AI only goes to that page instead of tracking back to the original sources.

So, then it said that ChatGPT was not right about Gemini 3 Flash at 92%, as it found that Gemini 3 Flash was at 88%, so they disagreed on that model so it is more accurate to leave the ceiling out.

At this point, I think most people would fatigue and say “what is right here” but I’m about to do a big old mammoth update to throw a whole bunch of data into my tool to help determine what is right but the conclusion – by the Ai models themselves – is going to be a powerful one if you can persist. Again, we’re running models against models. And in the end, students need to understand not only are there errors, but sometimes, those errors are there because of mistakes in looking at the wrong information and aren’t just “hallucinations.” That research is nuanced and that human eyeballs are valuable.

STEP 5: Multiple model fact checking

So, I just wanted to be done with this, so I took Claude’s results and put them into both ChatGPT and Perplexity. I specifically told Perplexity that I had used Claude and ChatGPT, and that I had put it in orchestrator mode, so it might need to use other models. I’ve linked the chats above for transparency and so you can see some of the exciting nuance that comes out of these.

Interesting tidbits:

• Perplexity and Claude both used the older November 2025 article instead of the newer article. As Claude said, “Perplexity confidently endorsed my wrong verdict using the same bad method that produced it.”
• ChatGPT was the tool that found that Llama 2 was the wrong example to use for Google’s ceiling.
• Because of “disagreements” between models and the fact they didn’t report the ceiling of some numbers, it is better to leave it out.

Now, if you look at my Claude chat for this, you’ll see a “retrospective,” which is where Claude analyzes what it got wrong. There is a method to my madness here with this and I’ll get there in a moment. But the Perplexity chat says something interesting:

"The teaching point in your footer is the real lesson: the primary sources (live leaderboard, paper abstract, system card) beat any AI summary of them, including one AI's summary of another AI's summary."

So, the AI itself acknowledges that we have a big old mess without consulting the original sources. So, then I took information from both ChatGPT and Perplexity and here’s the current output of AI evaluating the Google AI Overview.

OK, so this is interesting now. I want a graphic evaluating Google, and Claude is interjecting evaluations of itself. This is “mission drift” in action, particularly when you are fact-checking. So, I’m having to ask for a final graphic evaluating the Google generative results on the hallucination rates of AI models.

STEP 6: Re-generate the original graphic for the original purpose of this task

Now, I want you to note a few issues that I do not like about the information above:

• The citations are small and listed at the top. Again, we have a graphic and it is hard to fact check. I, thus, asked AI to generate a research box so I can read information and double check the conclusions.
• I would really like all models used to be documented somewhere somehow. It is citing the human, for sure, me – Vicki Davis- but it isn’t putting “Created using Claude Cowork” or any other models that I used in the fact-checking process. I think model disclosure will be very helpful in the future, even as we humans are held accountable. Being able to be cognizant of the need to document chats in this way is important. When you see the chat, you’ll see what I mean.
• Before I would produce this as “research,” I would need to sit down and read every single study, and I would argue that, with all of this back and forth, reading original source documents is more important than ever. We should be researching slower not faster when we see this happen.
• I would really like to be able to generate a link to share this claude check but because I run Claude Cowork on my computer, I’ll have to generate a PDF instead which I will paste below.

When you look at this, you’ll notice how I use Claude to fact-check my writing. Now, you might think – Claude was wrong; why would you use it to fact-check? Well, particularly in AI terminology, I’m learning and need to keep learning and understand various models. I believe that workflow is more important than ever, as are AI techniques, and to teach this, I have to understand and use the proper vocabulary. I live in rural Georgia — to call it the sticks might be an insult to sticks. So I can’t really go to my local coffee shop and hang out with the other AI nerds. I have to watch them on YouTube and read their articles on LinkedIn, so I’ve programmed the AI to help me be more accurate and precise in my AI speech. This is an example of using AI for learning. Also, the ability to produce a PDF of an AI chat is a valuable part of documentation as we look at the process of research.

STEP 7: Generate the Research Used for This Output with Hyperlinks.

I like to generate research as an HTML box I can paste into WordPress and then I can click on the links and review them in a new browser.

Research Citations

Every figure in the graphic above traces to one of the primary sources below, verified September 1, 2026. No aggregator numbers, no launch announcements, and no AI summaries were used as evidence.

Sources used

Grounded summarization
Vectara Hallucination Leaderboard (live repository, updated May 11, 2026). github.com/vectara/hallucination-leaderboard — source of the 1.8%, 3.1%, and 3.3% figures. The 9.6% median across 105 models was computed directly from this table.
Awadallah, A. and Mendelevitch, O. “Introducing the Next Generation of Vectara’s Hallucination Leaderboard.” Vectara, November 19, 2025. Read the announcement — cited for methodology only.

Open-domain factual recall
AA-Omniscience evaluation page, Artificial Analysis (live scores and metric definition). artificialanalysis.ai/evaluations/omniscience
Jackson, D., Keating, W., Cameron, G. and Hill-Smith, M. “AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models.” arXiv:2511.13029, 2025. Read the paper

Reasoning models
OpenAI. “OpenAI o3 and o4-mini System Card,” April 2025, Table 4. Read the system card

Legal research
Dahl, M., Magesh, V., Suzgun, M. and Ho, D. E. “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models.” Journal of Legal Analysis, 16(1), 2024, pp. 64–93. Read the paper · Stanford RegLab summary

Clinical decision support
“Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support.” Communications Medicine (Nature Portfolio), 2025. Icahn School of Medicine at Mount Sinai. Read the study · PubMed listing

Methods and terminology

Fiscus, J. G. “A Post-Processing System to Yield Reduced Word Error Rates: Recognizer Output Voting Error Reduction (ROVER).” Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, Santa Barbara, CA, 1997, pp. 347–354. NIST record — the documented technique for combining multiple speech-recognition outputs into one composite transcript more accurate than any single system.

Du, Y., Li, S., Torralba, A., Tenenbaum, J. B. and Mordatch, I. “Improving Factuality and Reasoning in Language Models through Multiagent Debate.” arXiv:2305.14325, 2023; published at ICML 2024. Read the paper — multiple model instances debate an answer across rounds, improving factual validity.

Goodfellow, I. et al. “Generative Adversarial Networks.” arXiv:1406.2661, 2014. Read the paper — cited for contrast. A GAN trains a generator and a discriminator together in one loop, with the discriminator’s judgment updating the generator’s weights. It is a training architecture, not a method for comparing finished models.

Sources Rejected, and Why

This list is the more useful one for classroom use. Each source below showed up during the research and was deliberately left out. Every link in this section is marked “nofollow” on purpose — see my note below.

1. Vectara’s launch blog post, quoted as current data. It is a snapshot from November 19, 2025, not the live leaderboard. Quoting its top scores as today’s produced the false claim that no model scores under 2%. The live repository shows 1.8%. The rule: a launch announcement records the day a benchmark was newest. For current numbers, open the artifact the announcement points to. See the announcement

2. AA-Omniscience’s launch article, quoted as current data. The same error on a second benchmark. Its “lowest at 26%” was quoted as the current floor; the live page shows 1%. Two details worth showing students: the article states its own figure inconsistently (28% in one bullet, 26% in another), and it explicitly says “For up to date AA-Omniscience scores, see the AA-Omniscience evaluation page.” The correction was printed right there and still got missed. See the launch article

3. Aggregator and content-hub pages. No methodology, no version history, no date-stamp showing which snapshot a number came from. The subtlety worth teaching: one such page’s numbers were actually correct. It was first dismissed for looking like a content farm rather than for any test of its figures — right conclusion, wrong reasoning. The real test is not whether a page looks credible but whether its numbers trace to a primary source you can open. See the page in question

4. “22% to 94% across 26 models,” attributed to the Stanford AI Index 2026. Could not be verified against the AI Index itself, and the descriptions that could be found identify it as sycophancy-induced hallucination — a different measurement. Two numbers both labeled “hallucination rate” are not necessarily measuring the same thing. See the AI Index report

5. “Gemini 3 Flash at 92%.” Sources disagreed and none was primary: reachable sources said 88%, two AI models later said 91%, the original claim was 92%. The live page publishes lowest scores but not highest, so the question stayed open. The graphic says “above 90%” instead. The rule: when sources conflict and no primary source settles it, report the range you can defend rather than the most quotable figure. See the Gemini 3 Flash analysis

6. Legal AI tool rates (Lexis+ AI 17%, Westlaw 33%). These come from a real follow-up study, but that paper was never opened during this check. Plausible is not the same as verified, so they were left out. See the follow-up study

7. A “1.47% real-world clinical hallucination rate.” Surfaced in passing, never traced to its source, so it was not used. No link — the source was never located, which is exactly why it was dropped.

8. The AI summary’s own citations. Worth pointing out to students: the visible citations under the search result pointed to content-marketing pages rather than to the Vectara leaderboard, the Stanford paper, the OpenAI system card, or the Mount Sinai study — the actual origins of every number it quoted. See one of the cited pages

A caution for anyone reusing this graphic: benchmarks update continuously. These figures were current on September 1, 2026. Check the live pages before quoting them later.

Notes about the research citations above from Vicki: So, do you see what I did there, it said “we” but then when it cited it, it cited “me” – Vicki. I don’t like this subtle use of pronouns. AI is a tool and I’m accountable. However, if I look at the chat, I am using the word we as well — I have to think on that. wow. Look at that.

Additionally, I would like the links to the articles that were rejected. Update: When I looked at adding those links, Claude pointed out something I had forgotten, that the presence of a true link passes SEO on for credibility to those sites, something I’m not really wanting to do so it will mark them as “no follow” links. This is an interesting aspect of citing rejected articles. Also, note that when I do this again, I’m going to create a version of this skill that stops and lets me make the decision as this chat was in “auto mode” for speed. I really had no idea this would be such a hard question to get right in Google generative search, but I’m glad I did this activity and it is one I will be doing with students.

STEP 8: Updating the Fact Checking Skill in Claude Cowork

Now, I’m coming back to the fact-checking skill I’ve built in Claude so you can see why I keep coming back. When I’m done, I ask it to analyze every mistake and then update the fact-checking skill. This is a whole other process because now, I’m working to teach the AI tool how I like to operate and to learn from interacting with other AI tools about the flaws and mistakes in that tool.

This is why skills will become valuable intellectual property for companies (if indeed they can be owned by the company and not harvested by the AI models themselves, which is a whole other topic.)

RETROSPECTIVE

In every chat, I ask Claude Cowork, ChatGPT, or Gemini to conduct a retrospective analysis of what worked, what mistakes were made, and how to prevent those mistakes in the future. It is a learning model. Our purpose in interacting with AI is both to get a job done and to make doing that job better in the future.

Additionally, by doing this sort of orchestration, we can build distrust for the overly simplified generative answers we’re getting from Google right now – or really any tool, for that matter. No tool is always right. Different domains of knowledge have different accuracy rates, and different ways of using AI can yield higher accuracy. Sometimes we need an ensemble in order to do complex tasks.

But here is a big takeaway: When in doubt, go to the original documents to check it out.

When we start using multiple AI tools to check for answers, we see that each tool is flawed and that we must use discernment to understand the nuances. Plus, rushing to get something out can lead to mistakes.

So, I’ve just written this as I’ve gone through the process, when I realized this wasn’t going to be an easy check. I was actually preparing a presentation about research using AI and just wanted to expose flaws, and the rabbit hole went much deeper than I thought!