AI Makes Mistakes. Here’s How You and AI Can Fix the Flubs
A dealer who skips this review step gains speed—until the first bad answer.
By John Marshall
I work from the assumption that general AI tools are about 85% right. It's a useful number, because it helps you realize how much the other 15% can cost you. It’s why a takeoff gets looked over by a second person before it goes out. AI answers need the same treatment.
In recent work for a manufacturer I advise, I had ChatGPT review my analysis, then checked its findings myself. That process caught two mistakes, with two very different causes.
The first was a wrong cost figure I had circulated, in which I had treated an administrative fee in the materials as premium. The second occurred when ChatGPT asked whether a margin I was using already included outbound freight. If it did, the freight subsidy I was modeling was partly double-counted. If it didn't, the gross profit on the account was overstated by the freight cost.
As it happens, when I went to operations to double-check, I found that both ChatGPT and I got the freight class wrong.
Build in Reviews
Ultimately, I decided that asking AI to reread its own answer wasn't enough. So I built review skills on both platforms I use, Claude and ChatGPT, and each works two ways.
First, I tell it to start over with the source documents, list the inputs and assumptions, and work through the answer again before comparing it with the original. If there are differences, I tell it to show me where. That goes a step past how a takeoff usually gets checked, where a second person reviews what the first one built. Starting over is how you catch an assumption the first pass never stated.
The second review practice calls for one AI tool to review the other's work. A different model (Claude vs. the Google Gemini tool used in the beginning) can catch what the first one missed, and the question you never asked is a good example. It can also be wrong, which is why the check has to run against a source.
The checklist given the tool changes with the work. For finance, rerun the math. For legal, check the jurisdiction and read the actual clause. For HR, check federal and state rules separately. For sales, tie the numbers back to the CRM or quote. Every review ends with a short summary of what was checked, what was assumed, what is inferred and not confirmed, and what needs a professional.
Most of the value came from having something to check against, such as a source document or a density chart. A second reviewer gives you a list of things to verify, and the verifying is still your job.
I haven't put a number on how close this gets me to 100%. What I can say is that the catches above were ones I had missed on my own.
Implications for Dealers
Checking the work is a cost of using AI well. A dealer who skips the review gains speed—until the first bad answer. The dealer who builds a review step gets both speed and a way to catch the bad answer before it goes out the door.
If you already use two AI tools, you need no new software and no IT department, only a habit and a checklist. By my estimate, the review adds about 25% to the time I spent on the original work, and the total is still significantly less than doing it without AI. The extra time is worth it to get closer to 100% accuracy.
You wouldn't skip the credit check on a large account to save time, and this is the same trade. On contracts, tax, and employment decisions, your attorney and CPA stay the last stop, and they receive cleaner work and a better-prepared question.
Where to start
Pick the three kinds of AI work where a wrong answer costs real money, such as contracts, financials, and personnel decisions.
Require the tool to list its inputs and assumptions before it answers, and to ask for anything missing.
Have a second tool, or a person, rebuild the answer from the source documents and compare. Don't let the author grade its own work.
Verify every finding the reviewer raises against the source before you accept it.
Keep your attorney and CPA as the last check on anything with legal or tax exposure.
AI changed how fast the work gets done. It didn't change who answers for the result.
John Marshall is owner of AI Growth Partners. Previously, he was President and COO at Wilson Lumber and President at AllTemp Windows. Along with his work at AI Growth Partners, he is interim CEO and Board Chairman at Tara Manufacturing.
Comments