The evidence built to persuade us is almost always better made than the evidence built to test us.
I discovered this firsthand when I clicked on the final screen, and the risk score instantly appeared. A loan application that once required an analyst to review bank statements, transaction histories, credit records, and identity documents had moved from submission to recommendation in less than forty seconds.
The demonstration took place in 2025 inside the conference room of a FinTech company. I was working with the leadership team as it considered an AI system designed to evaluate applications from small businesses across the United States seeking working capital lines between $250,000 and $5 million. The company had been losing borrowers because its approval process took too long, while competitors were promising decisions within hours.
The system appeared to solve that problem. It extracted information from the application, compared the company’s cash flow with its requested loan amount, identified inconsistencies in the supporting documents, and produced a recommendation with a confidence score. The result came with a summary explaining why the applicant qualified.
I had helped the product team prepare the presentation. I knew which features would matter to the executives, which numbers would hold their attention, and which comparison would make the investment feel inevitable. The human process took more than three hours of analyst time across several days. The AI, by comparison, produced its answer before I had time to finish describing the old workflow.
The head of lending asked whether the system could support the volume projected for the next year, and our technical lead said it could. The chief executive immediately became excited and wanted to know how soon we could deploy it.
Nobody in the room was careless. The people asking questions understood lending, risk, technology, and growth. They also knew that every day lost in underwriting gave another lender an opportunity to win the borrower.
I believed the system worked. I had watched it process sample applications, reviewed its accuracy results, and seen members of the lending team compare its recommendations with decisions made by experienced analysts. The model seemed faster without sacrificing judgment.
When the discussion turned to funding, I supported the investment. The company approved a larger implementation, committed a dedicated delivery team of six people to it for nine months, and began planning how to extend the system across more loan products.
The evidence that persuaded all of us was real. It was also incomplete.
The applications used in the demonstration had been selected from a set that the team understood. Their documents were legible, their transaction records followed formats the system had seen, and their financial histories contained few contradictions. The system had access to the information it needed, presented in forms it could interpret.
Production applications did not arrive that way. Bank statements were missing pages. Business names differed across documents. Revenue deposits were mixed with transfers between accounts. Applicants submitted files in formats the system struggled to read. Some borrowers operated seasonal businesses whose cash flows looked dangerous when viewed without context.
The AI still produced recommendations. That was part of the problem. It did not stop and announce that the conditions had changed. It returned answers with the same visual confidence it had shown in the conference room. The model had left the territory it was trained on, and nothing in its architecture told it so; confidence scores measure how familiar an input looks to the model, not whether the model has actually left familiar ground, so the two can point in opposite directions at once.
Analysts began checking more of the AI’s work. Applications that appeared to move faster through the system created review work elsewhere in the process. Some cases were routed back for manual examination, while others required analysts to reconstruct the financial picture before deciding whether the recommendation could be trusted.
The checkpoints multiplied. Every recommendation carried a verification step, and every exception carried a second one, until the analysts doing that checking became the bottleneck the system had been purchased to remove, a version of the reviewer bottleneck that shows up anywhere people are asked to rubber-stamp a fast machine instead of doing the judgment work themselves. Time to approval went up, not down. The borrowers the company had been losing to faster lenders were still waiting, and now they were waiting on a review queue nobody had staffed.
This article is extracted from Steve Taplin’s new book: THE AI VALIDATION GAP. To learn more, visit Amazon:





Evidence built to persuade versus evidence built to test is the whole problem in one line. In validated shops we split the two by having someone who didn't build the demo write the test cases, starting from the cases that went wrong last year. Did the team in your story ever run the model against applications the analysts had declined?