Claire Whitfield, the company’s senior underwriter, placed a loan application on the conference-room screen and asked Devin Park, the product manager to run it through the new AI system. It wasn’t one of the loan applications the team had used during development. Claire had pulled it from the company’s active queue that morning because it represented the kind of borrower the company served every day. The applicant owned three restaurants, moved money between several business accounts, and received revenue through two payment processors.
The bank statements contained deposits that appeared to be revenue in one account and internal transfers in another. The legal name on the application didn’t match the name printed on two of the statements. One restaurant had been operating for seven years, while the other locations had opened within the past eighteen months.
The AI underwriting assistant accepted the documents and began its review. Less than a minute later, it recommended declining the application because of unstable cash flow, inconsistent business identity, and a debt obligation that exceeded the company’s lending criteria.
Nobody spoke for several seconds. The product manager leaned toward the screen and opened the explanation behind the recommendation.
The system had counted transfers between the applicant’s accounts as revenue in one part of the analysis and as unexplained withdrawals in another. It had treated all three restaurants as separate businesses because their operating names differed. It had also interpreted a monthly payment to a food distributor as debt service.
Claire knew the application wasn’t simple, but she didn’t consider it unrecognizable. Her team handled cases like it each week. An experienced analyst would have reconciled the business names, removed the internal transfers, and distinguished the supplier payment from a loan obligation before making a recommendation.
The system hadn’t stopped when it encountered those contradictions. It had produced an answer with the same appearance of certainty and the same polished explanation it displayed during the demonstration.
A week earlier, the company’s executive team had watched the assistant return an underwriting recommendation in under a minute. By the time the meeting ended, the CEO wanted a deployment schedule.
Belief is Not the Same as Reality
I believed the demonstration showed promise. I’d seen the system extract information from financial records, identify inconsistencies, calculate cash flow, and generate a recommendation. I had also given enough demonstrations during my career to know how much preparation can disappear behind a smooth presentation.
A demo is a controlled environment designed to produce one result. That result is belief.
The team chooses the input, prepares the path, confirms the connections, and removes interruptions that could distract from the capability it wants people to see. The audience enters the room hoping the system works because the company has a problem it needs to solve. Nobody is assigned to want the demonstration to fail.
That doesn’t make the presentation dishonest. Most teams choose a favorable example because they want to communicate what they’ve built. A chef doesn’t introduce a restaurant by serving the dish that gives the kitchen the most trouble, and a product team doesn’t begin an executive demonstration with the case most likely to break its system.
The problem begins when the company mistakes that selected performance for evidence about the real operation.
The application used in the original demonstration came from an established business with one legal name, one primary bank account, and financial records in formats the system had already processed. Its deposits were easy to classify, and its existing obligations appeared under labels the system recognized. The case was real, but it occupied the part of the lending portfolio where the system performed best.
The application Claire selected came from the same portfolio. It represented the conditions the system would face after deployment, when nobody prepared the documents or chose the borrower according to the AI’s strengths.
The same system had produced both recommendations. The difference was the condition of the evidence it received.
Salma Wahwah, our Principal Data and AI Architect, begins with the information entering the system rather than the AI itself when she examines why a demonstration differs from daily business use.
Salma’s delivery experience shows that the information used in a demonstration has almost always been cleaned before anyone in the room sees it. Required details are complete. Records referring to the same business have already been matched. Documents arrive in formats the system knows how to read. None of that is deception. It is what the process looks like on a good day. The problem is that the good day is the one that gets demonstrated.
This article is extracted from Steve Taplin’s new book: THE AI VALIDATION GAP. To learn more, visit AMAZON:




