An AI pilot can deliver impressive results and still fall short of what a business needs. A translated page may contain mistakes; an automated report may need corrections before a client can use it. This requires human intervention, which increases time and cost spent on delivering a product. For tech leaders, the question is whether an AI-driven solution delivers enough value once you account for that additional work.
The experiences of five business and technology leaders suggest a useful distinction. Redesign may make sense when a different workflow or better review could resolve the problem. Stopping may be warranted when the tool cannot meet an essential requirement or the remaining risks outweigh the expected benefit. Narrower automation use may preserve value, too. The decision depends on identifying what failed and what, if anything, would make the investment worthwhile.
Evaluate What Customers Will Actually Receive
Anthony Guerriero, managing partner at Manhattan Miami Real Estate, wanted AI to help make his brokerage’s English-language website available in 18 additional languages. The pilot passed its checks, he says, but the resulting pages contained problems those checks had missed.
“What made me stop and redesign? The checks were all green and the pages were wrong. 18 language versions of one page, same English title on every one. Then a page showing zero Luxury Developments where the English had 38. Our checks read the code. Nobody read the page,” Guerriero tells Dice.
Guerriero, who also co-founded the AI-training business The Leveraged Years, says the requirement became more demanding than simply producing a translation. The content needed to read as if a local luxury broker had written it. He kept the project and rebuilt it around that expectation. “A page isn't done until someone has looked at the page itself,” he says.
His response addressed a gap in how the work was checked. Before deciding whether to abandon a use case like this, leaders need to evaluate the finished output against customer expectations. Passing a code check doesn't mean a translated page is ready to publish.
Alex Donovan, founder of (a)squaredstudio, encountered a different obstacle while testing AI-generated logo drafts. His studio hoped to shorten the time between receiving a client brief and presenting concepts. He says clients generally preferred the in-house designs when shown both options, but quality was only part of the decision.
“Any file we deliver to a client has to pass through review, and if they are going to use it commercially they have to know where all of its source materials come from. The tool could not say where its source materials came from,” Donovan tells Dice.
The tool could not satisfy the studio’s source-material requirements, preventing the pilot from proceeding. Better-looking drafts would not have resolved that obstacle. Before extending a trial, leaders should identify the unmet requirement and what evidence would show it can be met. Another round of testing needs a specific question to answer.
Test the Assumptions Behind the Business Case
Jason Long, founder and CEO of Support My Website, wanted report automation to free an analyst for more valuable client conversations. “Our goal was never to replace the analyst. Our goal was to give the analyst their time back,” Long says.
Testing against previously validated reports revealed persistent errors specific to some clients, according to Long. He says the team stopped that use case. The intended benefit depended on reliable reports, so generating a draft quickly wasn't enough. Leaders evaluating similar projects should compare the time needed to review, correct and prepare AI output for delivery with the time required by the existing process.
Evgenii Arsentev, CEO of AskDocDoc, asked whether changing how his company used coding agents could reduce costs. “We had a habit to clear the agent's context after every task, so I tested this habit,” Arsentev tells Dice. In practical terms, he compared starting fresh after each task with letting an agent carry information across several tasks. He tested six approaches across 36 runs. In his experiment, starting fresh after every task produced higher estimated costs under his model than retaining that information across several tasks.
His technical report is not peer-reviewed, and its corrected version documents revisions to the cost calculations. The agents also wrote different tests to check their work in different runs. Passing those tests therefore did not show that each approach produced work of the same standard, which limits the cost comparison. Arsentev also acknowledges that he chose his preferred session length after seeing the results.
Before stopping a pilot over cost, leaders should test whether changing the workflow could improve its economics. Arsentev’s experiment illustrates that question, but its limitations leave more to establish before another team adopts his approach. Any claimed savings still need to be tested against comparable work.
Make the Decision Defensible
Stopping a pilot also requires explaining why further investment is difficult to justify. Sponsors need enough detail to understand the recommendation and communicate it to others.
“Data-driven decisions are easier to swallow than opinion-driven ones,” Long says. He points to specific errors, the cost of correcting them and the potential damage to client trust. For him, the decision depended on whether the proposed savings were worth the risk of sending a client an incorrect report.
“I emphasized that we’d be gambling with client trust on reports, and losing their trust over one bad report would cost far more than what we’d be saving on the project,” he says.
His recommendation rested on that assessment of potential harm. Leaders making a similar case should distinguish observed errors from their judgment about what those errors could cost.
Give sponsors the same connection between the evidence and the recommendation: what failed, how it affects the business and what should happen next. For a redesign, specify the change and how the team will test whether it works. For a stop recommendation, explain why further spending is difficult to justify and what would need to change before reconsidering the use case.
Preserve Useful Work and Redirect the Team
An unsuccessful attempt at full automation can still reveal which individual tasks benefit from assistance.
Amit Raj, founder of The Links Guy, describes that outcome. His team wanted to speed up research and outreach for link building and digital PR. AI helped with data collection and enrichment, but it struggled to select the right contact within an organization and write suitable, personalized emails.
“So we settled on keeping a tool called Pitchbox in the process,” Raj tells Dice. The team kept assistance with outreach but required a person to vet every email. Raj says employees whose work centered on data collection and enrichment also needed to develop skills in content production and outreach strategy.
That decision gave the team a more specific role for automation and identified where employees needed to expand their capabilities. For managers, a redesign should include both elements: the revised process and the responsibilities people will take on within it. Retaining a useful tool is only part of the decision when the work around it also changes.
Conclusion
The practical test is whether another round of work has a credible path to meeting the business requirement. If leaders can identify a fix and a way to evaluate it, a redesign may be justified. If the tool cannot meet an essential requirement, or the likely benefit does not justify the remaining cost and risk, stopping the use case may be the better decision.
Also consider a narrower workflow when some tasks benefit from automation and others still require human judgment. Whichever direction a leader chooses, the team should know what work will continue, what will change and what evidence would justify revisiting the decision. A pilot’s findings are useful when they lead to that clear next step.
