Tech execs are getting wise about ROI from AI

09/10 14:15 - Tech execs are getting wise about ROI from AI
Most organizations have been largely unable to measure financial returns from AI, but analysts say new ways to calculate return on investment are emerging. “There’s a delay between the development of technology, even the investment in the technology, and the value that an organization can capture from it,” said Michael Chui, a senior fellow at McKinsey. But more executives are asking questions. “The CFOs are asking CIOs, investors are asking CEOs: ‘Where’s the ROI from this stuff, already?’” he said. In McKinsey’s “State of AI” survey released in August, about 80% of respondents said AI improved their productivity. But only 37% said AI’s impact showed up in profits, about the same as last year. An even smaller number — only 6% — said AI delivered significant value and accounted for at least 5% of their operating profit. In other words, there’s a drop-off between the value that individual workers are getting from AI and the value that organizations are getting from AI, Chui said. The biggest gains will come from redesigning workflows and processes in which humans and AI agents work together, according to McKinsey’s Technology Trends Outlook. Layering agents onto existing processes isn’t enough. “Usually an end-to-end workflow involves multiple individuals, and completely redesigning that with the use of AI… is characteristic of high-performing companies,” Chui said. Controlling costs Managing token costs and applying the right model for a task is part of realizing better returns, Chui said. “In many cases, there just isn’t transparency… Which workloads are actually driving your costs?” he said. Three out of five IT leaders are worried about AI agents running up unexpected costs, and this is already happening, said Gareth Herschel, a vice president analyst at Gartner, during a keynote at Gartner’s Data & Analytics Summit in Mumbai. “Some organizations have already discovered that the cost of tokens for coding assistance is much higher than the cost of human software developers,” Herschel said. As more agents work together, “your financial risk only grows. It’s like giving your teenager your credit card… I’m sure you will learn a lot, but mostly from the bill,” said Robert Thanaraj, a senior director analyst at Gartner and a co-speaker at the Mumbai keynote. Companies should track costs in prototyping, such as finding the cost of an individual agent per completed task, Thanaraj said. “It’ll help you to evaluate different large language models or help you to go with a more affordable option, such as smaller language models or open weights model.” A wider lens for ROI Analysts highlight numerous challenges in calculating AI ROI, such as unexpected costs, poor data quality, failure to scale, and slow adoption among users. But executives are skilling up in tracking what they spend on AI and the returns, said McKinsey’s Chui. “Between the CFO and the CIO, we’re starting to see these disciplines emerge.” In 2025, the odds of an AI initiative achieving ROI were one in five, the Gartner analysts said in their keynote. “ROI matters, but to achieve it, we must think of it not just as a financial metric, because value isn’t always just about money,” Thanaraj said. Companies should tie AI projects to both financial and non-financial outcomes, part of what Gartner calls a “return on intelligence.” “We need to shift the emphasis from cost to value,” Herschel said. “The outcomes can be financial, such as revenue, but they can also be non-financial, such as citizen experience.” The right foundation: context, infrastructure, governance The Gartner analysts said achieving ROI on AI requires a strong technical and contextual foundation. “Governance adds trust. Context adds meaning. Without strong foundations, AI may well stand for amplified ignorance,” Thanaraj said. For example, data quality can be a roadblock. “Without clear context, LLMs are just guessing,” Thanaraj said, and that amplifies misunderstanding. Poor data and poor AI design mean more hallucinations and bad output. “You can’t buy this context layer off the shelf. It has to be built to fit your needs,” Herschel said. A strong technical foundation, such as a robust networking backbone for data movement, is critical, said Jack Gold, principal analyst at J. Gold Associates. “Agent-to-agent interactions will become commonplace and mission-critical, even as the number and distribution of agents expands dramatically to include interactions across remote agent locations and devices,” Gold wrote in a research note. A majority of organizations are establishing harnesses — the software layer that controls and coordinates models, tools, and workflows — to govern AI use in business. According to a global KPMG survey released last month, 55% of organizations have a formal AI harness layer. That rises to 86% among organizations reporting established ROI. Organizations that “combine clear accountability, coordinated governance, resilience, and reliable value measurement will likely be best placed to turn broad adoption into sustained performance,” KPMG said. ...