Why Businesses Must Now Measure AI Usage
▼ Summary
– A significant gap exists between executive confidence in AI visibility and the reality described by managers, with a 16-point perception difference across roles.
– Widespread “shadow AI” usage and a lack of comprehensive tool inventories complicate governance and risk management, as adoption often outpaces measurement.
– Return on AI investment varies by industry, with sectors like retail and software seeing faster results than healthcare or hospitality due to workflow differences.
– AI outcomes differ by job function, with IT teams reporting the strongest results and customer support roles expressing the lowest ROI confidence.
– Most workers see modest time savings from AI, but a small group of power users achieves significant gains, with formal training strongly correlating to higher proficiency and value.
Understanding the true impact of artificial intelligence on daily operations has become a critical business imperative. While adoption accelerates, many organizations lack the visibility needed to manage costs, mitigate risks, and prove return on investment. A recent survey of enterprise leaders highlights a growing urgency to implement robust measurement and governance frameworks, revealing a significant disconnect between perceived and actual AI usage across teams.
There exists a notable confidence gap within leadership structures. Executives often express strong assurance in their organization’s AI visibility, while directors and managers closer to daily work report a very different reality. This divergence creates a 16-point gap in perception, a persistent issue regardless of company size or industry. A contributing factor is the prevalence of shadow IT, where employees utilize personal or unsanctioned AI applications. Over one-fifth of leaders acknowledge this as a barrier, even as most claim high confidence in their oversight. Procurement data offers a limited view, tracking licenses but failing to capture actual desktop-level usage patterns.
As one industry CEO observed, a dangerous assumption prevails in boardrooms: that AI is both visible and under control. In truth, adoption frequently races ahead of measurement, with inconsistent governance turning a potential strategic asset into a liability. The solution hinges on organizing efforts around real-time, accurate data.
Most companies now rely on multiple AI products, with high-performing organizations using an average of 2.7 specialized tools compared to 1.1 for lower performers. This diversification, while enabling specific workflows from coding to content creation, introduces redundancy and potential budget waste. The proliferation is compounded by embedded AI features within standard software platforms. The average large enterprise operates 23 distinct AI tools, with nearly half adopted outside formal IT channels. Alarmingly, only 38% of organizations maintain a comprehensive inventory of these applications, complicating governance, budgeting, and compliance with emerging standards like ISO 42001.
Return on investment varies dramatically across sectors. Industries like retail, software, and manufacturing often report a high likelihood of realizing ROI within six months. Their success is frequently tied to workflow structures that allow knowledge work to be broken into discrete, automatable tasks. In contrast, sectors such as healthcare, hospitality, and restaurants report slower progress and lower expectations, often anchored by physical operations or stringent regulatory processes. Healthcare presents a unique case, exhibiting high executive confidence in visibility alongside the lowest ROI expectations, a clear sign of governance friction.
Performance differences are equally stark across job functions. IT departments report the strongest outcomes and highest confidence, using AI for measurable outputs like code generation and system automation. Functions like customer support and logistics, however, show lower confidence. Their AI use often centers on drafting and summarization tasks that yield incremental, hard-to-measure gains. Despite heavy investment in chatbots and assistance tools, customer support roles report the lowest ROI confidence of all functions.
At an individual level, most workers see modest time savings from AI, with over 85% saving under ten hours monthly. A small cohort of power users, about six percent of the workforce, saves more than twenty hours each month by leveraging multiple tools and advanced capabilities. Formal training programs show a strong correlation with higher skill, satisfaction, and productivity gains, a nuance that simple utilization metrics like login counts fail to capture.
Structural challenges continue to hinder effective measurement. Thirty percent of surveyed leaders point to unclear responsibility for tracking AI impact, with fragmented ownership across teams being a major issue. While 69% of organizations have AI risk and compliance policies on paper, execution is inconsistent. Many lack clear visibility into adoption rates, risk exposure, and concrete value metrics. Notably, companies with formalized governance structures demonstrate a higher likelihood of achieving ROI, underscoring the importance of alignment between leadership, security, and operations.
Currently, tracked metrics tend to favor ease of collection, money saved, percentage of users, weekly time saved. Far fewer organizations monitor deeper indicators like investment per tool, functional maturity, or delivery speed improvements. These measurement gaps ultimately limit a company’s ability to definitively connect AI usage to tangible business outcomes, leaving significant value on the table.
(Source: HelpNet Security)