Brief #2: The Question Behind Every AI Budget
Welcome back to the Chief of AI Brief. Last week we covered why you need someone to own an AI implementation. This week: the question that's keeping every AI owner awake at night.
If someone shared this with you or you came from LinkedIn, subscribe to Chief of AI Collective to get your Briefs directly in the inbox.
Andreessen Horowitz's annual survey of 100 CIOs reveals AI spending in business will jump from $2.5 million in 2024 to $12.3 million by 2026 — that's 392% growth in two years. For context, cloud computing only grew 35% annually at its peak. It's a genuine space race where companies are spending like their survival depends on it.
But here's the real kicker: while executives are spending all this money, 47% cite "lack of clear, measurable outcomes" as their biggest challenge with AI pricing. Continuing our space analogy: we're all trying to land on the moon, but we can't tell if we're getting closer or just burning fuel in orbit.
📊 Reality Check
The measurement crisis is real, and the numbers are brutal:
Over 80% of organizations aren't seeing tangible impact on enterprise-level EBIT from their GenAI use. (McKinsey, March 2025)
42% of companies abandoned most AI projects in 2025, up from just 17% the previous year, with "unclear value" as a top reason. (S&P Global, February 2025)
39% of CIOs prefer usage-based pricing over outcome-based pricing of AI models, because it’s hard to quantify and properly evaluate what the outcomes of using AI are. (a16z, June 2025)
Why is this happening? Because most companies don't have proper telemetry—automatic data collection and transmission—in place to track AI impact.
Traditional metrics work for buying software: pay X, get Y features, measure usage. But AI impact shows up in ways that require new measurement approaches, and most systems aren't set up to capture that. The good news is that some companies are starting to figure it out.
📈 What To Track
1. Usage Depth Metrics
Not just "who logged in" but "who automated their daily tasks"
Track: Percentage of tasks completed with minimal or no human intervention.
Ask IT to track adoption rates before and after AI rollout in CRM/project tools. You want weekly reports showing total tasks attempted vs. those requiring no edits.
🚩 Red flag: High adoption, low task automation.
2. Velocity Metrics
How AI changes the speed of core business processes.
Track: Time from input (e.g. customer inquiry, document submission) to business output (response, approval).
Ask IT to measure timestamps in existing systems: time from customer inquiry to first response, document upload to approval, data request to analysis completion. Compare 90 days before AI vs. 90 days after.
🚩Red flag: AI tools used but process speed unchanged.
3. Cascade Impact Metrics
Secondary effects of AI improvements on other business areas.
Track: Customer satisfaction, error rates, capacity utilization.
Ask IT to pull existing metrics from customer support (ticket volume), quality systems (error rates), and capacity reports (same headcount handling more volume). Look for correlations with AI deployment dates.
🚩Red flag: Direct metrics improve but downstream metrics stagnate.
The bottom line: Successful companies are tracking four outcomes: cost savings, revenue impact, risk reduction, and efficiency gains—but through operational data, not just financial KPIs. Cost savings = fewer resources needed; efficiency gains = same resources, more output. Track both for the complete picture.
💬 What metrics is your organization using? Hit comment — we'd love to hear what's working (and what isn't)
If you're early in your AI measurement journey, here's a 30-Day Starter Plan. It’s not exhaustive, and some “weeks” may stretch depending on your org, but it’s a reliable place to begin.
Week 1: Audit what metrics you're currently tracking and begin telemetry planning with your IT team (many companies only track basic usage but have untapped data sources).
Week 2: Pick one metric from each category (usage depth, velocity, and cascade impact) to start measuring and begin implementing telemetry for high-priority systems.
Week 3: Set up baseline measurements before any new AI deployments
Week 4: Create a simple dashboard with your IT team using available data and finalize remaining telemetry infrastructure
More on the measurement challenge:
Proving ROI: Measuring the Business Value of Enterprise AI (Agile At Scale, 2025)
The Complexities of Measuring AI ROI (Devoteam, 2025)
The ROI puzzle of AI investments in 2025 (The CFO, 2025)
Thanks for reading,
Chief Of AI Collective