Boards face an AI cost reckoning as token costs start to bite

For the past two years, most discussions about AI have focused on adoption. Which model should we use? How do we encourage staff to embrace it? Where can it improve productivity?

Increasingly, a different question is landing in boardrooms: what is all this actually costing us?

Until recently, choosing an AI platform was mostly an IT and risk decision. Businesses looked at how well it integrated with existing systems, whether it met governance requirements and whether staff could practically use it.

And whichever platform you picked, the monthly per-seat licence fees were largely comparable and predictable, with usage pooled and included in the price.

The equation changes once hundreds of employees start using AI every day. Coupled with increased use of agents and new high-powered models, companies are chewing through a previously invisible commodity: tokens.

A token is the unit AI providers use to measure how much language a model processes. Every prompt you type in, every document you upload or reference, and every word the AI spits back contains tokens, roughly 100 tokens per 75 words.

That’s where costs can get out of hand surprisingly quickly.

If you upload a host of documents as a reference point, keep very long conversations live that require the AI to constantly reference past answers, demand extensive reports or visuals, and use higher-end reasoning models, you are going to be using tokens in volume.

If you hand that piece of work to an autonomous agent, which will determine its own workflow and repeatedly call on the model as it completes elements of the task, you can use exponentially more.

A million tokens is equivalent to an AI outputting about 1,000 images, 650 PDF pages, about eight novels or 10,000 work emails.

But there can be a 60x difference in cost between the most and least expensive models.

A million tokens can cost as little as $3 for Gemini 3 Flash, $50 for Claude Fable 5 or $180 for the newly released ChatGPT 5.5 Pro.

The conundrum for business is that without careful oversight, one inefficient prompting team member, or one over-enthusiastic AI agent, can rapidly run up costs, in part because there are no price signals for the average user.

And while AI agents promise to free up staff from mundane tasks, there are clear financial risks in building a set-and-forget workflow that calls on the most advanced reasoning models to solve simple problems: it may get the job done, but at what cost?

This is where boards and executives need to rethink the discussion.

It can’t just be about why people should use AI, or where they should use it, but what the return on investment needs to look like to justify it.

There are practical measures businesses can take to prevent runaway spend on AI tokens, while still accessing the benefits of AI in business.

The right tool for the job

In the rapidly developing field of AI, there are many providers and models available to choose from, at a wide range of token prices. For many businesses, it might be worth taking a two-tier or more-tiered approach, where everyday AI queries take place on cheaper models, and only high-value use cases get the heft of a complex reasoning model like Claude Fable-5.

It might be that all staff can continue to use the company’s preferred model, but there’s internal training around the best way to maximise efficiency in its use.

Efficient architecture

AI models return much better results when given lots of context: long, well-written prompts, background documents and tables of data. However, all that context requires a lot of processing. Similarly, lengthy responses consume more tokens than concise ones, and repeatedly asking the same questions can make the AI model work hard to respond each time – churning through precious tokens in the process. Another rule of thumb is: “don’t make the model think about something you already know”.

The message is that architectural decisions matter: there are ways to design an agentic workflow to use AI processing power where it matters, and look up known information where it can.

Token budgets

As AI-powered workflows become embedded in middle-market businesses, boards and executives should consider what disciplines and financial controls are needed to prevent runaway costs. Where agentic AI is on the agenda, this will require careful investment, starting with evaluating uses with a proper ROI calculation that includes token costs as a core line item.

It may be appropriate to set a budget of tokens per month for each department, or to set up a mechanism for IT to charge back AI token costs.

Boards are used to managing software licences, cloud costs and cyber security risk. AI now deserves the same discipline. That means understanding where tokens are being consumed, which use cases are delivering genuine productivity gains, and where expensive models or autonomous agents are adding cost without adding value.

What shouldn't change is the need for visibility. Businesses that understand their AI consumption today will be in a much better position if the economics shift tomorrow.

Martin Koval, client director, Pitcher Partners.

As a client director in the digital and data solutions team, Martin advises middle market businesses and not-for-profit organisations on how to maximise the value of their technology investments. He specialises in technology strategy, process optimisation, software and vendor selection, and guiding organisations through complex system implementations and integrations.