Executive summary
AI tools don’t agree because each one writes its own query, and the query is where a metric gets computed. A shared definition tells a tool what retention means, but the tool still decides which tables to join and which rows to count. In the mental_health database from BIRD-Interact, a public text-to-SQL benchmark, one join put 42 of 157 doctors’ rates wrong. It’s critical to compute each metric once, in a universal semantic layer so that people can use whichever AI tool they like and still get the same number.
Your team spent a quarter getting everyone to agree on what retention means, and it worked. Retention now has one definition, written up in a metrics doc and loaded into each AI tool as a skill or prompt.
- Compute each defined metric once, in a universal semantic layer (with the AI Computation Engine, ACE), and any AI tool your team picks returns the same number.
- Every model on the BIRD-Interact benchmark scored 72% to 137% higher on 410 business questions (best of three runs) with AtScale.
- Not every tool can read your definition, so a BI tool and an agent may each rebuild it their own way.
- Writing a metric definition down doesn’t make AI tools agree. Each tool still guesses which data to use and how to count it.
Then the CFO asks two of those tools for customer retention by region. The company copilot gives one number, and the in-house agent comes back with another.
Nobody changed the definition, and both tools read it. If every AI tool computes your metric separately, you’ve defined it once and computed it twice. Two is a low count, too, since each worker uses an average of four AI tools, according to one recent workplace survey.
Why don’t AI tools agree on the same number?
AI tools don’t agree because each one guesses which data your definition points to. Ask for retention, and the model scans the schema for tables and columns that look like retention, even when ten of them have the word in their names. It picks one and writes a query around that pick. The model never does the arithmetic. It decides what gets counted, and two tools rarely decide the same way.
Take retention. One tool might count a customer who downgraded as retained, while another drops them. One might start the quarter on the first of the month, and another on the first business day. A one-line definition doesn’t settle those calls, so each tool makes its own.
The gap grows with every model you add. In a survey of 100 enterprise CIOs, 37% now run five or more AI models, up from 29% a year earlier. Every new model or agent is another place your retention number gets computed. Once leadership sees two numbers, people stop asking the AI, or they check every answer by hand.
How does an AI tool compute a metric?
When you ask an AI tool for a metric, it writes its best guess at a query, and your database runs that guess exactly as written. Your database holds raw rows about customers and subscriptions, and every metric, retention included, gets rebuilt from those rows each time someone asks. The query is where the tool’s guesses get locked in. It sets which tables to join and the grain, meaning what one row stands for when you count, such as a customer or an account.
It also decides which rows drop out and where the date range starts and ends. Change any one of those choices and the number changes, even when the definition stays the same.
That’s why a definition alone can’t guarantee a correct answer. A definition is a sentence a tool reads. The query is what the tool writes after reading it, and each tool’s translation carries its own choices.
Isn’t defining the metric once enough?
Defining a metric once is the right first step, and it still leaves each tool to compute that metric separately. As one analyst put it, “An AI agent cannot function reliably if it encounters multiple definitions for the same metric” (Futurum, October 2025). That’s true, and it raises a second problem: access. A definition in a skills file only helps the agent that loads it, and the BI tool down the hall may never see it. Every tool that can’t read the shared definition rebuilds its own version, so the company has one definition on paper and several in practice. Even a tool that can read it still has to turn it into a query against messy data.
BIRD-Interact, a public benchmark for AI on business databases, shows how. One of its databases records patient visits and the treatments given at each visit. The question asks what share of each doctor’s patients take their medication as prescribed. The rate needs both tables, so a tool joins them.
A visit with three treatments then gets counted three times. Counting at the visit level keeps each visit worth one, but only if every tool knows to do it. Of the 351 visits, 123 carry more than one treatment, and 42 of the 157 doctors’ rates came out wrong.
Should you standardize on one AI tool?
You don’t need to, as long as your metrics are defined in a universal semantic layer. Define retention once in the semantic model, and it no longer matters which LLM or tool points to it, because every one of them gets the same number.
That flexibility matters, because AI changes faster than any company standard can keep up with. In Menlo Ventures’ 2025 survey of enterprise builders, when one provider released a new model, the previous version’s share of users fell from 83% to 16% within a month. In a 2026 Dataiku and Harris Poll survey of 600 enterprise CIOs, 55% said they’d already switched LLMs at least once. People also pick their own surface anyway. One large biotech planned 20,000 seats for one AI assistant, and more than a year later its employees were also using a different one (Tom’s Guide, citing Bloomberg).
With metrics in a universal semantic layer, adding a model or tool means connecting it to the layer, and the definitions stay where they are. Teams keep the freedom to choose, and the numbers stay consistent.
Where should AI calculate business metrics?
Business metrics should be calculated in a universal semantic layer. AtScale ACE, the AI Computation Engine, does the calculating in the AtScale semantic layer. The AI tool asks ACE for retention by name. ACE turns the definition into one tested query, runs it and returns the number. The copilot and the agent send their questions to the same engine, so the same question gets the same answer whichever tool asks. A new AI tool joins the stack without checking against the others, and a definition change lands once for every tool.
The difference shows up in AI accuracy. AtScale ran seven models from six vendors on the BIRD-Interact benchmark. In those preliminary October 2026 runs, every model scored 72% to 137% higher on the 410 business questions when it got its numbers from AtScale ([ebook landing page link]). With AtScale, each model’s score is its best of three runs, and without it, each model ran once.
Before you add the next AI tool, ask where its numbers come from
If the answer is that it writes its own query, you’ve defined the metric once and you’re about to compute it one more time. Count how many places your most important number gets computed today. That count is how many versions of it your leadership could see.
Each of those places also pays to rediscover the metric. When a Tier 1 bank ran five everyday analyst questions through Google BigQuery, the queries the AI wrote on its own cost $17.93 and scanned 3.15 terabytes. That’s nearly $9 million per year, or as we like to call it, the AI rediscovery tax. Through a semantic layer, the same five questions cost less than a tenth of a cent.
Let people choose the AI tools they like. Define it once, compute it once to get the same answer everywhere.
Read AtScale’s BIRD-Interact benchmark ebook for the full AI accuracy benchmark data and learn the five places AI gets business questions wrong.