Inside a Tier 1 bank’s $9-million-a-year AI rediscovery tax, and the architectural fix that cut compute costs by as much as 21,903x.
Every time an analyst at a Tier 1 bank asks, “What was revenue in the Northeast last quarter?” The LLM has to guess what those words mean to the business. The answers are often expensive. And often wrong.
One bank measured the cost of an LLM guessing at SQL: nearly $9 million a year, letting an LLM attempt to rediscover metrics the business already knew how to compute, and getting the answer wrong about a third of the time.
We call it the AI rediscovery tax. And you should stop paying it.
Your big AI bill is a SQL scanning problem.
One bank’s team ran five common analyst questions through their Google BigQuery warehouse in two ways. The first is the way most companies do it: hand AI your data sources and let it figure them out. Without semantics, AI doesn’t know how Finance defines “revenue,” how Sales defines the “Northeast,” or where the fiscal calendar starts and ends. So it makes up its own version and scans trillions of rows.
For five use cases they measured, guessing cost $17.93 and scanned 3.15 terabytes.
Then they added AtScale. Computation costs dropped to less than a tenth of a cent and read just 144 megabytes. The semantic layer used 21,903 times less warehouse compute.
That 21,903x is the lookup case, where the answer is pre-built. When a question is novel, and the semantic layer has to compute it live, it still wins. It knows the data model and writes the right SQL instead of guessing. The bank’s broader testing showed 95% to 99% less data scanned, even in this live-compute case, still a 20x to 100x cost cut.
AI accuracy problems are semantic problems.
Ask an unguided LLM the same business question twice, and you can get two different answers. Not because the model is bad. Because it has to make up a definition each time. “How many young customers do we have?” One LLM pass picks ages 25-34; the next picks 26-35. Neither matches what Marketing actually uses.
The semantic layer holds the definitions in one place. Every query for “young customers” uses Marketing’s bracket. Every quarter matches Finance’s. Every customer count ties back to the customer master. The model can’t invent the wrong number when the right one is already defined. AI accuracy on curated metrics goes from an industry-average 70% to 100%.
The path to 100% accuracy runs through semantics, not tokens.
Model. Measure. Compute. Repeat.
The answer is a flywheel with four phases. Model your semantics, including carefully constructed computations of your enterprise metrics. Measure that they work by LLM, with vague prompts. Compute them right at runtime when anyone asks questions. Then repeat so your layer never drifts.
The bank employed the AtScale enterprise flywheel. They handle more than 6,000 AI-driven questions a day with precise accuracy. They projected that the AtScale layer saves nearly $9 million a year in compute for the five questions they measured and avoided thousands of wrong answers that never impact a business decision.
They stopped paying the rediscovery tax with a single move: putting their definitions into a semantic layer. They didn’t swap their LLM. They swapped the layer under it.
Are you guessing?
Ask your data team three questions to see if your LLM is guessing:
- Are our business metrics defined in one place where AI can use them?
- When AI runs a query, does it have a defined, repeatable, measured computation layer that’s been carefully designed to compute answers?
- Can your measure our AI accuracy and cost? Have we measured multiple LLMs to understand which one is the most cost effective?
If any of those answers are “no,” you’re paying the guessing tax. The bank that measured it stopped paying it the day they put their metrics in a semantic layer, and the bill dropped.
>> Download the complete benchmark report, with full methodology and per-query SQL.
Source: production benchmark conducted by the UK commercial banking division of a global Tier 1 bank, measuring five representative analytical queries on BigQuery with and without an AtScale semantic engine in front. The nearly-$9 million annual figure applies the unguided per-query cost ($3.59) to the bank’s reported daily AI-query volume (over 6,000) served through the semantic-layer path; in practice, the unguided architecture would cap query throughput well below that level, so the figure represents the cost the bank would face if it tried to serve the same daily workload through the unguided path. The roughly 650,000 wrong-answer figure applies the public BIRD-SQL benchmark’s ~30% text-to-SQL error rate to that same daily query volume.