Executive summary
Your team makes most AI accuracy fixes for one model. When that model changes, custom training has to be rerun on the new model, and prompts and skills files have to be retested.
- Upgrades are routine. 55% of enterprise CIOs have already switched LLMs, and each switch means your team has to rerun custom training on the new model and retest every prompt and skills file.
- Accuracy can drift even when the model stays the same. Anthropic’s analytics skills slid from about 95% to 65% in a month because the data under them changed.
- Store business definitions in a semantic layer as versioned code that a person reviews, and start with the metric your team corrects most often.
Picture a ledger of every correction your team made to AI this year. “Churn means logo churn.” “The fiscal year starts in February.” “Exclude test accounts.” Each correction began with an analyst catching a wrong number and ended with a platform engineer writing the rule into a prompt, a skills file, or training dataset.
Then your provider upgrades the model. The catch: most corrections your team made to AI this year only work on the model your team made them for. Your team has to retest prompts and skills files and rerun custom training. Adding to the challenge is that model changes are routine: in a 2026 Dataiku and Harris Poll survey of 600 enterprise CIOs, 55% said they’d already switched LLMs at least once.
“You don’t want to store your business memory with a particular model,” says Dave Mariani, AtScale’s founder and CTO.
New model, new problems
Custom training has to be rerun on the new base model when that model retires. Prompts and skills files carry over but have to be retested, because a new model can interpret them differently. Unless you store your improvements where every model can read them, that upgrade creates inaccuracy and lots of additional work to fix it.
That’s the problem at the center of the upcoming second episode of The AI Accuracy Files, our series on AI accuracy and semantic layers. Dave Mariani, AtScale’s CTO and co-founder, CMO Mark Palmer, and Ryan Siegler, Head of Developer Relations, will talk through why a model upgrade can reopen fixes your team thought were done, and where those fixes should live to outlast model changes or updates. We collected some of their thoughts for a sneak preview.
What happens to your prompts, skills files, and custom training when you upgrade an LLM?
Prompts and skills files carry over, but your team tested them on the old model, so some keep working and some break without telling anyone. Custom training doesn’t carry over at all, so you have to rerun it on the new base model. Without regression tests, teams often find out what broke from a confidently wrong answer.
The clock moves faster than most teams can plan for. In Menlo Ventures’ 2025 survey of enterprise builders, when one provider released a new model, the previous version’s share of users fell from 83% to 16% versus the newcomer within just one month.
Providers publish retirement schedules with as little as 60 days’ notice for generally available models, and a fine-tune only works the base model you trained it on. A model can even shift while you’re standing still: one Stanford and Berkeley study found the same model’s accuracy on a task fell from 84% to 51% in three months.
Why does AI accuracy drift even when nothing changes?
Data and definitions keep moving, and fixes stay where you left them. Anthropic’s own data team watched its hand-written analytics skills slide from near 95% accurate to about 65% in a month, because the data underneath them changed while the model stayed put. (Check out what that means for semantic layers in our response to their post.)
That’s why prompt engineering is renting accuracy. You pay again whenever something shifts, whether it’s the model, a table, or the finance team’s definition of an active customer.
Where should AI business definitions live?
Definitions should live in a semantic layer, as versioned code that every model reads, so a fix outlasts the model it was made for. The AtScale AI Accuracy Flywheel keeps them there.
It’s a four-step loop that turns every AI miss into a permanent fix: CoModel defines what the business means, Ava clarifies vague questions, ACE computes the answer, and Canary tests the result. Each step hands the next one something it couldn’t produce on its own, and the last step feeds the first, so accuracy only moves in one direction.
Let’s take the flywheel for a spin, using an example of a metric called “churn.”
- Define it once. A data engineer, working with CoModel, writes churn into the semantic model. The change ships like code, with a pull request, a reviewer, version history, and a way back. “The semantic models are code… that pull request has to be reviewed by a human,” Dave says.
- Clarify it. When someone asks a vague question (churn of what, over which period?), Ava asks which one they meant before anything computes, then saves the answer as a rule for next time.
- Compute it. ACE writes the SQL and computes churn the same way every time, no matter which model asked for it.
- Test it. Canary checks the definition on every run, so a regression shows up in a test instead of a board deck.
Does the flywheel increase AI accuracy?
“It’s like a ratchet,” says AtScale’s Mark Palmer. “Every time you turn it, you get closer and closer to 100%.”
We saw this play out on BIRD-Interact, the text-to-SQL benchmark we ran against 22 messy PostgreSQL databases. One question asked, “How reliable is this equipment?” The answer depended on a Composite Reliability Index, a formula that lived in a document instead of the database, which left the model to work out the math on its own. A data engineer added the index to the semantic layer as a metric. ACE now computes it the same way every time, and every later run checks it. The business ensures that the change outlasts any model upgrade thanks to its semantic layer.
Does a semantic layer keep accuracy when you change models?
In our BIRD-Interact runs, it did. We cleared the layer’s memory, then ran BIRD-Interact’s 410 business questions three times. We also built runs 2 and 3 to use what the first run learned. The layer kept the query shapes it had verified as correct, then provided matching shapes to the model when a similar question came in.
Across the three runs (preliminary results from AtScale’s October 2026 runs, not yet verified by the BIRD team):
- Management tasks, which the layer never touches, stayed flat, so the gain came from memory rather than noise.
- Gains held on both a frontier model and an open-weight one. On the 410 business questions, Opus 4.6 rose from 45.4% on the cold run to 53.7% at its best of three runs, and Kimi K2.5 rose from 41.7% to 49.3%.
The freedom to upgrade your model, without adding extra work
When your fixes live in the semantic layer, a model upgrade doesn’t put them back on the table. Your team decides “churn” once, a person approves the change, and every model, BI tool, and agent uses the same answer. The new model reads the same definitions, rules, and tests as the old one, so you can swap models on price or policy without redoing a year of work.
“You can’t do it piecemeal,” Dave says. The goal is every business definition in one layer, but the first step is small: pick the metric your team corrects most often and move it out of the prompt before the next upgrade lands.
Keep a lookout for the second episode of The AI Accuracy Files, coming soon! In the meantime, check out the the full benchmark results, or book time with our team to discuss your AI strategy.