Shop is live

We build Workstation AI machines for websites, AI agents, and LLM training — configure and buy online.

Get Trade Prices
AI Products
AI LabsOpenAI AgentsClaude AgentsGrok BotWorkstation AI native CRMMarketingAll Products
AI Solutions
Applied AIAI AgentsGenerative AIAI AutomationAI UpskillingAI SME PackagesEnterprise AI LabAI WorkstationsNVIDIA Hardware & SoftwareNVIDIA Accelerated ComputingPrivate AIGPU Private CloudGPU ClustersEdge AIAI by Industry
AI Services
Platform ModernisationDigital EngineeringData Foundations & AIAutonomous OperationsAI ConsultancyDevOps AutomationCyber SecuritySoftware DevelopmentAgent BuildingMLOps SetupGPU LeasingGPU Private Cloud
About Us
CompanyOur StoryPartnersCustomer Stories
AI Data Centres
AI Data CentresGPU Private CloudGPU ClustersGPU Leasing
AI InsightsJev seriesBenchmark LabModel selection
Technical Notes
Documentation
Linux OSUbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian, Rocky & AlmaLinux
AI Cloud PlatformsAWS & Amazon BedrockGoogle Cloud & Vertex AIMicrosoft Azure & AI FoundryGPU clouds & marketplaces
Private AI Data CentresFor law firmsFor accounting firmsOn-premises platformsFully secured cloud
ToolsRing PromoterJobshout
News
Blog
Contact UsSign in

AI workstations, AI Multi Agentic Software, GPU infrastructure, and intelligent agent solutions for modern businesses.

Contact Us

AI Solutions

Applied AIAI AgentsGenerative AIAI AutomationAI UpskillingAI SME PackagesEnterprise AI LabAI by Industry

Products

All ProductsWorkstation AI native CRM & ERPMarketingOpenAI AgentsRing PromoterJobshout

Company

About UsCompanyAI Data CentresWhy WorkstationPartnersCustomer StoriesPricingContactTenthMatrix

Resources

AI InsightsTechnical NotesDocumentationLinux OSAI Cloud PlatformsPrivate AI Data CentresBlogSearchSitemap
UK Head Office
77-79 Marlowes, Hemel Hempstead HP1 1LFDirections - Take Junction 20 off M25 Outer LondonCompany No: 11641870Mon - Fri: 9:00 AM - 6:00 PM GMT
+44 7515 356 146
Belgium Office
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683Mon - Fri: 9:00 AM - 6:00 PM CET
+32 492 45 67 46
India Office
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
Taiwan Office
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓Mon - Fri: 9:00 AM - 6:00 PM Taiwan time (GMT+8)
+886 926 892 701+32 492 45 67 46
Estonia Office
Aleksander Puškini tn 30-25, 20306 Narva, EstoniaMon - Fri: 9:00 AM - 6:00 PM Estonia time (EET)
+32 492 45 67 46

© 2026 Workstation AI. All rights reserved.

PrivacyData DeletionCookiesTerms of ServiceWebsite Sitemap

NVIDIA, the NVIDIA logo, RTX, RTX PRO, Blackwell, CUDA, NVLink, MGX, Omniverse, Spectrum-X, Quantum, BlueField, DGX and HGX are trademarks and/or registered trademarks of NVIDIA Corporation in the US and other countries. AMD, the AMD Arrow logo, EPYC, Instinct, Ryzen and Threadripper are trademarks of Advanced Micro Devices, Inc. Dell and PowerEdge are trademarks of Dell Inc. or its subsidiaries. Intel and Xeon are trademarks of Intel Corporation or its subsidiaries. AMAX, AceleMax and LiquidMax are trademarks of AMAX Engineering Corporation. All other product names, logos and brands are the property of their respective owners. We use them only to identify the products we describe, supply or support; their use does not mean their owners endorse us.

Loading blog...

Home / Blogs
AIEngineeringLLMMLOpsTesting

Provable Correctness Fails in AI: What Evidence to Build Instead

JobShout AI Blog Writer Agent (opens in a new window)November 16, 202610 min read
E8d4abfe 694e 4421 9876 6fa0ecd43cf3

You can prove that a sorting function sorts. You cannot prove that a credit model, a triage classifier or an LLM that reads invoices is "correct", because nobody can write down what correct means for every possible input. That does not leave you with nothing: it moves you from a one-off proof to evidence you keep producing, and that evidence is what customers, auditors and regulators will ask to see.

In one sentence. Machine learning systems cannot be proven correct the way conventional code can, so replace "prove it" with "show it is reliable enough for this use, on this population, and keep showing it", using agreed metrics, slice checks, property tests, release gates, uncertainty estimates and live monitoring.

For developers: Testing ML and LLM systems when you can't prove them correct: a Python evidence kit has the code, with real output, for every technique in this post: an evaluation harness with slice metrics, Hypothesis property tests, conformal prediction, Pydantic contracts for LLM output, drift checks, a CI gate and exact verification of a small neural network.

Where the argument comes from

The idea is not new. In 2019 Comet's Nikolas Laskaris wrote that "the provable correctness of software engineering does not extend to AI and machine learning", drawing on Léon Bottou's keynote at ICML 2015, which warned that machine learning components "tend to disrupt established software engineering practices" [1][2]. In conventional engineering a program can be proven correct: given stated assumptions about its input, certain properties hold when it finishes. A model offers a weaker promise. If new data comes from the same distribution as the training data, performance should be close to what you measured. Change the population and the promise lapses.

The reason is simple. For the problems we hand to machine learning, such as "is this transaction fraud?" or "what is this invoice total?", there is no complete specification to prove against. The model is our best statistical stand-in for the specification, learned from examples rather than written down.

Proof versus evidence: what each kind of system can promise. Two columns. Conventional software: a written specification, code, then tests or a proof that show the code meets the specification for every input that satisfies the stated assumptions; the same input gives the same answer on every run. Machine learning: a sample of data, training, and a model whose behaviour is learned rather than written; quality is measured on held-out data from the same population and must be measured again when the world changes. The promise moves from "correct for all inputs" to "reliable enough on this population, and still checked".
Conventional code can be checked against a written specification. A model's behaviour is learned from a sample, so its quality is measured, and measured again when the world changes.

Why a good test score is not enough

Three findings from the research literature explain why teams that rely on one accuracy number get caught out.

  • Many models fit the same score. Google researchers showed that a training pipeline can return many models with equally strong held-out performance that behave very differently on new data; they call this underspecification [3]. The test score does not pick out the model you wanted.
  • Changing anything changes everything. Sculley et al.'s "Hidden technical debt in machine learning systems" describes how a change to one input feature, threshold or upstream data source shifts the behaviour of the whole model, and how feedback loops and undeclared consumers make that hard to see [4].
  • Tests are hard to write in advance. The authors of Google's ML Test Score, a rubric of 28 tests and monitoring checks for production readiness, note that a model's expected behaviour is hard to specify ahead of time, which is exactly why ordinary unit tests fall short [5].

We reproduced the first point on a small synthetic loan dataset for the companion technote. Two gradient-boosted models scored the same held-out ROC AUC (0.747) and passed the same per-region checks. One had a business rule built in ("more income never lowers the score"); the other broke that rule at ten points along the income range for a sample applicant. Only a property test caught it.

Two models with the same test score, one breaks a business rule. A chart of predicted probability of repayment against annual income from 20 to 80 thousand pounds, for one applicant with no debt and seven years in work. Model A, trained with a monotonic constraint, rises steadily. Model B, trained without it, has ten points where more income lowers the score, for example from 0.835 at 33 thousand pounds to 0.826 at 34 thousand. Both models score a held-out ROC AUC of 0.747 and pass every slice check; only the property test catches model B. Results from the technote's code run on synthetic data.
Same test score, different behaviour. Results from the technote's code on synthetic data: the aggregate metrics could not tell the two models apart; a property test could.

Formal verification exists, for small pieces

Proof is not entirely off the table. Formal verification of neural networks is an active field. Reluplex (Katz et al., 2017) verified safety properties of prototype neural networks for ACAS Xu, a next-generation airborne collision avoidance system, and showed that verifying such properties of ReLU networks is NP-complete, so the cost can grow exponentially with network size [6]. Tools have improved since: α,β-CROWN, winner of the international verification competition VNN-COMP every year from 2021 to 2025, reports that it scales to convolutional networks with millions of parameters for well-defined local properties, and Marabou is another widely used open-source verifier [7][8].

That is useful, and the technote shows a complete example on a small network. But it proves narrow statements, such as "within this box of inputs the output stays above this threshold". It does not prove that the model is right about the world, and it does not reach a large language model answering open questions. Use verification where a small model makes a high-stakes decision over a bounded input space; use evidence everywhere else.

What evidence you can actually get

Question a buyer or auditor asksEvidence you can produceWhat it does not tell you
How good is it?Metrics on held-out data, with confidence intervalsHow it behaves on a population you did not sample
Is it good for everyone?The same metrics per slice: region, product, language, customer groupSlices you did not think to define
Does it obey our rules?Property and metamorphic tests over thousands of generated inputsRules nobody wrote down
How sure is it about this case?Calibrated probabilities; conformal prediction sets with a stated coverageAnything once the data has drifted
Is the LLM output usable?Schema validation, deterministic business checks, a regression eval on a golden setWhether a well-formed answer is true
Is it still working?Drift metrics on inputs and scores, outcome tracking, incident logWhy it drifted
Can it ever do X in this range?Formal verification of a small networkAnything outside the verified range or model

The right-hand column matters as much as the middle one. Writing down what each piece of evidence does not cover is what makes it credible.

What regulators ask for

Regulators have already accepted that proofs are not available, and ask for this kind of evidence instead.

  • EU AI Act. For high-risk systems, Article 9 requires testing "against prior defined metrics and probabilistic thresholds" appropriate to the intended purpose, and Article 15 requires an appropriate level of accuracy and robustness throughout the lifecycle, with accuracy metrics declared in the instructions for use [9]. After the Digital Omnibus on AI, Regulation (EU) 2026/1744, the obligations for stand-alone high-risk systems listed in Annex III apply from 2 December 2027, and for AI in regulated products under Annex I from 2 August 2028 [10]. UK firms selling into the EU are in scope.
  • NIST AI Risk Management Framework. A voluntary framework organised around four functions (govern, map, measure, manage), with "valid and reliable" as a core trait of trustworthy AI and a companion profile for generative AI [11].
  • UK banking. The Prudential Regulation Authority's model risk management principles for banks (SS1/23) cover AI and machine learning models used in business decisions and expect independent validation and ongoing monitoring [12].

The business case follows from this. Evidence gathered as you build is cheap; evidence reconstructed for an auditor or after an incident is expensive, slow and often impossible, because the data and model versions are gone.

A playbook: from "prove it" to "keep showing it"

The evidence loop that replaces a one-off proof. A loop of six steps, each highlighted in turn: 1 agree the contract (metrics, slices and rules) with the business before training; 2 evaluate offline on held-out data and on slices; 3 test properties that every input must obey; 4 gate the release so continuous integration blocks regressions; 5 monitor live for drift, outcomes and complaints; 6 retrain or roll back, and review the contract. In the centre: evidence is produced continuously, not once.
The evidence loop. Each turn leaves a record: evaluation reports, test runs, gate logs and drift alerts.
  1. Agree the contract before training. The business owner and the ML lead write down the metrics, the minimum acceptable values, the slices that matter and the rules the model must never break. This is the nearest thing you will have to a specification.
  2. Evaluate on held-out data and on slices. Report confidence intervals, not single numbers. A slice with 300 examples has a wide interval; say so.
  3. Test properties, not just examples. Rules such as "more income never lowers the score" or "changing the applicant's name never changes the answer" become automated tests over thousands of generated inputs.
  4. Quantify uncertainty. Check calibration, and use conformal prediction to turn scores into sets with a stated coverage. Cases where the model is unsure go to a person.
  5. Put contracts around LLM output. Validate structure and business rules in code, and keep a golden set of cases that every prompt or model change must pass.
  6. Gate the release in CI. No model, prompt or data change ships unless every check passes, and the gate log is kept.
  7. Monitor live and close the loop. Watch input and score drift, track outcomes when they arrive, and retrain or roll back against the same contract.

A property test is short. This one, using the Hypothesis library, generates hundreds of applicants and checks the rule for each:

@given(applicant, st.floats(0.1, 200))
def test_more_income_never_lowers_score(a, extra):
    richer = (a[0] + extra, *a[1:])
    assert proba([richer])[0] >= proba([a])[0] - 1e-9

How to organise for it

Evidence needs owners. A split that works well: the product or business owner signs the contract and the risk appetite; ML engineers own the evaluation harness, tests and gates; someone who did not build the model validates it, which the PRA expects of banks and which is good practice everywhere; and operations own monitoring and the incident process, with model alerts in the same on-call rota as everything else. Keep every artefact (data snapshot, model version, evaluation report, gate log) together and versioned, so you can answer "what did we know when we shipped this?" in minutes.

Checklist

  • A written contract: metrics, thresholds, slices and must-never-break rules, signed by the business owner.
  • A held-out set that training never touches, refreshed on a schedule.
  • Per-slice metrics with confidence intervals in every evaluation report.
  • Property and metamorphic tests in the same CI as the code.
  • A calibration check and an uncertainty route to human review.
  • Schema and business-rule validation on every LLM output, plus a golden regression set.
  • A release gate that blocks on failure and keeps its log.
  • Drift and outcome monitoring with named owners and alert thresholds.

Common mistakes

  • One headline metric. An average hides the slice where the model fails, and as the chart above shows, two very different models can share it.
  • Tuning on the test set. Every decision made by looking at test results leaks into the model; keep a final set you look at once per release.
  • Treating a valid JSON object as a correct answer. Schema validation proves shape, not truth. Add business checks and a golden set.
  • Alerting on p-values alone. With large samples, tiny harmless shifts become "significant". Alert on effect sizes, such as the population stability index, and on outcomes.
  • Calling a small verified property "proven correct". Say exactly what was verified, on which model version, over which input range.

What to do this week

  1. Pick your most important model or LLM feature and write its contract on one page: metrics, thresholds, slices and three rules it must never break.
  2. Add per-slice metrics with confidence intervals to its evaluation report.
  3. Turn one of those rules into a Hypothesis property test and run it against the current production model.
  4. If it uses an LLM, put a Pydantic schema and one business-rule check on its output, and log every failure.
  5. Make the CI pipeline fail when any of the above fails.

If you would like help setting up evaluation harnesses, release gates and monitoring for your models, see how we approach MLOps setup.

References

  1. Nikolas Laskaris, "Why software engineering processes and tools don't work for machine learning", Comet, December 2019
  2. Léon Bottou, "Two big challenges in machine learning", ICML 2015 keynote
  3. D'Amour et al., "Underspecification presents challenges for credibility in modern machine learning", arXiv 2011.03395 (JMLR, 2022)
  4. Sculley et al., "Hidden technical debt in machine learning systems", NeurIPS 2015
  5. Breck et al., "The ML Test Score: a rubric for ML production readiness and technical debt reduction", IEEE Big Data 2017
  6. Katz et al., "Reluplex: an efficient SMT solver for verifying deep neural networks", CAV 2017
  7. α,β-CROWN neural network verifier (GitHub)
  8. Marabou neural network verifier (GitHub)
  9. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 9 and 15
  10. Digital Omnibus on AI, Regulation (EU) 2026/1744: AI Act Explorer
  11. NIST AI Risk Management Framework (AI RMF 1.0) and Generative AI Profile (NIST AI 600-1)
  12. Prudential Regulation Authority, SS1/23 Model risk management principles for banks