Input
Case file
The documents of the case: contracts, minutes, email and supporting records.
← Back to Governance
AI / GOVERNANCE
THE MODEL PROPOSES
THE RULE DECIDES
THE RUN IS WRITTEN DOWN
The deterministic engine that questions the probabilistic one
A system that answers differently every time cannot be validated. A rule the business wrote, versioned and tested can.
THE SAME REQUEST, SEVERAL TIMES
The request: should this loan be disbursed?
The file is missing the registered collateral.
Without a deterministic engine
The AI proposal becomes the result.
3 different answers
4 errors · last 6 runs
Input
Loan file
Extraction
Amount, income, collateral
Analysis · Generative AI
Proposes: Disburse
Empty
Result · Run 6
Disburse
Contradicts the policy
With a deterministic engine
The same AI proposes. The rule decides.
1 answer
0 errors · last 6 runs
Input
Loan file
Extraction
Amount, income, collateral
Analysis · Generative AI
Proposes: Disburse
Rules · Deterministic engine
Registered collateral? No
Overrode the AI proposal
Result · Run 6
Do not disburse
Rule: collateral not registered
Illustration: the runs and counts are simulated. The measured variation is in the quote.
At temperature zero, the setting that is supposed to be deterministic, the same request run 1,000 times produced 80 different answers.
AI proposes. Control decides. And control is deterministic.
José ÑáñezTechnology Advisor · Board Member
Published September 29, 2026Updated October 1, 20266 min read
WHAT THE MODEL CANNOT GIVE
Variability is not a defect. It is a property.
A large language model (LLM) does not answer the same way twice, even when asked to. The Thinking Machines Lab measurement at the top of this piece has a concrete cause: it lies in how the server batches requests, not in sampling randomness.
In decisions with consequence the effect is visible. Landon and colleagues (Journal of General Internal Medicine, 2025) asked six models the same clinical question five times: some changed the recommendation in two of the five runs. Atil and colleagues (2025), across five models and eight tasks in a "deterministic" configuration, found accuracy varying by up to 15 points between runs.
A better prompt reduces the problem; it does not remove it. For a regulated institution that means the model’s consistency cannot be the control. The control has to sit outside the model.
2 of 5
runs in which some models changed a clinical recommendation
Landon et al., J Gen Intern Med, 2025 · 6 models, 4 cases, 5 repeats
15 points
of accuracy variation between "deterministic" runs
Atil et al., 2025 · 5 models, 8 tasks, 10 runs
WHAT THE REGULATOR ASKS FOR
Nobody prohibits the technology. Everybody demands evidence.
In US insurance the reference is the NAIC Model Bulletin on the use of artificial intelligence systems by insurers (December 2023), adopted by 25 states and the District of Columbia as of August 31, 2026. It prohibits no model. It requires a written AI governance program, controls so that systems do not produce inaccurate, arbitrary, capricious or unfairly discriminatory outcomes, documentation of data, validation and mitigation, and full responsibility for what vendors do.
New York went further in Circular Letter No. 7 of 2024 from the Department of Financial Services: an insurer should not use an AI system in underwriting or pricing until it has determined that the system does not discriminate; testing happens before production and on a regular cadence afterwards; and an adverse-action notice must disclose whether AI was involved. In March 2026 the NAIC put an evaluation tool into pilot with twelve states, for use in market-conduct examinations.
In Latin America the most direct reference is in Peru. The Model Risk Management Regulation of the Superintendency of Banking, Insurance and Pension Funds (SBS Resolution No. 00053-2023, January 2023) requires the models institutions use to manage risk to be validated before their use is approved, by staff independent from those who developed them; and for models built with artificial intelligence techniques it requires, at a minimum, tests that assess their interpretability.
The rule is about governance and evidence, not technology. What nobody yet knows how to do well is validate a system that answers the same question differently.
The examiner cannot audit a model. The examiner can audit a rule that fired, with the input cited.
THE ARCHITECTURE
Two engines, and a clear boundary between them
The model does what only it can do: read documents and language, and propose. The rule does what the examiner needs: decide the same way every time and leave a trace.
What the engine produced is copied. It is not rewritten.
Extraction
Model
Reads the documents and pulls out the data the rule needs, citing where each item came from. A data point that is in no document is reported as missing, not filled in.
Analysis
Model
Summarizes, interprets and proposes. Its proposal is an input to the rule, not the decision.
Rules
Rule
Applies the arithmetic and thresholds the business defined in writing, and decides. Same input, same output, every time.
Result
Rule
Delivers the decision with its reason (which rule fired and on what data) and writes down the run: when, on what, what it found, how long it took, what it could not read.
What the examiner receives
01
The rule that fired, versioned.
02
The data point cited, with its source and date.
03
The record of the run.
04
The declared coverage: what the system checks and what it does not.
WHERE IT APPLIES
The same rule, in three industries
In all three the question is the same: does the document the process requires exist, does it say what it should, and does it agree with the others? That question needs no probability.
Insurance
Claim paid with no adjuster’s report on file; policy issued without the verification the policy requires; adverse decision without the human review state law demands.
Banking
Active customer with no documented source of funds; loan disbursed with no collateral registered; exception approved without the authority the policy assigns.
Project management
Milestone declared met with no acceptance record; budget spent above what was approved; vendor invoicing what has not been accepted.
THE LIMITS
What the deterministic engine does not fix
The engine can only judge what it can compute, and its inputs are produced by the model. If the model misreads an amount, the rule computes correctly on a false figure. Three things contain that risk: the citation per data point, which lets the reading be checked in seconds; the cross-check between documents that cannot both be true; and measuring the variability itself, by reading N times and reporting the agreement, which is the figure nobody hands the regulator today.
Rules have finite reach: what is not written as a rule is not seen. That is not a defect if the system declares its coverage. It is a defect when coverage is presented as total.
And the engine does not replace the accountable person. It decides on what it was given; the person still answers for what is done with the decision.
THE THESIS
A regulated institution cannot demand consistency from a model. It can demand that every material decision pass through rules the business wrote, versioned and tested, and that every run be written down.
That turns an impossible requirement into one that can be met, and it produces the evidence the bulletin asks for: the rule, the citation, the run and the coverage.
José Ñáñez
SOURCES
Regulation and measurements, as published.
NAIC — Implementation of the Model Bulletin on AI systems (status as of August 31, 2026)Adoption map: 25 states and the District of Columbia; four more states with insurance-specific AI regulation.→New York DFS — Insurance Circular Letter No. 7 (July 11, 2024)Use of AI systems and external consumer data in underwriting and pricing.→Peru SBS — Model Risk Management Regulation, SBS Resolution No. 00053-2023 (January 6, 2023)Initial validation before use is approved, validators independent from developers, and interpretability tests for models built with artificial intelligence techniques.→NAIC — AI Systems Evaluation Tool, pilot project summary (2026)Pilot from March to September 2026 with twelve states.→Thinking Machines Lab — Defeating Nondeterminism in LLM Inference (September 2025)1,000 completions at temperature zero; 80 unique.→Landon et al. — Variation in LLM Recommendations in Challenging Inpatient Management Scenarios (J Gen Intern Med, 2025)Six models, four vignettes, five identical prompts each.→Atil et al. — Non-Determinism of "Deterministic" LLM Settings (2025)Five models, eight tasks, ten runs each.→