Working Note · Method

Validating AI-Assisted Quantitative Analysis: Blind Execution and Answer Recovery as Minimal Controls

Tommaso Marson

Working note · 2026

1Introduction

A quantitative result carries no intrinsic indication of its own correctness. A number can be precise, reproducible, and wrong at once: precision and reproducibility are properties of the arithmetic, whereas correctness depends on whether that arithmetic was applied to the appropriate data under the appropriate assumptions. The properties are independent, yet the first are routinely read as evidence of the second.

Automated and AI-assisted pipelines widen the gap. Such systems produce output that is fluent and internally consistent by construction — formatted, complete, stable across repetitions — and none of these characteristics is diagnostic of correctness. Incorrect analyses rarely fail conspicuously; more often they return a plausible result that survives inspection precisely because it conforms to expectation. Large language models compound the problem: conditioned on an expected outcome, they tend to reproduce it fluently, a behaviour advantageous for text generation and hazardous for computation [Ji et al., 2023].

This note specifies a minimal protocol for establishing whether a computational result may be trusted. It comprises two independent controls — blind execution and answer recovery — and two governance conditions. The protocol is method-agnostic and stated at the level of principle: it prescribes what must be demonstrated, not the implementation by which a particular pipeline demonstrates it.

2The failure mode

Two error classes must be distinguished.

Expectation-confirming error. When the agent constructing an analysis is aware of the result it is expected to yield, the procedure may be adjusted — through choices of filtering, coding, or specification — until the output conforms, at which point construction halts. The analysis then passes review because it matches the anticipated result. The error is not detectable by inspection, because inspection is the mechanism it satisfies.

Implementation error. Independently of expectation, a procedure may contain defects — sign inversions, mishandled boundary cases, incorrect indexing — that a good-faith construction carries unnoticed. These are unrelated to what the analyst expects and are therefore invisible to any control targeting expectation alone.

The two classes are disjoint in origin and require distinct controls.

3Control I: Blind execution

Blind execution withholds the expected result from the agent constructing the analysis, whether human or automated. Because the target value is unavailable during construction, the analysis cannot be steered toward it; the result must emerge from the procedure or not at all. The technique is long established in experimental physics, where analysts finalize procedures on data from which the result of interest has been concealed, specifically to prevent the desired answer from influencing analytical choices [Klein & Roodman, 2005].

The condition extends directly to AI-assisted construction: a model executed without the expected result in its context has no target to reproduce. Blind execution addresses expectation-confirming error in full and implementation error not at all.

4Control II: Answer recovery

Answer recovery requires the procedure to reproduce inputs that are known by construction. Parameters are drawn from a prespecified domain; synthetic data are generated from those parameters; the data are processed through the identical procedure intended for empirical data; and the recovered estimates are compared against the generating parameters under a tolerance fixed prior to inspection.

Formally, a procedure M passes recovery if, for parameters θ drawn from a prespecified domain Θ and data D ∼ g(θ) generated under the assumed model g, the estimates satisfy

across independent replications, where the metric d and tolerance τ are specified a priori.

A procedure that fails to recover a known θ cannot be relied upon for an unknown one. A procedure that recovers θ across replications — including conditions constructed to be adversarial — establishes a bounded property: correct estimation on problems of the tested structure. This is a narrower claim than correctness in general and a substantially stronger one than apparent correctness. The approach is standard in simulation-based method validation [Talts et al., 2018].

5Complementarity of the controls

The controls are individually insufficient and jointly sufficient across the two error classes.

Blind execution removes the channel through which results are fitted to expectation but cannot detect a defect in an impartially constructed procedure. Answer recovery detects such defects but does not, by itself, prevent an analyst from selecting post hoc the synthetic conditions under which recovery succeeds — a vulnerability addressed by the governance conditions of Section 6. Applied together, the controls relocate the basis of trust from the appearance of the output to a demonstrated property of the procedure.

6Governance conditions

Two conditions are required for the controls to remain informative.

A priori tolerance. The acceptance tolerance τ and the evaluation domain Θ must be fixed and recorded before results are observed. A tolerance established after inspection can be enlarged to accommodate an observed discrepancy, which nullifies the test. This is the method-validation analogue of pre-registration in empirical research [Nosek et al., 2018].

Public retraction. A published result later found to be incorrect must be retracted explicitly: the superseded value is identified and withdrawn rather than left in circulation. A regime that only ever ratifies prior results does not constitute validation. Retaining a validation record with each reported quantity, and withdrawing it upon supersession, preserves the correspondence between claim and evidence over time.

7Illustrative application

The protocol is illustrated by one case from applied pricing research, stated at the level of the reported result rather than the measurement or estimation procedure, which is immaterial to the validation argument; the figures cited are either already public or reported here specifically as retracted.

A pricing study of a consumer-electronics product returned an initial single-figure threshold of approximately €129, precise and stable under re-estimation. Applying the protocol changed its status. Examination established that the figure was an average over a population made of two distinct groups — the midpoint between them, a value at which essentially no individual buyer actually transitioned — and it was retired as a reported quantity. It is cited here only as the estimate that was withdrawn.

The two underlying group thresholds, near €149 and near €101, were then evaluated by answer recovery: synthetic data generated at known thresholds were processed through the same procedure, and the recovered values met a tolerance fixed in advance. An independent instrument — a separate, direct elicitation of stated prices — corroborated that the two groups were genuinely distinct, their stated valuations differing by roughly €105 versus €160 on a matched question. That corroboration is evidence the division is a property of the population rather than an artefact of the analysis. The reported quantities are the two group thresholds; the single aggregate figure remains retired.

The case exhibits each element of the protocol: a precise, reproducible, and misleading headline estimate; retirement of a superseded number in the open; answer recovery of the quantities actually reported; and an independent check that the reported structure is real.

8Scope and limitations

The protocol establishes conditional, bounded reliability, not certainty. A passing result certifies that a specified quantity, produced by a specified procedure, recovered known answers under blind conditions within a predetermined tolerance. Reliability attaches to the quantity and the procedure — not to the tool that executed them — and lapses when the quantity is superseded.

The protocol does not adjudicate the appropriateness of the underlying model. A procedure may recover parameters correctly under an assumed data-generating process that does not hold empirically; recovery validates estimation, not specification. Nor does it bound error outside the tested domain Θ: extrapolation beyond the structures exercised by the synthetic data is uncertified. These limits are intrinsic, and they delimit the claim the protocol supports — correct estimation on problems of demonstrated structure, established independently of the analyst's expectation.

9Conclusion

The reliability of a computational result cannot be inferred from its precision, its reproducibility, or the fluency of the system that produced it. It can be established, conditionally, by requiring that the procedure be constructed without sight of the expected answer and that it recover known answers within a tolerance fixed in advance. For AI-assisted analysis — where fluent, confident output is the default and expectation-conformity is a documented failure mode — these controls are not supplementary but constitutive of any defensible claim. The pertinent question for a result produced automatically is not the speed of its production, but whether its procedure can recover what is already known.