--- title: "The Model That Explains the Past Perfectly: Backtest Overfitting in Institutional Decision-Making" description: "Backtest overfitting occurs when a model, tuned progressively closer to historical data, absorbs that sample's non-recurring variation into its parameters and fails once conditions shift. Its institutional form appears in bid pricing curves, demand forecasts, inventory policy and covenant calibration. The neutralizing mechanism is architectural rather than personal: a log of specifications tried, a locked holdout sample, and separation of the building and validating roles." url: https://www.beirek.com/en/blog/backtest-overfitting canonical: https://www.beirek.com/en/blog/backtest-overfitting published: 2025-04-15 modified: 2025-04-15 category: "Judgement & Decision Making" category_url: https://www.beirek.com/en/blog/category/judgement-decision-making language: en-US reading_time_minutes: 9 publisher: BEIREK LLC publisher_url: https://www.beirek.com license: "© BEIREK LLC — citation with attribution and link permitted" keywords: ["backtest overfitting","model risk governance","pricing model calibration","holdout validation","contingency sizing","forecast accuracy diligence"] topics: ["Decision and judgement biases in institutional settings","Model risk and validation architecture","Bid pricing and contingency discipline","Forecasting capability in valuation and diligence"] alternate_language_url: https://www.beirek.com/tr/blog/backtest-overfitting --- # The Model That Explains the Past Perfectly: Backtest Overfitting in Institutional Decision-Making > **In short:** Backtest overfitting occurs when a model, tuned progressively closer to historical data, absorbs that sample's non-recurring variation into its parameters and fails once conditions shift. Its institutional form appears in bid pricing curves, demand forecasts, inventory policy and covenant calibration. The neutralizing mechanism is architectural rather than personal: a log of specifications tried, a locked holdout sample, and separation of the building and validating roles. *How closely a model reproduces historical outcomes carries, on its own, remarkably little information about how accurately it will perform going forward; that first figure becomes interpretable only alongside a second one — the number of specifications tried before the fit was obtained. In institutional decision-making the second figure is almost never requested, and the cost of its absence surfaces not when the model errs, but when conditions change.* --- A recurring pattern surfaces in investment committee sessions and pricing review meetings: confidence in the room rises measurably at the moment a model is shown to reproduce historical outcomes closely. A cost curve that regenerates the last five years of completed tenders within a few points of deviation, or a forecasting function that tracks twelve quarters of demand fluctuation almost exactly, tends to be treated as evidence that closes the discussion rather than as a claim requiring interrogation. Yet one figure is never requested in that same session, and it remains on the analyst's own machine: how many distinct variable sets, how many alternative time windows, and how many weighting schemes were tested before that fit was reached. Knowing how well a model explains the past, taken alone, informs the question of future accuracy far less than is generally assumed; it becomes interpretable only when read together with that second number. The asymmetry is not accidental, arising instead from the fact that institutional review rituals are built to evaluate outputs rather than processes. What enters the presentation file is the final specification together with its historical performance, while the count of discarded specifications and the reasoning behind each discard never enters at all — nobody asked for them, and to the extent they were not asked for, they were never recorded. The decision-maker, believing the exercise to be an assessment of one model's correspondence with history, is in fact assessing the outcome of a selection process in which the best-fitting candidate was drawn from a larger set, and the statistical meaning of those two situations differs materially. The intent of the person preparing the analysis is not the operative variable here; improving calibration is that person's assignment, and pursuing it constitutes professionally correct conduct. The mechanism carries a name — backtest overfitting, the process by which a model, tuned toward closer and closer correspondence with historical data, absorbs into its parameters not only the durable relationships within that data but its incidental, non-recurring variation as well. Every dataset contains two components: a repeating structural relationship, and fluctuation specific to that period which will not repeat. Refining a specification against historical data captures the first component in its early stages; past a certain point, however, the only remaining route to improvement runs through accommodation of the second, because the durable structure available for explanation has been exhausted. The difficulty is that the person working on the model cannot observe the boundary between those two stages; what appears on screen is simply a fit measure improving slightly with each iteration. This tendency is not an error but a fully functional method under specific conditions. Where the underlying mechanism is understood in advance, where the number of specifications tested remains bounded, and where the sample exceeds the parameter count by several orders of magnitude, calibration against historical data genuinely improves accuracy — the whole of engineering practice operates on precisely this logic. The method degrades at the point where the number of specifications tested begins to approach the breadth of the sample; beyond that threshold, selecting the best-performing candidate among many amounts to selecting noise rather than signal. In institutional settings the threshold is crossed quietly, since every additional day spent improving a model increases the count of specifications tested while leaving the sample entirely unchanged. The sample itself is frequently non-neutral, and this second layer attracts even less attention than the first. A cost database assembled from completed projects excludes those that were cancelled, that never reached financial close, or that were restructured through a change of contractor; the model is therefore calibrated to the characteristics of work that concluded, with the conditions producing non-conclusion structurally absent from the dataset. By the same logic, a price elasticity estimate derived from the existing customer portfolio carries no information whatsoever about the behaviour of buyers who are absent from that portfolio precisely because they declined the price level in question. Historical fit may appear flawless under these circumstances, though the source of that flawlessness lies not in the model's accuracy but in the removal of the problem from the dataset. The institutional cost accumulates first within pricing discipline. A cost model overfitted to prior tender outcomes will typically compute the contingency line more narrowly than circumstances warrant, having interpreted the previous period's deviations as systematic structure, internalized them into its parameters, and left very little behind in the form of unexplained variance. A bid submitted with a thin contingency raises the probability of winning, so the model appears to vindicate itself in the short term; the margin profile of the work actually won becomes visible only during execution, through the frequency with which the liquidated damages cap is approached and through the accumulated cost of rework. The same dynamic repeats in the calibration of reorder points within inventory policy, exposure thresholds within credit allocation, and base-case DSCR assumptions within debt structuring. The second site of accumulation lies in capacity and investment decisions. Where the demand model underwriting a facility expansion or a new production line explains prior quarters with high fidelity, an investment committee will tend to carry that fidelity forward into the projection, notwithstanding that a substantial share of the model's explanatory power may derive from features of a period unlikely to recur. The consequence in such cases extends well beyond a forecast that deviates by some margin; the payback schedule of the fixed investment, the offtake volumes committed against it, and the supplier contracts written around those volumes are all locked to a demand curve that no longer materializes. In capital-intensive decisions, model error registers on the balance sheet not as forecasting variance but as idle capacity and as commitments that cannot be renegotiated. A third layer becomes visible in valuation, and it is generally discovered midway through a sale process. A company's forecasting capability is tested in diligence not by inspecting the architecture of its model but by comparing prior forecasts against realized outcomes; in organizations where forecasts were never preserved version by version, and only the most recent iteration was archived, that test simply cannot be run. Forecasting capability that cannot be tested is typically priced by a buyer on the assumption of its absence — the projection-dependent component of value is discounted, deferred into an earn-out structure, or pushed back onto the seller through an expanded representations and warranties package. Model quality at this point ceases to be an engineering matter and becomes a direct component of the closing price. The distribution of the cost across time makes the problem exceptionally easy to conceal. An overfitted model does not fail at the moment it is built; it continues producing plausible outputs for as long as the conditions from which its data was drawn persist, and throughout that interval confidence accumulates, buffers narrow, and the number of decisions anchored to the model grows. The break typically arrives at the moment conditions shift, which is also, and not coincidentally, the point at which the buffer is thinnest. The diagnosis that follows is nearly always identical — the market moved, input prices behaved unpredictably, the period was without precedent — and to the extent that this diagnosis leaves the method intact, it reproduces the same reflex in the subsequent cycle. The tendency is managed not through individual prudence but through an architecture constructed around the model, and that architecture has four components. The first is the specification log: every configuration tested is recorded at the moment of testing — before its result is observed and before anything is submitted for approval — together with its count, variable set and rationale, since the final model's performance can be interpreted only when that count is known. The second is the locked sample: a portion of the dataset is separated before modelling begins and opened once only, after the final specification has been fixed; where calibration resumes after that opening, the separation has forfeited its meaning entirely. The third is role separation, because when the person building the model is also the person validating it, the validation step converts inevitably into a continuation of calibration. The fourth is a decommissioning threshold declared in advance: the deviation level and the duration of deviation at which the model will be withdrawn from use are fixed in writing before the model is put into service. BEIREK treats pricing, cost and demand models within the capital-intensive projects it manages not as deliverables but as processes carrying an evidentiary record. In practice this means maintaining a specification log alongside the model file — which assumption set was tested, when, and on what grounds; which candidates were eliminated — and presenting the final model to the committee together with that record, so that the committee evaluates not a single model's correspondence with history but the search space from which that correspondence emerged. In parallel, a portion of the dataset is locked before modelling begins and opened for one-time validation once the final specification is fixed; the role conducting that validation sits outside the team that built the model and carries no authority to return to calibration. The second leg of this architecture governs the model's life after deployment. For every parameter committed to a decision — contingency rate, base-case DSCR, demand curve — the deviation band and the duration that will trigger review are written at the moment of deployment; where that threshold is breached during execution, the review is not opened for debate but commences automatically. Within the same rhythm, every forecast version is retained with its date and held in a form permitting later comparison against realized outcomes; that archive renders the model's own quality measurable while also allowing forecasting capability to be verified externally in any subsequent sale or financing process. What institutional memory performs here is not the recollection of the past but the maintenance of a record of the limits of what is known about it. The question of how well a model explains history appears answerable when posed on its own, yet it cannot be used for decision purposes in that form; it becomes usable only when the number of attempts behind that explanation and the dataset from which it was drawn reside in the same file. Whether that second body of information is produced within an organization is far less a question of the model's mathematics than a governance choice about who records what and at what moment — and the consequence of that choice is typically collected not in the period during which the model was built, but in the first period in which conditions change. ## Key Points - The quality of a model's fit to history cannot be interpreted without knowing how many alternative specifications were tried to obtain that fit; absent the second figure, the first carries far less information than it appears to. - As the number of specifications tested approaches the breadth of the sample, the selected model's apparent superiority derives increasingly from the sample's incidental structure rather than from any durable relationship within it. - The institutional cost materializes not at the moment the model errs but when conditions shift and the buffer has already been consumed, showing up through contingency, liquidated damages exposure and working capital pressure. - When a model's failure is diagnosed as a change in market conditions, the method itself survives the diagnosis and the same calibration reflex reappears in the following cycle. - Effective intervention is an architecture rather than an act of individual caution: a specification log, a holdout sample locked before modelling begins, separation of the building and validating roles, and a decommissioning threshold declared in advance. ## Questions ### What is backtest overfitting, and how does it appear in institutional decision-making? Backtest overfitting is the process by which a model, tuned toward ever-closer correspondence with historical data, absorbs into its parameters not the durable structure of that data but its non-recurring fluctuation. Institutionally it appears in tender pricing curves, demand forecasts, inventory reorder points and base-case DSCR assumptions. The model looks accurate for as long as the conditions under which it was built persist; when those conditions shift, the deviation emerges at a point where the buffer has already narrowed. ### How can it be determined whether a model has been overfitted to historical data? A fit measure taken alone does not answer the question. The decisive information is how many variable sets, time windows and weighting schemes were tested before that fit was reached; as the number of specifications approaches the breadth of the sample, the selected model's advantage most likely derives from noise. A second indicator is whether a validation sample was separated before modelling began and opened once only. Where no such sample exists, the distinction cannot be drawn at all. ### What does an overfitted pricing model cost the company? The first cost accumulates in the contingency line: once the model has internalized past deviations as systematic structure, very little remains as unexplained variance, and the buffer is computed more narrowly than circumstances warrant. This raises the win rate in the short term, while during execution it surfaces through the frequency with which the liquidated damages cap is approached, through rework cost and through working capital pressure. In capacity decisions it takes the form of idle investment and commitments that cannot be renegotiated. ### Which institutional mechanism, rather than individual attentiveness, manages model risk? A four-component architecture: recording every specification tested before its result is observed; locking a portion of the dataset before modelling begins and opening it once after the final specification is fixed; separating the roles of building and validating the model; and fixing in writing, at deployment, the deviation band at which the model will be withdrawn from use. Because these components rest on records and allocated authority rather than personal prudence, they survive changes in the team. --- Source: https://www.beirek.com/en/blog/backtest-overfitting Publisher: BEIREK LLC — https://www.beirek.com