--- title: "When Absence Has a Pattern: Data That Goes Missing Because of What It Would Have Shown" description: "When the likelihood of a record going missing depends on the value that record would have held, the gap is systematic bias rather than statistical noise, and every average computed from surviving data reads better than the truth. In institutional measurement this appears as bad outcomes declining to report themselves. The corrective mechanism is treating the missing record as a data point and reporting coverage alongside performance." url: https://www.beirek.com/en/blog/missing-not-at-random-bias-corporate-data canonical: https://www.beirek.com/en/blog/missing-not-at-random-bias-corporate-data published: 2025-04-29 modified: 2025-04-29 category: "Organisational Psychology" category_url: https://www.beirek.com/en/blog/category/organisational-psychology language: en-US reading_time_minutes: 7 publisher: BEIREK LLC publisher_url: https://www.beirek.com license: "© BEIREK LLC — citation with attribution and link permitted" keywords: ["missing-not-at-random bias","data quality governance","reporting incentives","contingency calibration","due diligence findings"] topics: ["Organisational measurement design","Selection effects in institutional reporting","Transaction structuring under information asymmetry"] alternate_language_url: https://www.beirek.com/tr/blog/missing-not-at-random-bias-corporate-data --- # When Absence Has a Pattern: Data That Goes Missing Because of What It Would Have Shown > **In short:** When the likelihood of a record going missing depends on the value that record would have held, the gap is systematic bias rather than statistical noise, and every average computed from surviving data reads better than the truth. In institutional measurement this appears as bad outcomes declining to report themselves. The corrective mechanism is treating the missing record as a data point and reporting coverage alongside performance. *Gaps in institutional data sets rarely distribute themselves evenly; the probability that a record never gets created is frequently a function of the value that record would have carried. That structure lifts the reported average on every measure from customer sentiment to schedule variance, leaving boards to decide on an optimistic cross-section of reality rather than reality itself.* --- A customer satisfaction dashboard holding its monthly average within a narrow band across six quarters, while the response rate over the same period declines visibly, is a configuration encountered often enough in institutional reporting to warrant attention. The dashboard itself reports nothing wrong; the average is stable, marginally improved even. The response rate sits on a separate line, frequently in a footnote, and rarely travels to the summary page of the board pack. The same pattern recurs in supplier performance scorecards, in site safety observation forms, in the attendance records of post-project lessons-learned sessions, and in exit-interview completion rates. What these share is that the measure continues to operate while the flow feeding it grows thinner. A second observation sharpens the picture. On a construction programme, the lateness of weekly progress reports tends to move with the lateness of the programme itself; when the works fall behind plan, the report arrives late, and in some weeks does not arrive at all. The weeks in which reports do arrive are, typically, the weeks in which matters proceed as intended. The average schedule variance computed at period end from the available report set therefore comes in below the variance actually experienced on site — a shortfall arising not from arithmetic error but from which weeks entered the record. This structure carries a name: missing-not-at-random bias, the condition in which the probability of a value going unrecorded depends on that very unobserved value. Distinguishing it requires placing three regimes of missingness side by side. In the first, absence is genuinely incidental — a server fails, a form is mislaid, lost records scatter without pattern, and what remains is a smaller but unbiased sample. In the second, absence tracks some other observed variable; weak connectivity in a particular region yields fewer responses from that region, yet the regional field is recorded and correction remains available. In the third, absence attaches directly to the quantity being measured: the dissatisfied customer does not complete the survey, the delayed contractor does not submit the report, the employee whose reason for leaving is sensitive does not attend the exit interview. Within that third regime, no statistical adjustment applied to the surviving data recovers what never entered it. Treating this tendency as simple dysfunction misreads the mechanism. Reporting is not costless; every form, every session, every explanation consumes time and attention, and carrying bad news costs structurally more than carrying good news, since bad news generates follow-up questions, additional meetings, and further justification. Where a system routes the creation of a record through the same channel that evaluates the person creating it, the choice not to report an adverse result becomes entirely rational at the individual level. The difficulty lies not in the choice but in the persistence of the configuration that makes the choice rational, because the same configuration quietly renders the institution's decision base more optimistic than the underlying reality. The first surface on which the institutional cost registers is budget calibration. Where an organisation converts the average historical schedule or cost variance into a contingency allowance for future projects, that allowance will be systematically thin, because the average feeding it is systematically low. On any single project the shortfall presents as a cost overrun and attracts a project-specific explanation; across the portfolio it accumulates as a recurring pattern. Contingency that depletes with regularity across a portfolio points less toward a sequence of unfortunate projects than toward the structure of the record set from which the estimate was drawn. The second surface lies at the intersection of valuation and the scope of representations and warranties in acquisitions. Where a target's data room contains comparatively sparse churn analysis, warranty repair records, or supplier dispute files, that sparseness admits two readings: either few problems arose, or problems arose and no file was opened. Separating the two requires examining the trigger that causes a file to be created rather than counting the files themselves. If record creation depends on a formal customer complaint and the complaint channel is burdensome, a low complaint count measures the height of the threshold rather than the level of satisfaction. On the transaction side this distinction expresses itself less in headline price than in the escrow percentage, the definition of earn-out triggers, and the survival periods attaching to warranties. The third surface operates more slowly and costs more: the direction of institutional learning. Where an organisation is fed only by the data of completed projects, won tenders, and retained employees, what it learns is not the conditions of success but the shared characteristics of survivors. If the price structure of lost tenders goes unrecorded, pricing discipline is calibrated exclusively against the margin of won work, and that calibration drifts over time toward either systematic aggression or systematic conservatism. The drift itself remains unobservable, since the comparative data that would reveal it was never generated. Structural intervention begins from the recognition that missing data constitutes a process design problem rather than a statistical one. Four components carry the design. The first is treating absence itself as a data point: reporting coverage alongside every performance indicator and reading the movement of that coverage together with the movement of the indicator. The second is separating the moment of record creation from the moment of assessment; where the channel through which a deviation is reported is the same channel through which the deviation is answered for, the reporting flow dries up in a predictable manner. The third is sampling the non-respondents — reaching a bounded subset of customers who did not complete the survey produces the only genuine information about the distribution of the missing mass on which any correction can rest. The fourth is inverting the default state of the record: replacing a design that requires action for an event to be captured with one that requires justification for an event not to be captured. Across programmes BEIREK manages, these components operate inside the existing project control rhythm rather than as a separate reporting layer. In the weekly progress set, each line carries the creation date of its source record and the identity of the person who entered it; a line left blank in consecutive periods generates an agenda item irrespective of the value it would have held. In contractor and supplier performance files, the opening of a record is not left to the counterparty's formal notification; field observation and counterparty notification are maintained on two separate tracks, and the divergence between them is monitored as an indicator in its own right. That divergence tends to become visible ahead of the delay itself. The second line of intervention concerns holding the decision record at the point of proposal rather than the point of approval. Where the rationale, assumptions, and expected range of an investment decision are committed to writing before the decision is taken, a comparable record survives whatever outcome subsequently materialises; where the record is assembled afterwards, which decisions had their rationale written down becomes correlated with how they turned out, and the archive optimises itself. The same discipline extends to lost work: the price structure, competitive position, and stated reason for rejection of unsuccessful bids are filed in the same format as won mandates. Holding those two record sets side by side releases the calibration of pricing and risk appetite from the characteristics of the survivors. None of these mechanisms eliminates missing data; their purpose is to render the direction of the absence observable. A board that does not know which population produced the indicator in front of it, or how that population was selected, is deciding not on the number itself but on the number as it emerges from a survival process. The maturity of a measurement system is better judged by its capacity to report its own blind spot than by the precision of the figures it produces. The question that tests an institution's data discipline is not which indicators it tracks; it is how long it takes to notice that a particular event never entered the record at all. ## Key Points - Where response rates decline while the reported average holds steady or improves, the working assumption should shift toward missingness being driven by the unobserved value itself. - The non-reporting of adverse outcomes is not primarily a question of individual candour; it is the predictable output of a system that places the cost of reporting on the party who owns the bad result. - The balance-sheet consequence of missing data typically surfaces not in a recorded cost line but in the absence of a cost line that was never opened. - In diligence, which categories of file thin out systematically constitutes a finding in its own right, quite apart from whether the data room is nominally complete. - Neutralisation runs through process design rather than statistical adjustment: the moment a record is created must be separated from the moment performance is assessed. ## Questions ### How does missing-not-at-random bias differ from random data loss? Under random loss the surviving set shrinks while retaining its representativeness; the average holds and only the confidence interval widens. Where absence depends on the unobserved value, the loss is selective: values of a particular magnitude systematically fail to enter the record, and the average computed from what remains departs from the true average. Enlarging the sample does not reduce that departure, because its source is the selection mechanism rather than sample size. ### How can it be established that missingness in a measure is systematic? The most practical test compares the movement of coverage with the movement of the indicator. Where the response rate falls while the average rises or holds flat, the assumption that absence correlates with the measured value becomes operative. A second test involves reaching a bounded subset of non-respondents directly and comparing their distribution. A third examines the threshold at which a record is created; where the threshold is high, a low event count does not signify low risk. ### How is missing-data risk managed in an acquisition process? Rather than counting files in the data room, the analysis examines the trigger that causes each file to be opened. Where complaint, dispute, or warranty records depend on formal notification, a sparse record set may measure the height of the threshold rather than the absence of incidents. That residual uncertainty is addressed in transaction structure rather than headline price: escrow percentage, warranty survival periods, and earn-out trigger definitions are the instruments that price the visibility gap created by the recording threshold. ### Which institutional mechanism prevents adverse outcomes from going unreported? Appeals to individual transparency do not resolve this; what does is separating the moment a record is created from the moment performance is assessed. Where the channel for reporting a deviation is also the channel through which it is answered for, reporting dries up predictably. Effective design inverts the default: instead of requiring action for an event to be captured, it requires justification for an event not to be captured, so that a blank line generates its own agenda item. --- Source: https://www.beirek.com/en/blog/missing-not-at-random-bias-corporate-data Publisher: BEIREK LLC — https://www.beirek.com