When the Model Is Wrong: Risks, Hazards, and Failure Modes of AI in Safety-Critical Track Inspection

Executive Summary

Every benefit claimed in this series depends on the AI being right, and this paper is about the ways it can be wrong. It develops a six-class hazard taxonomy for AI-based substructure assessment: model error hazards, data hazards, automation-relationship hazards, institutional hazards, security hazards, and accountability hazards. Each class is assessed for mechanism, observability, severity, and available controls, and each is anchored where possible in documented precedent rather than speculation.

The taxonomy draws on three bodies of evidence. Federal risk management doctrine, principally the NIST AI Risk Management Framework, supplies the failure vocabulary: validity and reliability as the base condition of trustworthiness, drift and robustness as measurable properties, and adversarial examples, data poisoning, and model exfiltration as named security concerns. The rail record supplies precedent: GAO’s documented remediation disincentive, the human-automation teaming design literature developed for track inspection, the negotiated verification and response protocols of the geometry waiver record, and the East Palestine derailment sequence, in which, according to Congressional Research Service reporting of National Transportation Safety Board preliminary findings, wayside detectors registered a rising bearing temperature that remained below the thresholds the railroad had set for stopping the train. That sequence is not a substructure AI failure, and this paper does not treat it as one; it is cited for the structural lesson it carries, that a monitoring system’s value is bounded by the response thresholds and protocols wrapped around it. The series’ own analysis supplies the remainder: the substructure-specific hazards no external literature yet documents.

Two findings organize the result. First, the most dangerous hazards in the taxonomy are the quiet ones: silent model degradation under distribution drift, label decay in the training corpus, and skill erosion in the inspector workforce, all of which reduce protection without producing an alarm. Second, most identified hazards have available controls, but three carry control gaps that Paper 5 must either close or explicitly accept: independent field validation depth for deep learning ballast models, response protocol design for AI-detected conditions that have no regulatory classification, and responsibility assignment when an AI-informed maintenance decision precedes a failure.

1. Purpose and Method

The taxonomy below classifies hazards, meaning conditions with the potential to produce harm, whether the harm is a missed deteriorating condition, a wasted maintenance program, an eroded inspection capability, or an unassignable failure. Each hazard entry records four attributes: mechanism, how the hazard produces harm; observability, whether the hazard announces itself or operates silently; severity, the character of the worst credible outcome; and controls, the technical, procedural, or regulatory measures available, with gaps noted where no adequate control exists. The federal vocabulary throughout is that of the NIST AI Risk Management Framework, which identifies validity and reliability as the necessary base condition of AI trustworthiness, supported by safety, security and resilience, accountability and transparency, explainability, privacy, and managed bias, and which organizes risk treatment through its govern, map, measure, and manage functions.

One scoping rule governs the whole paper: hazards are stated with their evidentiary status. Where precedent exists in the rail record or the AI failure literature, it is cited. Where a hazard is anticipated on analytical grounds from this series’ own findings, it is identified as such. The two are not the same weight of evidence, and the register in Section 8 distinguishes them.

2. Class 1: Model Error Hazards

False negatives. The defining hazard: a model that fails to flag deteriorating substructure leaves the condition invisible exactly where the technology promised sight. Mechanism: classification or regression error at inference, whether from inadequate training coverage, sensor noise, or genuinely novel conditions. Observability: low by construction, since the error’s signature is the absence of a flag; it surfaces only through ground truth encounters, recurring geometry defects at unflagged locations, or failure. Severity: the highest in the taxonomy, because substructure deterioration expresses itself ultimately in geometry degradation and, unchecked, in derailment risk. Controls: the three-construct validation architecture of Paper 2, in-service performance monitoring comparing predictions against subsequent geometry outcomes, and, critically, retention of the geometry measurement layer as an independent downstream check; a substructure false negative does not disable geometry inspection, which is why this series has consistently positioned substructure AI as additive capability. Control gap: independent field validation depth for deep learning ballast condition models, identified in Paper 1 as the thinnest in the sensing portfolio, remains the weakest link in false negative control.

False positives. The mirror hazard, economically rather than physically dangerous. Mechanism: over-flagging consumes maintenance capacity, triggers unnecessary investigation or remediation, and, if thresholds carry operational consequence, imposes unnecessary slow orders whose costs the rulemaking record documents as central to the economics of inspection regulation. Observability: high, since every flag is inspectable. Severity: economic, with a second-order safety cost, alert fatigue, developed under Class 3. Controls: precision requirements in validation, tiered response protocols that route low-confidence flags to verification rather than remediation, following the verification-desk model the geometry waiver record established, and the maintenance-limit versus safety-limit distinction of Paper 2.

The threshold lesson. The East Palestine sequence, as reported by the Congressional Research Service from NTSB preliminary findings, is the documented cautionary case for detection-response design in rail: the train passed multiple wayside detectors registering an increasing bearing temperature that did not exceed the thresholds the railroad had set for stopping, and the bearing subsequently failed. The structural lesson, applied here without analogical overreach, is that model accuracy and threshold-response design are separable failure surfaces: a detection system can perform exactly as designed and still fail to prevent harm if the response protocol wrapped around it is miscalibrated. For substructure AI, whose detected conditions currently have no regulatory classification at all, the response protocol surface is entirely unbuilt. Control gap: response protocol design for AI-detected substructure conditions is unaddressed by any existing standard and is assigned to Paper 5.

3. Class 2: Data Hazards

Training data unrepresentativeness. Mechanism: models trained on the available corpus underperform on ballast materials, climates, moisture regimes, and traffic profiles the corpus underrepresents, and Paper 1 established both the scarcity of annotated substructure data and the field’s resort to synthetic particle generation whose transfer to field conditions at fleet scale is unproven. Observability: moderate; geographic performance stratification is measurable if anyone measures it. Severity: systematic regional blind spots. Controls: representativeness requirements in the validation protocol, stratified performance reporting, and the shared-corpus architecture of Paper 1, which widens coverage faster than any single operator’s data can.

Distribution drift. Mechanism: the field changes under a validated model, as ballast sources, maintenance practices, sensing hardware generations, and climate-driven moisture regimes shift, so a model that was valid at deployment degrades without any code change; the NIST framework identifies resistance to drift and degradation as a component of reliability precisely because drift is the normal condition of deployed AI. Observability: low, the defining quiet hazard, inherited from Paper 3 as silent model degradation corrupting the maintenance program built on the forecasts. Severity: progressive, compounding, and program-wide. Controls: periodic revalidation on schedule and on trigger (sensor fleet changes, material source changes), drift detection statistics on input distributions, and the closed survey-model loop the pavement precedent demonstrated, in which each network survey cycle revalidates the prediction model.

Label decay and spurious structure. Mechanism: two documented variants. Ground truth labels age as the track they describe is maintained and renewed, so a corpus without maintenance-history linkage trains on conditions that no longer exist; and models trained across maintenance interventions learn correlations that vanish after each tamping, a failure mode documented in the degradation modeling literature, where researchers found models attempting to establish relationships not present after maintenance events. Observability: low without deliberate audit. Severity: quietly corrupted forecasting. Controls: maintenance-history linkage as a mandatory corpus field, intervention-aware model structures of the kind the Bayesian degradation literature demonstrates, and label stewardship responsibility, the governance element Paper 1 found missing from current arrangements.

4. Class 3: Automation-Relationship Hazards

The FRA-sponsored human-automation teaming literature decomposes inspection into four tasks, data collection, data analysis, decision making, and action, and provides the design frame this class uses.

Automation bias and misallocated trust. Mechanism: personnel over-weight automated outputs, accepting model assessments without verification, or under-weight them, discounting valid flags after false positive experience; both are trust miscalibrations, and the teaming literature identifies transparency, the human’s ability to understand what the automation is doing or suggesting, as the design property that supports calibration. Observability: moderate, measurable through verification audit rates. Severity: bias converts model error hazards into acted-upon errors; discounting converts valid detection into ignored detection, the alert-fatigue path. Controls: explainable outputs consistent with the NIST explainability characteristic, confidence communication with every flag, per the uncertainty quantification research priority Paper 1 assigned, and verification sampling that audits accepted flags at a defined rate.

Skill degradation. Mechanism: as automated assessment absorbs the detection task, the inspector workforce’s independent condition-assessment skill erodes, thinning the human check exactly as reliance on it deepens; the teaming report frames this as a consequence of task allocation choices, and it is the hazard with the longest time constant in the taxonomy. Observability: low over any short horizon. Severity: loss of the independent verification capability every other control in this paper assumes exists. Controls: deliberate task allocation that retains hands-on assessment in the inspector role, the verification, interpretation, and remediation-management role definition this series has proposed, training programs that maintain unassisted proficiency, and workforce transition planning before deployment rather than after.

The substructure twist. For substructure assessment the teaming problem has a property geometry automation lacked: the automation contributes measurements for which no human baseline exists, so the inspector cannot check the model against personal observation of fouling percentage or trapped moisture. Verification must therefore be designed around checkable correlates, surface symptoms, geometry behavior, and sampled ground truth, rather than direct human re-measurement. This is an anticipated hazard from the series’ own analysis; no external literature yet addresses it, and Paper 5’s teaming pillar inherits it.

5. Class 4: Institutional Hazards

The remediation disincentive. Documented by GAO: technologies that identify defects perceived as too insignificant to pose safety risk nonetheless trigger remedial obligations once the defects are identified, deterring adoption of the technologies that reveal them. For substructure sensing the mechanism operates prospectively and at scale, as Paper 2 established: every newly measured mile is potentially obligation-bearing. Controls: the maintenance-limit and safety-limit distinction, thresholds set from performance-referenced evidence, and remediation criteria negotiated at the test program stage, all specified in Paper 2’s pathway.

Threshold-adjacent behavior and metric gaming. Mechanism: once a condition index carries consequence, incentives arise to manage the index rather than the condition, through measurement timing, calibration choices, or contesting classifications near thresholds; this is an anticipated hazard, inferred from the general behavior of consequential metrics rather than from a documented rail instance. Observability: moderate, through audit of measurement practices and threshold-proximal distributions. Controls: independent measurement audit, the federal inspection program’s independent survey role, which the ATIP fleet already performs for geometry, and metric definitions fixed in recommended practice rather than operator discretion.

The accountability gap. Inherited from Paper 3: prediction capability without published condition targets and performance reporting produces the oversight outcome the California pavement record documents, in which sustained state auditor and legislative analyst criticism attached to a program whose funding lacked specific performance mechanisms. Controls: condition-target accountability on a published cadence, assigned to Paper 5’s economic pillar.

The contested automation dispute. [CONTESTED TERRAIN] The unresolved conflict over whether automated inspection should permit reduced visual inspection frequencies, documented through the advisory committee’s non-consensus, litigation, and opposing docket positions of labor and industry, is itself an institutional hazard for substructure programs: any deployment that touches inspection frequencies inherits the dispute, and program designs that entangle substructure sensing with visual-inspection reduction import a conflict the technology does not require. This series carries the dispute as contested and does not resolve it; the available control is the positioning this series has maintained, substructure sensing as additive measurement of what visual inspection cannot see.

6. Class 5: Security Hazards

Pipeline integrity. Substructure AI creates a data pipeline from field sensors through processing, model inference, verification desks, and maintenance systems, and that pipeline is an attack surface. The federal AI risk vocabulary names the model-layer concerns: adversarial examples, data poisoning of training corpora, and exfiltration of models and training data through system endpoints. The rail-specific regulatory context already treats cyber compromise of operations as a live threat: TSA’s security directive series for freight and passenger railroads, issued beginning in 2021 and maintained through successive revisions ratified by the Transportation Security Oversight Board, requires designated carriers to report cybersecurity incidents to CISA, maintain an always-available cybersecurity coordinator, assess vulnerabilities, maintain incident response plans for critical cyber systems, and, under the 2022 series, establish TSA-approved cybersecurity implementation plans and assessment programs for identified critical cyber systems.

Mechanism: corrupted training data biases models; manipulated inference inputs or outputs falsify condition pictures; and a falsified substructure picture corrupts maintenance decisions with a long detection lag, since the ground truth encounter rate is low. Observability: low to moderate, depending on integrity monitoring investment. Severity: a corrupted condition picture is functionally equivalent to the false negative and drift hazards, with intent behind it. Controls: bringing substructure data pipelines and model infrastructure within the carrier’s critical cyber system determination under the TSA directive framework, integrity verification on measurement archives, provenance controls on training data consistent with the data governance pillar, and the NIST secure-and-resilient characteristic’s protection mechanisms. Whether inspection-analytic systems fall within a given carrier’s designated critical cyber systems is a determination the directive framework assigns to the carrier subject to TSA review; treating them as in scope is the conservative design choice.

7. Class 6: Accountability Hazards

Responsibility assignment after failure. Mechanism: when a track-caused failure occurs on territory managed through AI-informed substructure programming, the responsibility question fragments across the model developer, the data steward, the verification desk, the maintenance planner who acted or declined to act on the forecast, and the railroad that set the thresholds. The NIST framework makes accountability and transparency a load-bearing trustworthiness characteristic precisely because diffuse AI decision chains resist after-the-fact reconstruction without deliberate design. Observability: the hazard is invisible until the day it is the only thing that matters. Severity: beyond the immediate failure, unassignable responsibility corrodes the regulatory trust the standing pathway of Paper 2 depends on. Controls: decision logging that records model version, input data, output, confidence, verification disposition, and responder action for every consequential flag, the record-keeping analog of the verification workflow the geometry waiver record established; defined decision rights among the roles, assigned in Paper 5’s framework; and model and data versioning sufficient to reconstruct any historical assessment. Control gap: no existing rail standard defines the investigation-grade record set for AI-informed maintenance decisions; Paper 5 must specify it.

8. Consolidated Hazard Register

9. Findings

Finding 4.1: The quiet hazards dominate the risk profile. H5 drift, H6 label decay, and H8 skill degradation share low observability, long time constants, and program-wide reach; each reduces protection without producing an alarm, and each is controllable only by standing processes, revalidation, stewardship, and proficiency maintenance, that must be resourced permanently rather than performed once.

Finding 4.2: Detection and response are separable failure surfaces, and the response surface for substructure AI is unbuilt. The documented rail precedent shows a monitoring system bounded by the thresholds and protocols around it; substructure AI’s detected conditions currently carry no regulatory classification and no standard response protocol, making H3 a structural gap rather than a design detail.

Finding 4.3: Most hazards have available controls; three carry genuine gaps. Field validation depth for deep learning ballast models (H1), response protocols for unclassified AI-detected conditions (H3), and an investigation-grade decision record standard (H14) lack adequate existing controls, and Paper 5 must close each or document its explicit acceptance.

Finding 4.4: The taxonomy’s institutional hazards are as consequential as its technical ones, and better documented. The remediation disincentive, the accountability gap, and the automation dispute all carry direct documentation in the oversight and regulatory record, while the technical hazards rest more heavily on general AI risk doctrine and this series’ analysis; a deployment program that treats risk as a purely technical property inverts the actual evidence base.

Finding 4.5: The additive positioning is itself a control. Retaining geometry measurement as an independent downstream check bounds H1, and frequency-neutral program design avoids H12; both derive from the same design stance this series has maintained, that substructure sensing measures what nothing else measures and substitutes for nothing.

10. Limitations

The taxonomy is bounded by its evidence. No documented in-service failure of an AI-based substructure assessment system exists in the public record, so the model error and accountability classes rest on general AI risk doctrine, adjacent rail precedent, and this series’ analysis rather than on substructure-specific incident history; the register’s evidentiary status column exists so that readers weight entries accordingly. The East Palestine citation is confined to the detection-response structural lesson as reported by CRS from NTSB preliminary findings and implies no claim about substructure technology or about the parties involved. Severity is characterized qualitatively; quantitative risk assessment requires the deployment-specific exposure data a pilot program would generate. Insider threat and physical security of sensing assets were noted during research and assigned to the post-series agenda rather than treated superficially.

11. Threads Passed Forward

Everything in this paper is input to Paper 5. The fourteen-hazard register transfers whole: the capstone’s framework must allocate every entry to a closing control, a mitigating control with monitoring, or a documented accepted risk, with the three gap hazards (H1, H3, H14) requiring explicit resolution. The quiet-hazard finding shapes the assurance pillar, which must institutionalize revalidation, stewardship, and proficiency maintenance as permanent functions. The verification-design problem where no human baseline exists (H8’s substructure twist) transfers to the teaming pillar. The critical-cyber-system scoping discipline transfers to the data governance pillar. And the contested automation dispute transfers, still contested, as a boundary condition the framework must design around rather than through.

Bibliography

Tier 1: Government and Oversight Sources

Congressional Research Service. East Palestine, OH, Train Derailment and Hazardous Materials Shipment by Rail: Frequently Asked Questions. CRS Report R47435. Washington, DC, 2023. https://crsreports.congress.gov/product/pdf/R/R47435.

Department of Homeland Security. “Ratification of Security Directives.” Federal Register, January 21, 2025. https://www.federalregister.gov/documents/2025/01/21/2025-01422/ratification-of-security-directives.

Federal Railroad Administration. “Track Geometry Measurement System (TGMS) Inspections.” Notice of Proposed Rulemaking. Federal Register, October 24, 2024. Docket No. FRA-2024-0032.

Federal Railroad Administration / John A. Volpe National Transportation Systems Center. Human-Automation Teaming in Track Inspection. Washington, DC: U.S. Department of Transportation. https://rosap.ntl.bts.gov/view/dot/66915.

National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD, January 2023. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf.

Transportation Security Administration. Security Directive 1580-21-01 series (Enhancing Rail Cybersecurity) and Security Directive 1580/82-2022-01 series (Rail Cybersecurity Mitigation Actions and Testing). Washington, DC, 2021-2025. https://www.tsa.gov.

U.S. Government Accountability Office. Rail Safety: Federal Railroad Administration Should Report on Risks to the Successful Implementation of Mandated Safety Technology. GAO-11-133. Washington, DC, 2010.

Docket Filings

Association of American Railroads and American Short Line and Regional Railroad Association. Comments on Track Geometry Measurement System (TGMS) Inspections NPRM. Docket No. FRA-2024-0032, January 2025 (verification workflow and response window record).

Tier 1 and Tier 4 State and Research Sources

California State Auditor. Report 2017-601. Sacramento, CA, 2017.

Ding, Kelin, et al. “Augmented Dataset for Vision-Based Analysis of Railroad Ballast via Multi-Dimensional Data Synthesis.” Algorithms 17, no. 8 (2024): 367.

“Predicting Track Geometry Using Machine-Learning Methods.” University research report documenting spurious correlation across maintenance interventions, 2024.

Wang, et al. “Prediction Models for Railway Track Geometry Degradation Using Machine Learning Methods: A Review.” Sensors 22, no. 19 (2022): 7275.