● Phase 2 · Vignette Experiment
When Context Contradicts the Algorithm
A pre-registered experiment on how administrative reviewers weigh field-level context against a rule-based algorithmic recommendation in welfare eligibility decisions, and on what an “override” actually is when they depart from it.
Research Question
How do reviewers weigh informal context, and what is an override?
When a case carries a formal eligibility record and an algorithmic recommendation, how do reviewers incorporate contextual field observations the rules cannot encode? And when they depart from the recommendation, is that departure a reversal (a counter-determination) or a refusal to resolve the case? The standard override measure codes both identically.
Findings
Reliance is conditional; the aggregate is a design artefact
Field context does not produce blanket algorithm aversion or appreciation. Its effect reverses depending on whether the contextual note reinforces or contradicts the recommendation. The identified, robust result is this interaction; the aggregate “effect of context” is not treated as a finding, for the reason given below.
| Cell (recommendation × signal) | Control | Treatment | Δ (pp) |
|---|---|---|---|
| Approve × Contradicts | 3.0% | 52.9% | +49.9 |
| Reject × Contradicts | 37.1% | 53.4% | +16.3 |
| Approve × Reinforces | 24.6% | 23.5% | −1.1 |
| Reject × Reinforces | 35.8% | 11.7% | −24.1 |
Override is not one behaviour. Because reviewers could reverse the recommendation or decline to resolve the case, “override” combines two distinct acts. Under evidentiary conflict the dominant departure is escalation — a refusal to resolve — rather than reversal. This is a point about how the measure is interpreted, not a claim that abstention is the underlying mechanism.
Why the aggregate is not the finding. Contextual information raised override by ≈10 pp on aggregate, but that figure is a weighted average of two opposite-signed effects, and its value depends on the mix of contradicting and reinforcing cases in the stimulus pool. Across defensible pool compositions it ranges from −13 pp to +33 pp and changes sign at fewer than five of sixteen profiles. The interaction, by contrast, is stable across estimators, exclusion rules, and pool composition.
What this study does not establish. It is one setting with a screened online sample, not serving officers; absolute rates are not estimates of officer behaviour. Escalation in the task was a single low-cost action, so why departures took the form of non-commitment, considered abstention versus a low-cost default is not identified here. The reasoning-text analysis is corroborative only. Details and limitations are in the paper.
Experimental Structure
A 2 × 2 × 2 within-subject design
Three crossed factors, balanced to break the confound between recommendation direction and signal direction. Signal direction is defined only under treatment; in the control arm it is a latent property of the profile, which lets the treatment effect be estimated within profile.
Each participant evaluates 12 cases from the pool, 6 control + 6 treatment, order randomised, cells balanced 4/4/4/4 across the recommendation × signal-direction crossing. No participant sees both versions of the same profile. A gated practice case and comprehension check precede the main task.
Deployment
A mixed-mode design
Phase 2 was pre-registered as an in-person field experiment with welfare-housing reviewers in Mumbai. Recruiting active reviewers in adequate numbers proved infeasible, the few willing would participate only collaboratively, which breaks the within-subject independence the design requires so the primary inferential arm was moved online, with a small field component and Mumbai informant interviews for ecological grounding. The pre-registration amendment is timestamped before online data collection.
Prolific · June 2026
130 participants completed; 120 retained after the pre-registered exclusion, yielding 1,440 case-level decisions. Screened for public-administration, policy, governance, or law background.
Mumbai · May 2026
Informant interviews with welfare-housing-adjacent professionals, reported descriptively. Informs the methods and discussion; does not contribute to the primary analysis.
Grounding & Sources
Signals paraphrased from field observation and rule documentation
Treatment-arm signals are paraphrased from a corpus of verifier-observed cases recorded during PMAY field work in Maharashtra and from disqualification clauses in the 2018 SRA Government Resolution. Applicant profiles are synthetic composites; no names, addresses, or survey numbers corresponding to identifiable individuals are used.
- Cutoff-date proof & voter-roll evidence
- Family property holdings and household composition
- Tenancy and continuous occupancy
- Documentation authenticity
- Alternate-property declaration integrity
- Hardship framing (corpus-attested only)
Measurement
What the instrument records, and how override is defined
| Layer | Variables |
|---|---|
| Participant | Prolific ID, education, occupation, experience, country |
| Design | arm, recommendation, signal direction, randomisation seed |
| Decision (primary) | decision (approve/reject/escalate), override flag |
| Timing | response time, time to first action |
| Reasoning (corroborative) | brief optional written explanation |
| Comprehension | gated pre-task practice case and comprehension check |
| Post-task | salience and feedback survey |
Override is any decision that differs from the algorithmic recommendation — not rejection of the applicant. It comprises two distinct acts: reversal (the opposite determination) and escalation (declining to resolve, routing onward). Both count as override; escalation is not concurrence. Reasoning text is treated as corroborative, not as a basis for inference.
Status & Openness
Working paper available; materials open
Field component completed May 2026; online data collection completed June 2026. The pre-registration is timestamped before the field component and the amendment before online data collection. The working paper (Draft 1) is available above. Findings are reported regardless of direction, including null and heterogeneous results. This study was conducted as independent research without institutional ethics review, under a documented self-governance protocol (consent, voluntariness, withdrawal, data handling, no personally identifying information); the protocol and consent materials are in the archive.