Why Explainable AI Can Still Produce Bad Decisions

A recommendation can make sense and still miss the decision

Consider a hypothetical full-service hotel preparing for breakfast and a morning banquet. The system recommends moving a cross-trained supervisor from the restaurant to banquet setup. Its explanation is clear: expected breakfast covers are lower, banquet setup demand is higher, and total scheduled labor hours are sufficient.

Designated lead coverage must be maintained. That known requirement should be encoded in the allocation logic or a reliable validation rule. In this example, it is: a qualified backup lead is scheduled for breakfast, so moving the supervisor appears feasible.

But the backup is on light duty today and cannot perform the required function. The manager knows this; the system still represents the employee as available for that role. The coverage rule is present, yet the recommendation rests on an incomplete picture of today’s operation.

The useful question is specific: who can actually cover breakfast during the overlap? A display identifying the assumed backup could help the manager connect that assumption to the current restriction. Repeating “total hours remain sufficient” would not establish that coverage is feasible.

Moving the supervisor is not inherently wrong. If the backup can perform the function, the recommendation may be reasonable. The criterion has to come before the judgment. HDL’s implication is to encode known constraints and use review for consequential information or reasoning the system lacks. That review is not guaranteed to improve the decision.

This full-service example will not map directly to select service. A hypothetical parallel is a housekeeping recommendation based on occupancy that misses an unusual same-morning group departure.

Article 1, When Hotel Managers Override the Algorithm, introduced informational advantage, diagnostic competence and governance alignment as proposed conditions for potentially useful review. Article 2, When Local Knowledge Beats the Forecast, examined whether a manager’s local information adds signal. This article asks the next question: does the information accompanying the recommendation help the manager recognize a consequential defect? It is part of HDL’s work on AI in hotels.

Understanding is not diagnosis

In experiments on apartment-price estimation, Poursabzi-Sangdeh and colleagues found that a small, transparent model was easier for people to simulate. That did not establish better correction of its errors. Transparency impaired correction of sizable mistakes in some conditions, with results changing across presentation variants. This was a study of model interpretability, not a test of every kind of explanation. Poursabzi-Sangdeh et al. (2021).

The authors discuss information overload as an explanation for some of these results. That interpretation is not a universal account of why people miss AI errors. What the study directly supports is the narrower distinction: predicting a model’s output and recognizing when that output is defective are different abilities.

The same discipline applies to the phrase “the explanation worked.” Worked for what? A person might understand the model better, find it more acceptable, follow it more often, or make a better decision. Those are not interchangeable claims.

What exactly is being explained?

“Explainable AI” can describe quite different information. A global explanation concerns how the model behaves across cases. A local explanation concerns a particular output. Feature attributions, examples and counterfactuals are methods for presenting aspects of that behavior. “Why,” “why not” and “what if” identify questions the user wants answered, not interchangeable algorithms. Liao, Gruen & Miller (2020).

An explanation of model behavior is also different from a justification of an operating decision. Showing why a model favors lower labor cost does not establish that its objective adequately accounts for service coverage or employee consequences. A counterfactual showing what would change the model’s output is not, by itself, a guarantee that the change is feasible or would cause the desired real-world result.

There is a second check: does the explanation faithfully describe the system? NIST distinguishes explanation accuracy from the accuracy of the system’s answer. A faithful account can explain a wrong output. A fluent rationale should not be assumed faithful merely because it sounds sensible. Phillips et al. (2021).

Nor is adding an explanation the only design option. Rudin argues for considering interpretable models instead of relying on post-hoc explanations of black boxes in suitable high-stakes settings. That argument is primarily about model transparency and explanation fidelity; it does not by itself establish that a human reviewer will detect consequential operating errors more accurately. The hotel implication is to compare model designs, not simply choose which explanation to place beside an unchanged system. Rudin (2019).

Explanations can help. The comparison matters.

Bansal and colleagues studied human–AI decisions across three datasets under conditions selected so human and AI performance were broadly comparable. Teams achieved complementary performance, meaning they exceeded both solo benchmarks. Explanations did not significantly increase that gain beyond confidence information. They did increase acceptance of AI recommendations whether those recommendations were correct or incorrect. That is a problem for an explanation offered specifically as a way to catch errors. It is not proof that explanations never help, and the comparable-performance design limits how directly the result transfers to settings where one party is clearly stronger. Bansal et al. (2021).

Vasconcelos and colleagues provide important counterevidence. In maze tasks with a simulated AI, explanations reduced overreliance under conditions where checking them was sufficiently worthwhile. Task difficulty, explanation difficulty and incentives affected behavior. Vasconcelos et al. (2023).

Their cost-benefit account suggests that people may allocate effort strategically. An explanation can help when it makes verification easier than working through the whole task unaided. HDL’s inference is that a coverage view identifying who is assumed to cover each service may be more useful than a long narrative repeating aggregate demand. That hotel comparison remains a proposition, not a finding from the maze study.

The broader performance literature also warns against a weak benchmark. Vaccaro and colleagues’ meta-analysis found that human–AI combinations improved on humans alone on average but significantly underperformed the stronger of the human-only and AI-only alternatives (g = −0.23). When AI alone was the stronger party, the combination fell further behind it (g = −0.54), and decision tasks performed worse on average than content-creation tasks. Explanation and confidence were not significant moderators in the coded analysis. These are transferred findings across heterogeneous tasks, not a direct estimate for hotel labor. The authors also note an important high-stakes qualification: even when AI is more accurate on average, human involvement may still be worthwhile when rare errors are very costly and people can reliably reduce them. Vaccaro, Almaatouq & Malone (2024).

A hotel therefore needs more than “managers performed better with the system.” Did explanation improve on the same recommendation without explanation? Did the combined process improve on a credible human-only process? When the comparison is meaningful, did it improve on the AI alone? These are different tests.

The uncomfortable counterexample: more accuracy and more bias

Vered and colleagues found an especially useful contradiction. Their first study, a planning task, found no significant explanation-condition differences in automation bias, trust, or decision accuracy. In their second study, online participants identified letters overlaid on mammogram images; this was not a clinical-diagnosis task. Local explanations improved final accuracy. Yet after participants first saw successful AI examples, those same local explanations also increased agreement with the AI’s particular erroneous suggestions. The local display visually highlighted the region the AI had searched, making it conceptually close to the kind of coverage view HDL proposes. The simulated AI was wrong on six of ten trials, a reliability regime that may differ substantially from a mature hotel production system. Vered et al. (2023).

The accuracy improvement was not confined to cases where the AI was right. It also appeared on wrong-advice trials. The measures captured different events: agreeing with the AI’s exact mistake and producing a fully correct response. Both rates can rise if other kinds of wrong responses become less common.

This rules out two easy conclusions. Increased automation bias does not automatically mean an explanation had no performance value. Increased average accuracy does not demonstrate that a review safeguard reliably catches the errors it was meant to catch. HDL must measure the benefit and the failure mode, not choose whichever metric favors the argument.

Uncertainty is different information

A reason describes aspects of how an output arose. Uncertainty concerns a defined estimate or event. A prediction interval for breakfast covers is not an explanation of the staffing objective. A feature-importance chart is not evidence that a confidence score is calibrated. Missing-data, stale-data and coverage warnings supply still other information.

Zhang, Liao and Bellamy found that model confidence affected switching in relation to confidence without clearly improving combined accuracy. Their second, much smaller study did not establish additional calibration or performance benefits from local explanations. These results do not identify a universal winner between uncertainty and explanation. Zhang, Liao & Bellamy (2020).

For the hypothetical hotel, demand uncertainty may matter if the breakfast forecast is unreliable. It may do little to reveal that the assumed backup cannot perform the role today. The practical question is which information addresses the possible failure and whether the reviewer can use it correctly.

Expertise is not one thing

Wang and Yin found that explanation effects differed across recidivism and forest-cover tasks. Some methods improved understanding without supporting calibrated reliance. Their comparison points to task dependence, but it does not establish that experienced hotel managers outperform less experienced managers: different tasks also differ in more than familiarity. Wang & Yin (2021).

A manager may understand breakfast operations but not a model’s uncertainty display. Another may understand the model but lack the authority or operational knowledge to repair the allocation. Domain knowledge, task skill, AI literacy, statistical literacy and system familiarity should be assessed separately. Tenure alone is not a test of diagnostic competence.

This also sharpens the connection to Article 2. A manager’s independent local fact may expose a mismatch with the model. But an explanation need not add a new external fact to be useful. It may help a reviewer use information already present. Guo, Ustun and Hullman distinguish potential information value from the behavioral value of supplying an explanation. The hotel interaction between local information and explanation remains open. Guo, Ustun & Hullman (2026).

What hotel-specific evidence establishes

Mohammed and Denizci Guillet interviewed 27 stakeholders about integrating XAI into hotel revenue-management systems. Participants described information needs and barriers, including questions about why recommendations arise and what would happen under alternatives. End users emphasized local explanations. This is useful evidence for designing relevant interfaces. Mohammed & Denizci Guillet (2026).

It is not a test of whether managers caught more wrong recommendations. The study did not quantitatively establish the anticipated economic benefits. Preferences, perceived usefulness and expected trust gains cannot stand in for actual managerial choice, decision accuracy or hotel operating results.

No reviewed study in the current Article 3 evidence set directly establishes that explanations improve hotel managers’ detection of consequential labor-allocation errors. That is a bounded statement about this review, not a claim that no one has studied hotel XAI. The evidence discussed here distinguishes stakeholder studies, hospitality surveys and transferred decision experiments.

Consider a separate hypothetical revenue-management example. An RMS explains a rate recommendation through pace, historical demand or competitive positioning. The rationale can be coherent while a group block has not loaded, a competitor feed is stale or a relevant local event is absent. Those possibilities illustrate input and coverage checks; they are not findings from the stakeholder interviews or proof that a manager’s rate adjustment would improve revenue.

The Diagnostic Usefulness Test

HDL proposes a practical operating test for an explanation used in human review. “Diagnostic usefulness” is not presented as a new scientific construct. It overlaps existing work on appropriate reliance, signal discrimination and explanation value. Guo and colleagues’ decision-theoretic work is a direct reason to keep that claim modest. Guo et al. (2024).

Here, diagnostic usefulness means the contribution a particular information display makes to a particular reviewer’s ability to recognize decision-relevant problems, compared with a stated alternative. Whether the reviewer then acts appropriately, and whether the action improves results, must be tested separately.

1. What should the system handle before review?

Design and implementation owners, before deployment or policy changes: define objectives, encode known constraints and identify what can still be wrong even if the recommendation makes sense. Specify which remaining problems a reviewer could realistically detect.

2. Does the explanation describe the actual system?

The designated system or procurement owner, with implementation and provider specialists, before deployment and after material changes: verify explanation fidelity and distinguish rationale, uncertainty and data-quality warnings. Frontline managers should not be asked to certify technical fidelity.

3. What current information can the reviewer contribute?

The designated manager, at decision time when review is triggered: check relevant local facts and operating assumptions within their knowledge. What evidence would change the recommendation? Is there a feasible alternative, or a reason to pause and escalate within assigned authority?

4. What evidence beyond the explanation matters?

The designated reviewer, on a flagged decision: check available evidence of missing or stale inputs, uncertainty and known exceptions. Ask what can still be wrong. A plausible rationale is not a substitute for these checks; missing evidence may require escalation.

5. Does review help, and what should change upstream?

Accountable operational and system owners, during evaluation and recurring governance reviews: compare displays and review policies against useful baselines. Track missed errors, false alarms, feasible repairs, time and consequences. Assign repeated defects to a rule, data, integration or system owner and retest the correction.

Hotel Decision Lab proposed operating test. Not validated in hotel operations. These are responsibilities across the decision system, not five checks for every recommendation. A completed review does not certify safety or establish that a decision is well-founded.

“The organization” must mean named owners. Depending on the hotel’s structure, responsibilities may sit with property leadership, a management company, ownership, a brand, corporate functions or implementation and technology-provider teams. HDL proposes assigning each responsibility explicitly, including who may accept, modify, pause or escalate. A property GM should not be presumed to control labor standards, configuration or AI policy.

The test is deliberately narrower than all the reasons an organization might want explanations. Learning, accountability and contestability can matter even where immediate error detection does not improve. Those purposes should be evaluated on their own terms.

What hotels can reasonably consider now

HDL proposes selective review. Routine recommendations within an evaluated policy can flow with less friction; exceptions receive more scrutiny. Illustrative triggers include unusually large labor changes, cross-department or classification moves, required supervisory or certified coverage, stale or missing inputs, uncertainty, known local exceptions, and consequential or hard-to-reverse actions. The accountable owners should define operational thresholds and monitoring. These are examples to test, not validated HDL rules.

Review should preserve a route to challenge the recommendation. “Total hours remain sufficient” should not become pressure to accept a move when current coverage is unresolved. “The system recommended it and the manager reviewed it” is a record of steps, not evidence that the decision is sound.

Define the consequences before testing: overtime, departmental labor attribution, pay treatment, role restrictions, actual proficiency, supervisory coverage and applicable break or meal requirements. These are operating considerations, not universal legal rules. Record missed errors, false alarms, feasible repairs, review time and service effects. Rejecting more AI advice is not inherently an improvement. A poor outcome does not automatically prove that the decision process was bad; assess what was knowable at the time.

Do not assume that more review friction is costless. Buçinca and colleagues found that cognitive forcing could reduce overreliance, with usability and preference tradeoffs. That supports testing targeted checks, not mandating an identical pause before every hotel decision. Buçinca, Malaya & Gajos (2021).

Each exception should also inform the next system decision. HDL proposes distinguishing a one-time local condition from recurring missing data, an encodable constraint or a structural defect. Repeated corrections should prompt the responsible owner to improve the rule, data, integration or system and test the result. Known recurring constraints should migrate into the system rather than remain permanent human-memory tasks. The aim is less avoidable review, not permanent manual checking; whether this loop improves hotel outcomes remains a proposition.

The next research question

A useful HDL experiment would give hotel managers matched recommendation instances in four conditions: recommendation alone, explanation, uncertainty, and both. Each assisted condition would contain acceptable and unacceptable recommendations defined against a pre-specified decision standard before exposure. Adjudication should use documented operating objectives and constraints, multiple qualified reviewers where feasible, and agreement checks for disputed cases. Separate human-only and AI-only benchmarks would prevent “better than unaided managers” from becoming the whole definition of success.

Five propositions follow: failure-relevant explanations may improve discrimination more than generic rationales; explanation effects may differ with recommendation correctness; uncertainty may help more with forecast ambiguity than with objective defects; verified local information may help when an explanation exposes the affected assumption; and lower verification effort may help reviewers use the same information. These are claims to test, not findings. The companion protocol states their moderators, instance-level measures and falsifiers.

The initial setting should be a controlled simulation, not deliberately bad live staffing. Assess task skill separately from tenure and AI literacy. Report diagnostic discrimination, false acceptance, false rejection, correction, response time and expected decision loss. Employee and guest consequences should remain visible rather than disappearing into one unexplained accuracy score.

The limit, and the contribution

This essay is a targeted update within HDL’s structured integrative review, not a systematic review or original hotel experiment. Much of its causal evidence comes from short, controlled nonhotel tasks with online or lay participants rather than hotel professionals accountable for operating consequences. Vaccaro and colleagues did not find participant type to be a significant moderator, but hotel transfer remains uncertain. The operating test is unvalidated. Persona review challenged its realism; it was not a research sample or external practitioner validation. The HDL methodology explains the broader research approach; the limitations above identify the evidence boundaries for this article.

The unresolved question is not whether explanations are good or bad. It is what an explanation helps this manager do, with this recommendation, under these operating constraints. The HDL research program can advance by testing that question rather than treating understandable advice as evidence of effective oversight.

Make the recommendation understandable. Then test whether the manager can judge it.

References and evidence access