Human-AI interaction · Product design · Evaluation
Does an AI explanation help someone make a better review decision?
Design explanations around a reviewer’s task, distinguish faithful reasons from correct conclusions, and test whether people can identify and correct consequential errors.
Start with the judgment the reviewer must make
Identify what the person needs to confirm, challenge or defer. A reviewer checking a proposed record match needs the relevant records and differences; someone choosing a next step needs the alternatives and constraints. Select explanation content around that judgment and the person’s available time.
NIST’s explainable-AI principles distinguish an explanation’s meaningfulness to its audience from its accuracy in describing the system. They also separate explanation accuracy from decision accuracy. A clear account of how a result arose can accompany an incorrect result.
Show the evidence the reviewer can inspect, the assumptions affecting the recommendation and important missing information. Label generated narrative separately from retrieved evidence or recorded rules. Present any confidence measure with its defined meaning and limits; a reassuring number should not substitute for the required check.
Give disagreement a workable path
Let the reviewer inspect the relevant input, correct a material error and see whether the recommendation changes. Provide a route to defer or escalate when the evidence is insufficient. Explain which downstream action waits for review and what happens after an override.
In a 2023 study involving two prediction tasks, Chen and colleagues found that the feature-based explanations they tested increased overreliance, while example-based explanations supported better performance in that setting. The result motivates task-specific testing; it does not establish a universally superior explanation format.
Use representative reviewers to compare explanation approaches with the current workflow. Include plausible wrong recommendations, ambiguous evidence and cases where the system is right. Observe which information people actually examine and whether the interface makes disagreement practical under realistic time pressure.
Measure review quality as well as confidence
Track errors caught, correct recommendations unnecessarily rejected, unresolved cases and review time. Inspect outcomes by task type and reviewer experience. Ask how confident people felt, but keep that report separate from whether their final decision was supported by the evidence.
Test the explanation itself when the model or supporting data changes. Check that its stated reasons still correspond to the system’s behavior and that relevant limits remain visible. Explanation quality cannot by itself establish the truth or fairness of the underlying decision.
Use the findings to change the review task, available evidence or action controls. If reviewers cannot reliably identify the important failure, narrow the automated role or strengthen independent checks before expanding it. Preserve unresolved limits in the release decision rather than treating an override button as sufficient oversight.