Under what conditions does providing chain-of-thought explanations to users actually improve human decision-making accuracy, and when does it instead increase over-trust in an AI system’s outputs?

cross-lingual-cot-trust | Updated at

Answer

Providing chain-of-thought (CoT) explanations improves human decision-making accuracy when:

  1. Users have enough relevant knowledge to critically evaluate the reasoning

    • Domain experts or moderately knowledgeable users can spot gaps, unjustified steps, or domain violations in the CoT.
    • The task is one where humans can, in principle, verify steps (e.g., math, logic puzzles, well-specified decision rules, checklist-style procedures).

  2. The AI’s reasoning steps are mostly valid and aligned with ground truth

    • Model errors are relatively rare and localized rather than pervasive.
    • The reasoning decomposes the problem into verifiable substeps that map cleanly to domain concepts.
    • The explanation is faithful (it tracks the real internal process closely enough that checking it is meaningful).

  3. The interface and instructions actively encourage critical engagement

    • Users are prompted to verify or critique each step (e.g., checklists, explicit “find any mistake” prompts, uncertainty ratings).
    • The CoT is concise, structured (numbered or modular), and not so long that it overwhelms users.
    • The UI visually separates model reasoning from final recommendation and may show model uncertainty or alternative options.

  4. There is timely feedback or ground truth to calibrate trust

    • Users can see outcomes or receive corrections over time, enabling them to learn how often the reasoning is reliable.
    • In repeated tasks, users can adjust their reliance on the AI based on observed error patterns.

  5. The decision stakes and context allow for deliberate reasoning

    • Users have time and incentive to deeply process the explanation (e.g., educational settings, analysis tasks, safety reviews).
    • Tasks reward accuracy over speed, and there is institutional or workflow support for double-checking the AI’s reasoning.

Providing CoT instead increases over-trust and can harm decision-making when:

  1. Users lack the expertise or bandwidth to evaluate the reasoning

    • Non-experts or cognitively overloaded users cannot distinguish plausible-sounding but incorrect reasoning from correct reasoning.
    • In complex, opaque domains (e.g., nuanced medical diagnosis, legal strategy, macroeconomic forecasting), humans cannot reliably validate intermediate steps.

  2. The explanation is persuasive but not faithful or not well-calibrated

    • The AI produces confident, fluent, detailed CoT even when its internal reasoning is weak or wrong (making errors look more credible).
    • The system hides its uncertainty, fails to signal when it is extrapolating, or presents speculative chains as authoritative.

  3. Users are implicitly encouraged to treat CoT as evidence of reliability

    • Interface design or social context suggests that a longer or more detailed explanation means the answer is more trustworthy.
    • Users are not warned that LMs can generate convincing but incorrect reasoning.
    • Organizational norms encourage deferring to the AI (e.g., AI output seen as the default, human review seen as a formality).

  4. Tasks are high-stakes but verification is weak or absent

    • In medical, legal, or safety-critical applications where ground truth is delayed or hard to observe, detailed CoT can create a false sense of rigor.
    • Decision-makers may anchor on the AI’s chain and underweight dissenting human judgment, especially under time pressure.

  5. CoT length and complexity exceed users’ ability to check it

    • Very long or intricate chains cause users to skim rather than verify, leading to a “looks good” heuristic.
    • Users interpret the mere presence of complex reasoning structure (equations, jargon, multi-step arguments) as evidence of correctness.

In practice, CoT improves human accuracy when users can and are nudged to scrutinize the reasoning, the system is reasonably calibrated and faithful, and the environment supports verification and feedback. It increases over-trust when users cannot effectively critique the reasoning, the explanation is more persuasive than it is accurate, and UI or organizational incentives push people to treat detailed chains as proof of correctness rather than hypotheses to be checked.