Why Users Ignore AI Explanations (And What We Can Do About It)

A recent study reveals why well-designed AI explanations often go unused and what it means for UX practice

Generated with Chat GPT

As AI systems become more sophisticated and opaque, the field of explainable AI (XAI) has emerged to address a fundamental problem: how can users appropriately trust and rely on systems they don't understand? The core premise is that by providing insights into AI reasoning users can make more informed decisions about when to trust AI recommendations and when to override them. Some examples of these insights include visualisations, confidence scores, feature importance, and decision trees.

This transparency is supposed to solve several critical issues such as reducing over-reliance on flawed AI systems, increasing adoption of helpful AI tools, and enabling users to catch AI errors before they cause problems. The theory makes intuitive sense — if you understand how the AI reached its conclusion, you're better equipped to evaluate whether that conclusion is reasonable¹.

XAI research typically focuses on different levels of explanation, from basic performance information (what researchers call "Level 1") to more complex reasoning about why the AI made specific decisions. Most XAI evaluation, however, happens in controlled settings, which don’t always mimic real world usage. It’s not clear whether these explanations actually help users in real-world conditions where they might be stressed, distracted, and juggling multiple priorities.

What if users simply don't engage with these carefully designed explanations when it matters most? A new study by Alami et al. (2025) attempted to answer this question by observing how people actually use AI explanations under realistic working conditions. The findings suggest that even well-designed explanations often go unused when users face competing demands and time pressure, exactly when those explanations might be most valuable.

Testing AI Explanations Under Pressure

The researchers created a simulation that mimicked high-stakes, multi-tasking environments. Thirty participants simultaneously managed an AI-assisted UAV (drone) routing task (avoiding no-fly zones) and a manual target detection task (spotting flares in video feeds). The setup varied both in cognitive workload (4, 8, or 12 UAVs) and task priority to see how these factors affected AI explanation usage.

The AI system provided three key features: alerts when drones needed rerouting, visual explanations showing drone paths to verify those alerts, and automatic rerouting capabilities. Importantly, the alerts had a realistic error profile with low miss rates but noticeable false positives, mimicking many real-world AI systems.

The main research question was whether people would actually use the explanations to verify AI alerts, especially when busy or when the AI-assisted task wasn't their top priority.

The results gave some insight into how cognitive load and task priority affect AI explanation usage. More specifically, when workload increased, participants relied more heavily on AI alerts but were significantly less likely to verify them using the available explanations. Under high cognitive load, people detected fewer AI misses and identified fewer false alerts, yet explanation usage remained consistently low across all conditions (around 10%)

Task priority emerged as a stronger predictor of explanation usage than workload itself. People accessed explanations more frequently when the AI-assisted task was high priority, but the absolute numbers remained low regardless of conditions.

When the AI task was low priority under high workload, participant performance dropped below baseline levels. Participants would have been better off ignoring the AI assistance entirely, but the combination of cognitive pressure and competing priorities led to poor AI reliance patterns.

Implications for UX

These findings challenge some common assumptions in AI design. The study suggests that simply providing explanations isn't enough — we need to consider when and how users will actually engage with them under realistic conditions.

For UX researchers, this points to a gap in how we typically evaluate AI systems. Most usability studies test AI explanations in controlled, low-stress environments where participants can focus exclusively on the AI interaction. Real users, however, are distracted, multitasking, and operating under various pressures. Testing AI systems in isolation may miss critical usage patterns that emerge in realistic contexts.

This is consistent with previous studies on automation-related complacency and automation bias that showed that users often over-rely on AI recommendations while ignoring verification mechanisms, particularly under high workload conditions (Parasuraman & Manzey, 2010). It’s worth noting that this occurs in both experienced and naive users.

The research also highlights the importance of measuring actual behaviour rather than stated preferences. Users might express appreciation for AI explanations in interviews but consistently ignore them in practice, especially when cognitive resources are strained.

For designers, the findings suggest that optional explanations may not be sufficient. If users predictably skip verification under pressure, design patterns need to account for this reality. This might mean building verification into required workflows, reducing AI false positive rates more aggressively, or adapting explanation complexity based on user context.

The study also reinforces the importance of task prioritisation in AI system design. When users treat an AI-assisted task as lower priority, they're more likely to over-rely on AI suggestions without proper verification, even when explanations are readily available.

1. I refuse to stop using em dashes…