Embedding Equality, Diversity, and Inclusion in Usability Testing

Transforming Usability Research to Ensure Accessible and Intuitive Digital Interfaces for All

Image generated with ChatGPT

Imagine a world where every digital interface, from smartphone apps to complex industrial systems, is intuitive and accessible to all users, regardless of their background, abilities, or life experiences. Now consider the reality: countless individuals struggle daily with technologies that weren't designed with their needs in mind. This disconnect isn't just frustrating — it's a barrier to full participation in our increasingly digital society.

This is also one of the biggest challenges in usability research. While it has long aimed to improve user experiences, traditional approaches have often fallen short in representing the true diversity of users. This article explores the importance of embedding equality, diversity, and inclusion (EDI) principles in usability testing, drawing inspiration a recent paper by Rutter et al. (2024).

The Evolution of Usability Testing

Usability testing has come a long way since its inception in the early days of computer interface design. Nielsen (1993) defined usability as encompassing learnability, efficiency, memorability, errors, and satisfaction. However, as our understanding of user diversity has grown, so too has the need to expand our approach to usability testing.

Shneiderman's (2000) concept of "universal usability" marked an important shift towards considering a wider range of users in design and evaluation. This idea has since evolved into more nuanced approaches that recognise the intersectionality of user identities and experiences (Schlesinger et al., 2017).

The Importance of Diverse Representation in Usability Testing

Recent research has highlighted the significant gaps that can occur when usability testing fails to include diverse participants. The most common ones are:

1. Accessibility issues: Studies have shown that many websites and apps still fail to meet basic accessibility standards, disadvantaging users with disabilities (Power et al., 2012).

2. Cultural biases: Design choices that seem intuitive to one cultural group may be confusing or offensive to another (Reinecke & Bernstein, 2013).

3. Gender and age disparities: Technologies designed primarily by and for young, male users often fail to meet the needs of women and older adults (Oudshoorn et al., 2004).

4. Socioeconomic factors: Users from different socioeconomic backgrounds may have vastly different technology experiences and needs (Harris et al., 2017).

Key Principles for Inclusive Usability Testing

Rutter et al. (2024) conducted a multiphase study that brought people together from different backgrounds to discuss EDI in usability testing, focusing on people whose identities can be underrepresented in usability testing, and usability researchers. After analysing the interviews, roundtable discussions, and participant feedback they identified three key principles for embedding EDI in usability testing:

1. Sense of value: Participants need to feel that their contributions are meaningful and respected. This aligns with broader principles of participatory design (Muller & Kuhn, 1993) and co-creation (Sanders & Stappers, 2008), which advocates for collaborative creation with users rather than for them.

In practice, implementing this principle involves a fundamental shift in how we view and interact with participants. It's not enough to simply recruit diverse users; we must actively recognise and honour their expertise and contributions. This can be achieved through several key strategies:

• Fair compensation: Go beyond token payments. Consider the true value of participants' time and lived experience. This might involve higher monetary compensation, but could also include other forms of recognition such as acknowledgment in final reports or invitations to product launch events.

• Communicate the value participants add in the study: Recognise their expertise and lived experience.

• Collaborative research design: Involve participants in setting research questions and designing tasks. This could take the form of pre-study focus groups or individual consultations with community members. By doing so, we ensure that the usability tests are addressing issues that are genuinely relevant to diverse user groups.

• Impact communication: Clearly articulate how the study might improve technologies or services as well as the participants. This involves not just explaining the immediate goals of the usability test, but also providing a broader context of how the research fits into the product development cycle and potential real-world impacts.

• Outcome sharing: Establish a system for sharing research outcomes with participants. This could involve creating accessible summary reports, holding debrief sessions, or even involving participants in the presentation of findings to stakeholders. This approach not only demonstrates respect for participants' contributions but can also provide valuable additional insights during the interpretation phase.

2. Trust: Establishing trust is crucial, especially when working with marginalised communities that may have historical reasons to be wary of research (Christopher et al., 2008). Establishing trust requires a comprehensive approach that spans the entire research process:

• Transparency: Be open about research goals, processes, and potential outcomes from the outset. This includes being clear about who is funding the research, how the data will be used, and any limitations or potential negative impacts of the study.

• Data privacy and ethics: Implement robust data protection measures and clearly communicate these to participants. This goes beyond standard confidentiality agreements to include explanations of data storage, access, and deletion policies. Consider allowing participants to review and approve any direct quotes before publication.

• Safe testing environments: Create comfortable and accessible testing spaces and check that your participants are feeling safe during and after the study.

• Accessible communication: Use clear, jargon-free language in all communications. Provide information in multiple formats (e.g., written, audio, video) and languages as needed. Consider working with community liaisons to ensure cultural appropriateness of all materials.

• Ongoing consent: Implement a process of continuous consent throughout the study. This means regularly checking in with participants about their comfort levels and willingness to continue, and being prepared to pause or stop the test if needed.

3. Agency: Empowering participants to make decisions about their involvement respects their autonomy and can lead to more authentic insights. This principle aligns with the concept of "user-sensitive inclusive design" proposed by Newell et al. (2011) and is crucial for ensuring that participants feel comfortable and valued throughout the research process. This can be achieved by:

• Offer flexible participation options (e.g., remote vs. in-person, timing): This could involve setting up secure video conferencing for remote sessions, providing options for asynchronous participation (e.g., diary studies or take-home tasks), and allowing participants to choose their preferred time slots, including evenings or weekends. Also, design studies to allow participants to take their time and have breaks, if needed.

• Allow participants to choose how they share information: For example, be open to unexpected forms of feedback and allow participants to share information in a format they feel more comfortable with.

• Provide ongoing opportunities to give feedback on the research process: Create multiple channels for participants to provide feedback on the research process itself (e.g., regular check-ins during the study, post-study debriefs)

Previous research has also identified a number of additional principles:

• Intersectionality: Recognising the complex, intersecting identities of participants is crucial for truly inclusive research (Schlesinger et al., 2017). This approach acknowledges that individuals' experiences are shaped by the interaction of multiple aspects of their identity, rather than by single demographic categories. Some ways to achieve this is by:

  • Avoid reducing participants to single demographic categories: Move beyond simple demographic categories in your participant screening. Use open-ended questions that allow participants to describe their identities in their own words. Develop complex, multifaceted personas that reflect the intersectionality of real users.

  • Consider how different aspects of identity interact in technology use: Look for patterns that emerge at the intersection of multiple characteristics (e.g., age and socioeconomic status, or gender and disability). Be cautious about generalising findings to entire demographic groups

  • Be mindful of power dynamics in the research setting: Consider how your own identity and position might influence participant responses. Be mindful of historical and cultural contexts that might affect trust and openness

• Accessibility: Ensuring that the usability testing process itself is accessible to all participants is fundamental (Henry et al., 2014). This goes beyond just testing for accessibility – it means making the entire research process inclusive from start to finish. Some approaches involve the following:

  • Offer materials in multiple formats (e.g., large print, screen reader-compatible): Ensure that digital platforms used for remote testing (e.g., video conferencing tools) are compatible with assistive technologies

  • Provide interpreters or assistive technologies as needed: Have a range of assistive technologies available for participants to use during testing (e.g., screen readers, alternative input devices)

  • Design flexible testing protocols that can adapt to different abilities. For example, break tasks into smaller, manageable steps that can be completed at the participant’s pace.

These recommendations are just a starting point and should be adapted to each study's objectives and target participants.

Challenges and Solutions

As most UXers know, implementing these principles is not without challenges. Rutter et al. (2024) identified several obstacles faced by researchers, including organisational pressures, getting guidance, stakeholder culture, and recruitment difficulties.

The challenges for researchers when embedding equality, diversity and inclusion in usability testing. Rectangles indicate primary themes and ovals the sub-themes. Image from Rutter et. al, 2024

Addressing these challenges requires a multi-faceted approach:

1. Organisational Change: Adopting a learning orientation within organisations can support more inclusive practices (Davies & Nutley, 2000). This involves:

   1. Fostering a culture of continuous improvement

   2. Allocating resources for EDI initiatives

   3. Providing training and support for researchers

2. Stakeholder Education: Educating stakeholders about the value of inclusive usability testing is crucial. This can involve:

   1. Sharing case studies demonstrating the business and social impact of inclusive design

   2. Highlighting legal and ethical obligations around accessibility

   3. Framing EDI as a driver of innovation and market expansion

3. Improved Recruitment Strategies: Reaching diverse participants requires thoughtful, targeted approaches:

   1. Partnering with community organisations and advocacy groups

   2. Using inclusive language in recruitment materials

   3. Offering multiple channels for participation (e.g., online, in-person, phone)

4. Methodological Innovation: Developing new methods that centre EDI principles:

   1. Exploring participatory action research approaches (Kemmis et al., 2013)

   2. Incorporating critical race theory and feminist perspectives into usability research (Bardzell, 2010)

   3. Utilising mixed methods to capture both quantitative and qualitative insights

Conclusion

Having experienced multiple obstacles in applying EDI principles in my own work, it’s encouraging to see more research on this topic. In particular, Rutter et al.’s recommendations come from the perspectives of underrepresented individuals, distinguishing them from guidelines typically informed by researchers' interactions with participants. As researchers (and UXers in general), it is imperative that we consider these universal principles to advance inclusive usability research and ensure that digital interfaces are accessible and intuitive for all users.