Demand Characteristics in UX Research

...and what can we do about it?

Does taking part in a study alter the way participants behave? Research suggests that people change their natural behaviour so that it matches their interpretation of the aims of a study. More specifically, participants are often aware of or trying to guess what the researcher is trying to investigate, which makes them more likely to behave in a way they are expected to behave. This happens even in cases where participants are not explicitly told what the aim of a study is. In particular, Martin Orne argued that in an experimental situation there are many cues that can give participants an idea of what the study is about and what behaviour is expected of them. Those cues can sometimes reveal the hypothesis to the participant and are known as “demand characteristics” (Orne, 1962).

Isn’t that similar to “the Hawthorne effect”? Well, yes. Sometimes you see the terms “Hawthorne” and demand characteristics effects used interchangeably (see Mummolo, & Peterson, 2019). Even though they are related effects they have some subtle differences. When we’re talking about the Hawthorne effect we are referring to changes in participant behaviour due to the knowledge they are being observed. Demand characteristics on the other hand refer to the participants’ efforts to validate a researcher’s hypotheses. Another term that is often confused with demand characteristics is the “social desirability bias”; the tendency people have to give responses that will be viewed favourably by others. This bias can result in behaviour that may or may not coincide with the researcher's aims.

Even though demand characteristics were first identified in lab-based studies, UX research studies are often hypothesis-driven, suggesting they can be susceptible to their effects. This can be a bigger concern in usability studies, in particular in studies observing more naturally occurring tasks (Draper, 1993; Hornecker & Nicol, 2012).

What effect do “demand characteristics” have on research?

Participants could react to demand characteristics in several ways. They may play the role of the good participant and attempt to detect the experimenter’s hypothesis and confirm it. Orne demonstrated that by showing that participants were willing to do a boring experimental task for over 5 hours (!) just to help progress science and satisfy the experimenters!

Participants, however, don’t always play the role of the good participant. Masling (1966) suggested that in certain cases participants, they do the opposite — also known as “screw-you effect”). In these cases, participants behave in a way that discerns the hypotheses of the study.

Weber and Cook (1972) also identified the roles of the faithful participant, who follows the instructions given by the experimenter to the letter, and the apprehensive participant, who is so concerned about the evaluation of their answers by the experimenter that they behave in a socially desirable way.

What can we do?

“It is futile to imagine an experiment that could be created without demand characteristics. One of the basic characteristics of the human being is that he will ascribe purpose and meaning even in the absence of purpose and meaning. In an experiment where he knows some purpose exists, it is inconceivable for him not to form some hypothesis as to the purpose, based on some cues, no matter how meagre...” (Orne, 1962, p.780).

According to Orne, demand characteristics are unavoidable, because people will always try to make sense of things. One of the most popular (but controversial) ways psychologists have been trying to counteract the effect of demand characteristics is deception. Experimenters try to conceal the true aim of their experiment. This can be done by providing purposefully vague information (incomplete disclosure) or by giving participants false information (deception). Two famous studies that employed deception are Milgram’s (1974) study of obedience and Zimbardo’s (1973) prison study. This approach, however, is not recommended for UX research as it creates ethical problems and potential risks…!

A method widely used in academic and medical research is the “double-blind” approach. In double-blind studies, the person who comes into contact with the participant is not aware of the hypotheses of the study. This minimises the number of available cues participants have about the purpose of the study. In the context of UX, this could be achieved by hiring an external agent or by having a user researcher or someone from the team who wasn’t involved in building the product/prototype conduct the research. Participants are more likely to play the role of the good participant if they know that the moderator was involved in the creation of the product.

Being aware of the potential impact of demand characteristics on our research is important in minimising their effect. This is something we should start considering when designing a study. For example, certain types of experimental design such as “repeated measures (or within participants)” (the same participants take part in all conditions) are more susceptible to demand characteristics. A way to minimise the effect is using a “between-participants” design (different groups take part in each condition). During the study we should pay attention to any cues we reveal to the participant — try to remain as neutral as possible.

Recruiting the right participants for UX research can also help us reduce the effects of demand characteristics. In particular, user testing a product with internal users (employees working in the company developing the product) is more likely to suffer from this effect as employees are more likely to play the role of the good participant and perform in a way they think is expected of them. For example, they are more likely to perform tasks in a different way than they normally would or persevere in an attempt to complete a task for much longer than usual. Similar behaviour might be seen in external users if they are fans of the product. For example, avoid conducting research only with your most active users or your promoters.

Methodological triangulation – or cross-examination – is another way we can overcome demand characteristics effects (as well as other biases!). We should aim to collect insights from multiple perspectives and use multiple methodologies to answer a research question. For example, we can complement usability testing with questionnaires, observational techniques, and product analytics. Agreement and consistency among results we obtain can improve the validity and the generalisability of our findings.

How concerned should we be?

Surprisingly, there aren’t many studies investigating the impact of demand characteristics in user research. Even though they probably play a role, there are cases where the effect is smaller than originally thought. For example, a recent study by Mummolo and Peterson (2019) found that in online surveys participants exhibit a limited ability to adjust their behaviour to align with researcher expectations even when the aim of the study was revealed to them. Providing an incentive slightly increased the strength of the effect but the finding wasn’t consistent in all the studies they conducted.

References

Draper, S. W. (1993, April). The notion of task in HCI. In INTERACT'93 and CHI'93 Conference Companion on Human Factors in Computing Systems (pp. 207-208).

Holleran, P. A. (1991). A methodological note on pitfalls in usability testing. Behaviour & Information Technology, 10(5), 345-357.

Hornecker, E., & Nicol, E. (2012, June). What do lab-based user studies tell us about in-the-wild behavior? Insights from a study of museum interactives. In Proceedings of the Designing Interactive Systems Conference (pp. 358-367).Chicago

McCambridge, J., De Bruin, M., & Witton, J. (2012). The effects of demand characteristics on research participant behaviours in non-laboratory settings: a systematic review. PloS one, 7(6), e39116.Chicago

Mummolo, J., & Peterson, E. (2019). Demand effects in survey experiments: An empirical assessment. American Political Science Review, 113(2), 517-529.

Orne, M.T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776-78