Understanding the Hawthorne Effect in UX Research
What is it and what can we do about it?
The Hawthorne Effect refers to the phenomenon in which individuals modify their behaviour due to being observed or knowing that they are being observed. This effect was first identified during a series of experiments by a group of industrial researchers at the Hawthorne works of the Western Electric Company in Cicero, Illinois (Roethlisberger & Dickson, 1939). In this series of experiments, researchers discovered that workers' productivity increased simply because they were aware of being studied. The original Hawthorne study was conducted to examine the impact of changing light levels at the Hawthorne Works on work practices. However, the increased attention on the workers during the study led to a temporary boost in productivity (which became known as “the Hawthorne Effect”) rather than any changes in work practices1.
Research since then has shown that the Hawthorne Effect can impact various types of behaviours, such as dietary habits and hygiene practices, or even alter the results of clinical trials. It can also influence the results of research studies, including UX research. For example, in the context of a moderated usability study the Hawthorne Effect could lead users to perform tasks better than they would if they were not being observed. This could be a problem for UX.
Why does it occur?
There are a few possible explanations for the Hawthorne Effect. Social psychologists propose that the Hawthorne effect happens because people become aware that they are being observed or having their behaviour judged. This makes them think that the researcher has certain expectations of them. They then act in line with those expectations as they desire to fit in and be socially accepted (Does this sound familiar? That’s because it relates to Demand Characteristics that we have covered previously).
Another possible explanation is social comparison theory, which proposes that individuals often compare their own behaviour and performance to that of others in order to evaluate themselves. When individuals are aware they are being observed, they may feel a sense of pressure to perform better and to conform to expectations.
The Hawthorne Effect is a form of the observer effect, which suggests that researchers are interacting with the system, usually through the instruments of measurement, and changing the phenomena being studied. This is something not limited to social sciences but can also occur in “hardest” natural sciences (e.g., physics).
"In spite of popular conceptions to the contrary, laboratory and quantitative methods are not more objective or less biased than field and qualitative methods. Mathematics and physics are no less socially constructed than anthropology and sociology... All knowledge is contingent on the interests of the scientists creating it, the tools and procedures they use to measure the phenomena under investigation, and the analytic frameworks they use to interpret their results”
What does this mean? Well, we’re entering philosophy territory but achieving unbiased and objective knowledge is not possible. Everything is filtered through our imperfect lenses and affected by the tools and methods we’re using. To put this in a UX research context, if your stakeholders challenge your findings by bringing up the Hawthorne Effect remind them that this is something going beyond user research and that you’ve already taken all the measures you could to limit its effect. No methodology is without fault — it is important to acknowledge this and try our best to improve our study design.
Alternative explanations of the Hawthorne Effect
There is some evidence against the Hawthorne Effect, which suggests that the change in behaviour observed in participants is not solely due to their awareness of being observed. Other factors, such as changes in the work environment, the attitudes and expectations of participants, and the influence of other variables, may also play a role in producing the observed changes in behaviour.
Critics of the Hawthorne Effect argue that the changes in behaviour could be attributed to other factors, such as improved working conditions, increased job satisfaction, or a general increase in motivation, rather than simply being due to increased attention from researchers. For example, Parsons criticised the original studies and suggested that the Hawthorne Effect was more accurately attributed to the learning and feedback mechanisms present during the study. Normally, the workers did not receive daily feedback on their productivity, but during the study, the experimenters collected data and provided feedback on their performance. This caused the workers to become more engaged in setting and achieving higher goals, leading to increased satisfaction in their work.
What can we do?
In the field of user experience (UX) research, the Hawthorne Effect can have a significant impact on the validity of research results. When users are aware that they are being observed or studied, their behaviour may change, resulting in behaviour that is different from what would be seen in a normal setting. This can lead to inaccurate or misleading data and insights.
For example, in a usability study of a website, a user may navigate and interact with the site more cautiously and deliberately because they know they are being observed and that their actions are being recorded. As a result, the results of the study may not accurately reflect the user's typical behaviour when using the site.
To mitigate the effects of the Hawthorne Effect, we can take the following steps:
Use control groups to minimise observation effects. For instance, a usability study could be conducted with two participant groups. One group, referred to as the "test group," would use the new design, while the other group, referred to as the "control group," would use the original design2. If the Hawthorne Effect were to occur, it would impact both groups, and thus any improvement in the performance of the test group would not be attributed to the Hawthorne Effect. This can be time consuming and costly to do for each study but it’s worth considering if you’re planning a major redesign or dealing with demanding clients.
Don’t give feedback to the participants during the study. Since feedback and learning mechanisms are usually involved in cases the Hawthorne Effect is observed, minimising that can help us mitigate its effects. Feedback in a usability study could take the form of confirming whether a participant has performed an action as expected or giving them hints about how to perform a specific task.
Focus on cause-and-effect relationships behind significant findings: According to Macefield, the best way to achieve this is through the use of qualitative methods, such as verbal protocols and pre- and post-test semi-structured interviews.
Minimise the visibility of the observation process. This can be done by opting for remote studies and or using unobtrusive cameras or product analytics. Remote unmoderated research is another way to do this.
Create a judgement free environment: Assure participants that their actions will not be judged and that there are no right or wrong actions. Encourage participants to use the product or service as they normally would, rather than attempting to follow specific instructions.
Encourage anonymous feedback: this encourages users to be more honest.
Triangulate. I covered this on a previous newsletter but it’s worth reminding you that triangulation allows us to have more confidence in the research data, can reveal unique unexpected findings, reduce bias, and allows us to understand a phenomenon more clearly.
As Macefield (2007) states it is important to note that there are significant differences between the studies conducted at the Hawthorne Works and typical usability studies, therefore caution should be exercised when applying any interpretation of the Hawthorne Effect to the field of usability. We can never remove all bias from our research, but being aware of it and using mitigation strategies when needed can help.
The Hawthorne Effect isn’t always positive. Studies have also observed “the Negative Hawthorne Effect”
When using control groups it’s important to ensure participants are not aware they’re part of a control room to avoid additional biases.