Gamification in UX Research
Using game elements to gather user insights
Gamification refers to the use of game elements in non-gaming contexts (e.g., education). Game elements have been successfully employed in various applications and products aiming to improve learning and lead to behavioural change (Smiderle et al., 2020, Hervas et al., 2017). A potential explanation behind the positive effect of gamification is the fact that games can provide a safe environment for users to fail and to make mistakes (Slegers et al., 2015).
There are a few examples of designers employing game elements to encourage idea generation. For example, design games are common in the field of participatory design (PD) and have been shown to enhance communication and teamwork, and improve the articulation of user requirements (Brandt & Messeter, 2004).
More recently, user researchers have started employing game elements for data collection. In these cases, game elements are used to improve participant engagement and identify their needs in the early phases of the design process. These games have been referred to as research games (Geerts et al., 2018). Both design and research games have similar aims; involving end-users to improve the design. However, according to Geerts and colleagues: “research games are used much earlier in the design process to facilitate better participation when collecting insights (as opposed to communicating insights) by integrating several game elements to this end”.
Why use game elements in user research?
Providing a safe environment is crucial in UX research. Users are more likely to open up to the researcher and communicate any issues and concerns they have when they are in a relaxed environment. Games are seen as environments in which it is safe to experiment (Gee, 2003; Slegers et al., 2019). As a result, games could be used in user research in order to create an environment that allows users to feel free to fail and express their opinions freely. In addition to this, game-based approaches have been shown to prevent socially desirable responses and biases often seen in traditional testing (Panagiotidi, 2017; Montefiori & Panagiotidi, 2018; Lochner et a., 2017)1.
Another potential benefit of gamification in user research is that it could ease the unfamiliarity that is common in groups of participants in workshops (Slegers et al., 2019) and even act as an ice-breaker (Geerts et al., 2018). Having clear rules and employ techniques such as turn-taking can also help reduce the role of individual participants who tend to dominate group conversations, resulting in more voices being heard.
Games often use narratives (Mitgutsch & Alvarado, 2012). Narratives allow researchers to present to users a number of possible future experiences and can be used to introduce futuristic elements (e.g., new products or services) without needing to justify their feasibility and credibility (Geerts et al., 2018). When future scenarios are presented through a game setting it is easier to explore possible future user experiences. This could lead to insights that cannot be generated through traditional research methods.
A case study
Not many case studies on research games have been published. One of the first ones was by Slegers and colleagues (2015) who used gamification as a technique in UX research. Their methodology is presented below.
The goal of their project was to develop a train information system (TIS) that would integrate multiple sources of information (e.g., travel information, weather, how crowded the station is, waiting times at various shops and bars there) and personalise it based on the profile of the traveller. The traveller profile was constructed using information about an individual’s travel preferences and goals, as well as their actual behaviour.
In the first stage of the research, they conducted qualitative user research (observations and stimulated recall interviews) to discover travellers’ current train experience and their needs (before, during, and after their train journey). They also wanted to understand travellers’ needs regarding the future TIS concept. However, this information was difficult to uncover using traditional research methods, and that’s when gamification came into play. They invited experienced train travellers to play a board game, which was designed to represent a fictitious door-to-door train trip. Here’s a brief description of the game by Slegers et al.:
“The starting point was home, from which the players had to get to the train station, catch a train, connect to another train, get off the train, and reach their final destination. During the game, players could ask questions to a futuristic, omniscient TIS via a chat program on a tablet. One of the researchers would answer the questions remotely, Wizard of Oz style (players were not informed about the human nature of the TIS beforehand).”
Before inviting external users, the team conducted a pilot session with internal users (who were also frequent train travellers) to test the game. This led to some changes and fine-tuning.
After the pilot, 30 participants were recruited and two workshops were organised. The sessions were recorded and transcribed. After the workshops, the participants were asked to fill in a short online survey, which included eight questions about the participants’ general attitudes towards board games and their experiences during and after the workshop. The data collected were used to identify underlying user needs as well as examine how participants felt about using game principles as a research technique. The video below explains in more detail how each session was run, what the game involved, and what data were collected.
Does using a game lead to new insights?
In the case study reported by Slegers and colleagues, including a board game in the initial phase of user research led to new insights that had not surfaced in previous user research (using 'traditional' observations and interviews). More specifically,
Playing a game that stimulates a train travel experience allowed researchers to focus more explicitly on what is on train travellers’ minds during each phase of a train trip. Using game elements made the participants' tacit knowledge visible. Tacit (or latent) knowledge refers to personal, private knowledge that is not necessarily available for conscious introspection (Reber, 2001). As a result, it is difficult to measure with traditional UX research methods.
Several of the needs identified through the analysis of the game board data were new and had not been previously revealed by the interview conducted by the team. These needs were very useful to the design team as they were specifically related to the future TIS under development.
The game workshops resulted in discussions of some event-related topics, which had not come up during the interviews or the observations. Using a board game uncovered user needs related to important, but infrequent events (e.g., safety).
How do participants feel about the gamified testing?
Results from the short survey administered by Slegers et al. suggested that most participants had positive attitudes towards this new approach. In particular, the majority of users indicated that “they felt at ease while playing the game” and that “they could say what they wanted to say”. Some participants felt that playing a game facilitated discussion and sharing. Overall, the game was seen as a pleasant experience. The majority of the participants stated that they like playing board games in general (17 out of 20). However, games are not for everyone. More specifically, 3 out of 20 participants in the Slegers et al. study expressed disappointment when they found out they’d be playing a game.
Limitations of this approach
Games are rich in unwanted variance and diversity, which makes them difficult to control (Gundry & Deterding, 2019). This is a potential threat to the validity of this approach. Using clear rules and experienced moderators can help with this.
Gamification does not appeal to everyone. Certain groups are more likely to enjoy game elements (Hamari, 2013) and this must be taken into account when deciding to use game elements in user research.
Creating good research games is hard. Research games are a genre of serious games - games with a purpose whose primary objective is not fun or entertainment. Most user researchers are not trained to develop games and a solid framework and collaboration with domain experts are required (Wetzel et al., 2019).
“Making a good game is hard. Making a good serious game is even harder. The reason it is so difficult is that rather than simply trying to optimize the entertainment aspect of the game, or the so-called fun factor, one must also optimize to achieve a specific set of serious outcomes.” (Winn, 2009, p. 1011)
References
Brandt, E., Messeter, J. (2004). Facilitating collaboration through design games. In Proceedings of the eighth conference on participatory design: Artful integration: Interweaving media, materials and practices-Volume 1 (pp. 121-131). New York, NY, USA: ACM.
Geerts, D., Nouwen, M., Van Beek, E., Slegers, K., Chocron Miranda, F., & Bleumers, L. (2019). Using the SGDA framework to design and evaluate research games. Simulation & Gaming, 50(3), 272-301.
Gundry, D., & Deterding, S. (2019). Validity threats in quantitative data collection with games: A narrative survey. Simulation & Gaming, 50(3), 302-328.
Hamari, J. (2013). Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service. Electronic commerce research and applications, 12(4), 236-245.
Hervas, R., Ruiz-Carrasco, D., Mondejar, T., & Bravo, J. (2017, May). Gamification mechanics for behavioral change: a systematic review and proposed taxonomy. In Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare (pp. 395-404).
Lochner, K., Montefiori, L., Preuss, A., & Panagiotidi, M. (2017). Gamified and “Classical” Psychometric Assessment–Convergent, Divergent, and Incremental Validity. In Society for Industrial and Organizational Psychology Conference (SIOP), 27-29/04, Florida, USA
Montefiori, L., & Panagiotidi, M. (2018) Does coefficient size really matter? The impact of social desirability and common method variance on the construct validity of game-based assessments. In Division of Occupational Psychology Annual Conference 2018, Stratford-upon-Avon.
Panagiotidi, M. (2017) Gaming the game: Game-Based Assessment and self-presentation bias. In Division of Occupational Psychology Annual Conference 2017, 4-6/01/17, Liverpool, UK
Reber, A. S. (2001). Tacit Knowledge, Psychology of. In International Encyclopedia of the Social & Behavioral Sciences (pp. 15431-15435).
Slegers, K., Ruelens, S., Vissers, J., & Duysburgh, P. (2015, April). Using game principles in ux research: A board game for eliciting future user needs. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1225-1228).
Smiderle, R., Rigo, S. J., Marques, L. B., de Miranda Coelho, J. A. P., & Jaques, P. A. (2020). The impact of gamification on students’ learning, engagement and behavior based on their personality traits. Smart Learning Environments, 7(1), 1-11.
Warr, A., & O'Neill, E. (2005, April). Understanding design as a social creative process. In Proceedings of the 5th Conference on Creativity & Cognition (pp. 118-127).
Wetzel, R., Bachour, K., & Flintham, M. (2019). Tensions within the ministry of provenance: Reflections on co-creating a research game together with artists. Simulation & Gaming, 50(3), 329-358.
Winn, B. M. (2009). The design, play, and experience framework. In Handbook of research on effective electronic gaming in education (pp. 1010-1024). doi:10.4018/978-1-59904-808-6. ch058
I did some research on this area while working at Arctic Shores and while I was in academia - get in touch if you want to talk gamification and psychometrics.
Cool post on the topic, informative! In this article https://gapsystudio.com/blog/gamification-in-ux-design/ more about gamification in UX, areas, and methods of its implementation.