Making Sense of the ChatGPT User Experience
A brief literature review
The advent of conversational AI agents like ChatGPT represents a paradigm shift in human-computer interaction. These tools enable natural language conversations, aiming to provide intuitive and useful responses on any topic. As this emerging technology promises to transform user experience (UX), understanding its impact from the user's perspective is essential. A small number of recent studies have begun investigating how people interact with and perceive conversational AI using methods like content analysis, surveys, and experiments. This article reviews the existing literature and discusses their implications for our field.
Examining the UX of ChatGPT
Only a small number of studies have looked into issues related to ChatGPT UX. Choi et al. (2023) performed a content analysis of 452 Reddit posts to explore ChatGPT uses and user sentiments. The researchers found diverse applications for the tool, spanning writing, academic work, and everyday tasks. However, users also expressed concerns about issues like misinformation and plagiarism risks, highlighting the need for ethical guidelines. For instance, some users warned that ChatGPT confidently provides erroneous information, underscoring the importance of verifying quality.
Skjuve et al. (2023) were interested in exploring positive and negative experiences users have with ChatGPT. They administered an online survey to 194 ChatGPT users and used thematic analysis to identify key themes. They found that pragmatic attributes, features that enable productive outcomes like efficiency and useful information, are highly influential on user experience. For instance, users appreciated how ChatGPT eased work tasks by providing detailed, customised information. Hedonic attributes like enjoyable creative interactions also emerged as relevant. This shows the importance of both productivity and fun even during early adoption.
Other studies are more focused on how ChatGPT performs in comparison to existing products and platforms. Xu et al. (2023) conducted an online experiment, comparing ChatGPT and Google Search. They evaluated user experience metrics including perceived usefulness, enjoyment, satisfaction, and ease of use. On fact retrieval tasks, ChatGPT users spent less time with comparable performance, demonstrating enhanced productivity. However, ChatGPT struggled with nuanced prompts and fact checks, unlike Google which excelled at keyword search. Despite slightly higher perceived information quality, users showed similar overall trust in both tools. Interestingly, usefulness, enjoyment, and satisfaction were significantly higher when using ChatGPT, although ease of use was similar. This reveals some potential UX advantages rooted in ChatGPT's conversational nature, making it more accessible to users.
Sakirin & Said (2023) compared the ChatGPT experience to traditional communication methods such as phone support and email. The researchers surveyed 175 participants aged 18-55 about their preferences and found that 70% chose ChatGPT over traditional techniques, citing greater convenience, efficiency, and personalisation. Participants also reported much higher satisfaction with ChatGPT — 85% had a positive experience versus only 50% for traditional methods. ChatGPT responses were more accurate at 90% vs 70% for traditional techniques. ChatGPT was significantly faster too, averaging 3 second response times versus 5 seconds for traditional methods.
Implications for UX Field
The research shows that while conversational AI has many benefits, it also comes with notable risks that need to be addressed. For example, an issue that came up in multiple studies was that of information quality and misinformation. The topic of AI hallucinations has been in the media since ChatGPT started gaining popularity; Large Language Models regularly generate outputs that do not reflect true information, essentially "hallucinating" content that is not real. To tackle these issues, we should consider using build in features that encourage users to evaluate the information critically. This could include adding credibility indicators that show how reliable a response is, as well as using multi-step prompts that walk users through verifying facts.
Additionally, being transparent about what the AI can and cannot do will help set proper expectations so people do not overly rely on it. The key is to strike a balance - leverage the strengths of conversational AI while also guiding users with thoughtful design to minimise harm. If designed well, these tools can be incredibly useful while still being safe.
More research needed…
It’s an exciting time to be a UX professional. As the adoption of AI tools grows, understanding and improving UX becomes crucial. Here are some types of research that we can do to achieve this.
Comparative usability tests having users complete representative tasks with and without the conversational AI.
Surveys, interviews, and observational data will provide insights into subjective metrics like perceived satisfaction, naturalness of interaction, and ability to meet user needs.
If you develop AI products, iteratively testing with a diverse participant pool uncovers preferences and behaviours across demographics.
Longitudinal studies assess how utility evolves with sustained use. It can also control for any novelty effects current studies might be suffering from.