Online Reviews: An Untapped Resource for UX Research?
A guide to effectively leveraging online user reviews for UX insights
Online user reviews on platforms like app stores, travel sites, and e-commerce sites can become a free source of user experience (UX) data. For UX researchers and designers, comprehensively mining this data represents an opportunity to understand user pain points, desires, and perceptions. Reviews also provide insight into competitors and general issues users are facing. However, there are also limitations that need to be addressed. This article will discuss the opportunities and challenges of using online reviews for UX purposes and provide recommendations for practitioners.
The Opportunities of Online User Reviews for UX
Easy access to real user feedback
A key benefit of online reviews is that they provide easy access to feedback from real users based on actual field use (Hwang, 2019). This can be used to triangulate findings and insights from research studies. Online reviews go beyond predefined questions, allowing for the discovery of critical elements like user profiles, context, and product features that might be missed in traditional research methods. For example, one study by Khalid et al. (2014) found that in some cases app reviews covered usability problems not identified through heuristic evaluations. As a result, online reviews can help us expand the scope of UX data collection and uncover new issues.
Large datasets
The vast scale of review data enables analysis not possible through small-scale UX testing. For example, sites like Yelp hit 209 million monthly users and 190 million reviews, while Amazon sees over a billion monthly visitors. This massive data can help us uncover weak signals such as pain points affecting a small subset of users - that traditional research would likely overlook, especially in companies with small research budget.
Sentiment and emotion analysis
Online reviews often contain emotional language and sentiment analysis of reviews can unveil user frustrations, needs, painpoints, and more. For example, there are several systems to allow sentiment analysis on online reviews such as app store reviews. Such analysis can pinpoint where experiences fall short of expectations.
Beyond overall sentiment, reviews can also provide an opportunity to analyse sentiment toward specific features and aspects of an experience. For example, reviews in G2 allow us to examine specific aspects of the user experience such as Pricing. This can spotlight weaknesses and strengths in an experience.
Competitor research
Online reviews live in the public domain, which means that they can also be a source for analysing competitors’ strengths and weaknesses. This can help us gain insights into market position, user expectations, and uncover user unmet opportunities. For example, in previous roles I have used reviews in sites like G2 to analyse reviews for my company as well as our top competitors uncovering gaps between user needs and current solutions, and potential opportunities.
Limitations and Biases of Online User Reviews
While online reviews can be used in UX research, there are also inherent limitations and biases to consider.
Selection bias
Not everyone writes product reviews. Unless externally motivated, only a small subset of users opt to write reviews. These means that there are biases in who chooses to contribute (Hwang, 2019). Certain motivations and demographics may be overrepresented, while other absent. Studies have found differences in ratings between reviewers and non-reviewers (Liu et al., 2021). This makes generalisation of the findings difficult.
Reporting bias
Space constraints mean users selectively report certain experiences over others in reviews. Users may focus more on negatives than positives, for example. A comparative analysis of app reviews versus user interviews revealed differing proportions of issues (Pagano & Maalej, 2013). Not all data is captured equally.
Recall bias
Reviews rely on user memory and may miss experiences users have forgotten. UX encompasses anticipation of use, but reviews only capture post-use memories. Time decays memory, introducing recall biases. Comparisons to real-time user observations are needed to control for this.
Misrepresentation and fake reviews
The authenticity of online reviews can be questionable, with instances of fake reviews designed to artificially boost or damage a product's reputation. For example, research has shown that review fraud is quite common (Luca & Zerva, 2016). Companies also often encourage specific users to leave reviews (e.g., loyal customers), which can skew data.
Analysis complexity
Extracting meaningful data from a large volume of unstructured text requires sophisticated analysis tools and techniques, which can be resource-intensive. New AI tools can help with this but many UX professionals require additional training or support to use them.
Ethical and privacy concerns
Using online reviews for research purposes raises ethical considerations, particularly around user privacy and data use consent. Responsible practices like anonymisation, audits for bias, and communication on use are ways to minimise this risk.
Extracting Data from Online Reviews: A Methodology
There is no one way to do this but Yang et al. (2019) have proposed a systematic approach to extract and analyse UX data from online reviews. It involves the following stages:
UX discovery: This stage involves extracting individual pieces of UX data from reviews. It includes identifying customer sentiments, product features mentioned in reviews, and the context or situations in which these features are discussed. Techniques like natural language processing (NLP) and sentiment analysis can be employed to dissect the unstructured text of reviews.
Data integration: In this phase, similar pieces of UX data are grouped together. This involves classifying the data into different facets like product features, user sentiments, and usage contexts. Advanced data analytics methods are used to find patterns and commonalities among the disparate data points.
Network formalisation: The final stage involves creating a network that maps the relationships among the different UX data categories. This network helps in understanding how different aspects of user experience are interrelated. For instance, it can show how a particular product feature influences user sentiment in a specific context.
In their 2019 paper they describe a case study of how this methodology is applied to analysing smart mobile phone reviews. This application demonstrates how insights into key product features, user sentiments, and usage contexts can be extracted from user reviews. For instance, analysis of reviews might reveal that a phone's battery life is a crucial feature affecting user satisfaction in a business context.
Recommendations for UXers
Despite the limitations, online reviews remain a valuable UX data source when used carefully. Here are some recommendations:
Triangulate reviews with other UX methods to control for biases. While we have already discussed some potential biases, other forms of bias may also significantly impact the data. It's important to thoroughly assess these biases before interpreting the data. For instance, Hedegaard & Simonsen (2014) compared usability issues identified from online reviews with those found in traditional evaluations like heuristic analysis. Similarly, Weichbroth & Baj-Rogowska (2019) correlated frequent keywords in online reviews with established usability attributes and UX dimensions. By juxtaposing findings from online reviews against established UX theories and previous studies, we can identify and measure the extent of these biases. Remember, analysing online reviews does not replace primary UX Research!
Follow-up and further test the findings. Online reviews can help us identify issues and areas to focus on. Use mixed methods by supplementing reviews with follow-up interviews or surveys to fill gaps and test findings.
Develop data correction methods to adjust for biases. One approach is to interpret online review data while consciously accounting for identified biases. This might involve weighting certain aspects of the data more heavily or reading between the lines to understand what users might not be explicitly stating.
Online user reviews can provide us with a low cost source of real user data. While inherent limitations exist, we can can take steps to evaluate potential biases and compensate for them in our analysis. With thoughtful and ethical use, online reviews can enrich UX research and practice.
Fantastic article. I am using reviews as a low budget solution and had doubts about biases and how to validate results. I have also used ai for this. This articles provides not only concrete research around the matter but also actionable tips on how to handle reviews. Thanks a bunch!
Mining online reviews for UX insights can be tricky due to biases and fake entries. You could try HiFiveStar to monitor genuine user feedback and spot trends across platforms. It helped me get clearer signals without drowning in noise.