Quantitative research for new user researchers | How To Be A Games User Researcher š®š°
Quantitative Research | An interview with Blizzardās Adam Lobel | Essential reading for games user researchers | Jobs with Epic Games & EA
Letās start with my confession. I have been a user researcher working with games for over ten years. I have run hundreds of studies, and overseen thousands of hours of playtests (over 25,000 player hours at last count!). And yet, I know very little about stats.
There are two quantitative research things I know how to do. Today, I will explain both (read this issue in your browser).
Comparing two sets of numbers
The first thing is how to compare two sets of numbers. I was taught this by Cyril and Mirweis at PlayStation, and I am grateful to them both for teaching me the only stats I know - how to compare two sets of numbers. This is useful when comparing things such as āhow many times did the player failā or āhow long did it take people to complete this levelā
This method is appropriate for when the data is numerical rather than categorical (or ordinal). Hereās a short explanation of what that means.
When you have some numerical data, itās quite common to want to compare it. This allows you to learn āis there a difference between these two thingsā, and then inspire conversations such as ādo we want players to fail more times on this level than on the next one?ā.
To do this, you want to find the average, and then work out some confidence intervals to anticipate whether the difference between them is real or whether it was potentially caused by not measuring enough people.Ā
So, after counting how many times people died on level 3, you can take an average - which looks like this.Ā
We can see that on average, players died around 2.5 times on level 3.Ā
We can then do the same thing for the next level.
(This is probably a good moment to mention there is a template that does the maths for you later in this post...)
Looking at the average for Level 4 shows us that people died on average more often on Level 4 than they did on Level 3.Ā
But we donāt know if this is because Level 4 causes more deaths, or just random chance that it occurred in this study.
To identify that bit, we calculate confidence intervals. Which looks like thisā¦
And we can see that the confidence intervals (the uppey-downy bits) overlap. The top of Level 3 overlaps with the bottom of Level 4.
Level 5ās confidence intervals do not overlap with any of the other levels. If the confidence intervals donāt overlap, there is a real difference between them. It's true that more people died, and will die, on level 5 than level 4.
This hopefully means that Level 5 harder - although you should watch people play to understand actually why the difference in deaths occurred.
If the confidence intervals donāt overlap, we canāt tell if there is a difference. This is the case for Level 3 + 4. This either means that the number of times people die are the same, or that we havenāt seen enough players to draw an appropriate conclusion.
(There are probably errors in the terminology above, but as I said, I know little about stats - I just know how to compare two sets of numbers).
I use this all of the time - to count and compare deaths, completion time, etc. I made a template that you can duplicate to see the formulas required, and to have a go at doing it yourself.
Go deeper into quantitative research
Beyond this one technique, Iāve found two other tools very helpful.
Adjusted wald calculators like this allow you to state your completion rate (e.g. 3 out of 10 people encountered this issue), and from that anticipate how many people in the real world would encounter the same issue (between 10% and 60% apparently).
And the book āQuantifying the User Experienceā which has lots of nice decision maps like these, which tells me what tools I should (and shouldnāt) be using ā¦ and includes a crash course in stats to explain how to do them!Ā
Avoid common quantitative research errors
The second thing Iāve learned is a collection of things not to do. By recognising some stats errors, it helps me know when I should seek out someone better with stats than me to help out.Ā
Avoiding common errors include:
Donāt do the kind of maths I described above on ordinal data (such as likert scales). People often do, and get away with it, but itās somewhat inaccurate as youāre treating categories like they are numbers.
Think about the sampling bias you have created in your study, and donāt over-emphasise how representative your conclusions are
Donāt assume that because you are measuring what players say they think or do, you are actually measuring what they think or do.
Recognise that when you are limiting the options you allow people to select from, you are limiting the range of results you will get back, potentially distorting the truth.
Avoid dogmatic rules about sample sizes. Thereās lots of rules out there that have become dogmatic (āquant studies need 30 usersā, āqual studies need 5 usersā), and many people repeat them without understanding the reason behind them. Understand why those guidelines exist, think about what you are trying to learn, and make conscious decisions rather than following ārulesā.
The job is not just āqualitative researchā
I sometimes encounter the idea that user researchers are synonymous with qualitative research. I donāt think that is appropriate or correct. Even if you are more comfortable with qualitative research, you shouldnāt allow your skillset to determine the method you apply for answering research questions.Ā
Instead always lead with āwhat does the team want to knowā, and then āwhat is the most appropriate way of discovering thatā. If that method isnāt one you are comfortable with, use it as an opportunity to learn how to do a new thing, ask for help from the community, or bring in some help from someone who is comfortable with it. Our job is to āhelp the team make evidence-based decisionsā, regardless of the methods we are most comfortable with.
What quantitative research skills should I be ready for in the job interview?
If you can answer the following questions, I would say you would be a stand-out candidateā¦
What is p-value?
How would you compare the difficulty between two levels? What would you measure, and how should that be interpreted?
How would you measure if players are enjoying a game?
How would you handle being asked āI think this study should have a larger sample sizeā?
You will notice that these questions are often not about āhow do I do the statsā, but much more interested in āwhen is quantitative research appropriate, how should it be applied, how should I explain things to my colleagues, and what are the caveats for this kind of workā. Which I think is where the real challenge lies!Ā
Interview with Adam Lobel - User Experience Researcher at Blizzard Entertainment
On the subject of quantitative data, I was lucky to speak to Adam Lobel recently about his journey into games user research. He has been working as a user experience researcher at Blizzard for two years, and we talked about meeting new people in the workplace, turning a PhD into a games job and spotting opportunities in the data that others miss.
Read the full interview with Adam Lobel here.
Games user research jobs
Sprung Studios are looking for a Junior User Researcher to work with their clients which include EA, Activision, LucasArts, Rare, Ubisoft, Microsoft Studios, ArenaNet, and many others. Find more details and apply on the GUR Jobs board here.
Epic Games are hiring a contract user research analyst to work with their fantastic team on titles such as Fortnite. Apply on Epicās website here.
EA are looking for an Associate UX Researcher to work on FIFA. Apply on EAās website here.
More games user research reading
Last month we featured the first of Alistair Greoās articles about Getting Started in Games User Research. Part two is out now, and answers questions such as āwhat research methods should I knowā, āhow to get and demonstrate experienceā, and being prepared for interviews. Read Alistairās article on Getting Started in Games User Research.
If you are writing a cover letter this month, read this twitter thread on how to write a good cover letter.
Also, check out this article from Splash Damage's Jason Tzaidas on tips for a successful games industry interview.
Thanks, and hello...
Thanks for making it to the end, and particularly thank you to all of the new subscribers - weāve had over 500 people sign up for this newsletter over the last few months. Many of the new readers joined this month, so welcome to all the new people starting their games user research careers.
Iāve written a book about how to be a games user researcher, do take a look if you havenāt already. If you had, Iād really appreciate a review on Amazon - it does have a huge impact on the book, and I value it very much.
As always, do email me or tweet me with feedback, questions, etc and Iāll see everyone next month!Ā