How to write a playtest survey [How To Be A Games User Researcher] 🎮📝
Survey Design | Games UX Challenge | Kabam Games' Krista Parham | Accessibility | Essential GUR Reading | Jobs with WB, Riot and Rockstar
Survey design is one of those topics that looks simple, but it’s incredibly easy to make big errors that invalidate your results. They are particularly risky - unlike some other research methods, it’s hard to spot when your data is junk, and your conclusions are unreliable.
Despite this, they are a very common research method, often used at the end of qualitative playtest studies, and an important skill for games user researchers to master.
In this issue we’ll look at the steps to designing a robust and reliable survey or questionnaire for your games user research study. Not a good time? Open this issue in your browser
Designing surveys for games user research studies.
Decide what you want to learn
Before you start writing questions, you need to decide what you want to learn. These are your research objectives, and are different from the actual questions you will ask players. A research objective could be “is the difficulty of our game correct”, but you probably don’t want to ask that question directly to players in those words.
Research objectives are best gathered by speaking to design colleagues, and understanding where the biggest risks for the game are & what do we need to learn to make good decisions. Our first #gamesUXchallenge this month gives you the opportunity to practice generating them.
Having identified research objectives, consider whether a survey is the best method for answering them. Surveys are good for measuring things, for example opinions (“Does this level look good or bad”) or behaviour (“How many times did it take to complete this level”). They are less suitable for deep qualitative feedback which would require observation and probing to answer (“Why did players take 20 attempts to complete this challenge”)
Remember - the research method should always be informed by the research objectives. Starting with a method (“today I’m going to run a survey”), will lead to low impact findings
Decide your sample
After deciding what you want to learn, think about who needs to answer your questions - who are your real players.
A common mistake I see online is sampling by convenience - e.g. getting whoever you can to fill out the survey. This is a waste of time. If you’re not confident that they are really your players, you’re gathering a lot of opinions and thoughts from people who would never buy your game. Pointless.
Some places to consider looking include forums for competitor games, using twitter hashtags and using facebook ads to recruit players. However any recruitment methods will introduce a bias into your participants that you should anticipate, and consider. Offering incentives - money for taking part - can help make your respondents more ‘typical’.
Putting careful thought into “who are the right people to ask this question to”, “how can I find them” and “how can I convince them to fill out my survey” are important for making sure your data is representative of your real players.
Write your questions
Each research objective should have one or more questions that try to find an answer. Some rough examples are here:
Scales are a very common way of capturing answers to surveys and can include:
An overall numerical score - asking players to rate something out of 10.
A two-directional scale - with two conflicting ideas on each side of the scale. Make sure that they are opposites, or this one gets confusing. For example (“Very bad” to “very good”)
An agreement scale - asking players if they agree with a statement. (“Strongly disagree” -> “Strongly agree”)
When writing the text for your questions, keep the following points in mind:
Will my player understand the question? Do they know what I'm asking about. Do I use language they are familiar with.
Will they be able to remember the thing you are asking about? Are we asking about something that is recent enough for them to have a real opinion?
Will they be able to decide on a correct answer? Is this something they actually have an opinion on? And are they comfortable telling you the answer, or is it socially embarrassing to say?
Will they be able to enter the answer that matches their true feeling? Make sure your scale will allow them to give their true opinion. Asking them to rate from "neutral" to "good" is no help if they think it's "bad"
A really powerful combination is combining rating questions with a open-ended question asking 'why' This does not replace interviews though, because a survey doesn't allow the deep interrogation that is often required to really understand what players are thinking or doing.
Some tips for questions:
Avoid leading players by providing opposite statements in the question text e.g. instead of asking how difficult was the game, ask how easy or difficult was the game.
Bold key words in the question, so it's easy to pick up the meaning from skimming it: How were the graphics in this level?
Keep question text short and direct, so they are easy to understand. Rate this level is clearer than Please provide a rating based on what you thought about this level
Be clear - use the same words that players do (interviews can help reveal this), and avoid jargon. What did you think about this character's DPS won't make sense to many players.
As with interviews, a good structure is to go from broad questions to direct questions. Ask 'overall, how was the game' before asking specifics such as 'how easy or difficult was using the jetpack'
Avoid yes/no questions as they are too closed and force players to give strong binary answers. Rating scales, or asking players to explain, allow more nuance to emerge.
Keep the survey short.
Never ask about future behaviour. Questions like will you buy this game have always been to correlate poorly with real behaviour, giving you unreliable conclusions. It's much safer to ask about things they have previously done "What was the last game like this you bought" to get answers that are accurate
I’ve also gathered some excellent tips for survey design from the community. Find them all at the end of this section.
Pilot your survey
Ask a colleague or friend to answer your questions before you send it to players, and ask them what they think each question means. Doing this while sat with them, allows you to ask them questions about each question as they fill it out. This quickly reveals where they haven’t understood your question or the words you use.
Make your playtest survey for real
Use a survey tool to create your survey for real. Google Forms is free, and has some basic survey logic so can be appropriate for simpler surveys.
More advanced tools, like SurveyMonkey or Qualtrics can be a good next step once you’ve reached the limit of what Google Forms can do.
Distribute your survey
Send your survey out to participants (or ask them to fill it in after coming in for a playtest). The best method of doing this is probably clear after doing the work to ‘decide your sample’ earlier.
Analyse the findings
Analysis is a whole topic in itself, but before we get started, we need to clean up our data.
I personally would always recommend exporting my data into a spreadsheet, rather than using the survey tool’s built in analysis tools, to help explore it. We looked in a previous issue at some tips for how to handle quantitative data.
The data we have collected will usually have some issues with it. Some people will not have filled out the survey properly, and would have clicked through as fast as they can, or given some impossible answers, and we want to remove them from the data set. Read through the responses, review the start and end times to see how long people took on the survey, and look out for tell-tale signs such as ‘rating every question the same’ to indicate which participant data needs to be removed.
Analysis and finding meaning in the data is another skill to practice - but we’ll save that for a future newsletter deep-diving into analysis.
Learn more about writing playtest surveys
When writing this post, I asked the community for their tips. I received so many that I had to spin them off into a separate post - read the expert tips for writing a playtest survey.
Elizabeth Zelle (who kindly submitted some of our expert tips), gave an excellent talk about survey design for games at a previous GRUX conference. Watch Elizabeth’s talk here.
I also am a huge fan of the work of Erika Hall, and it would be remiss of me not to link her post about the dangers of surveys. Essential reading before you write your own.
The Games UX Challenge
This year, we want to go beyond ‘reading about user research’, and give everyone the opportunity to practice user research core skills. This series of real-world UX tasks will allow you to create portfolio-worthy examples of doing real games user research.
To do that, we’ll be running regular Games UX Challenges throughout the year. This first one is focused on setting research objectives - an essential part of a researcher’s skillset, and crucial to ensure that a study gets set off on the right path.
I’d love to see how you get on - as you’ll see in the challenge, everyone who takes part will get sent a free games user research sticker.
Read more about the first Games UX challenge.
Games User Research Career Interview - with Krista Parham - User Researcher at Kabam Games interview
This month I’ve been talking to Krista Parham, User Researcher at Kabam Games about her experience working in games, and top tips for people looking to join the industry.
She shared what it’s like working on mobile games, how games mirror the film industry and why it’s important to be honest about the games you play.
Read the full interview with Krista here.
Microsoft’s gaming accessibility course
Microsoft have announced a new free gaming accessibility course, teaching concepts such as game design best practice, community collaboration, and the use of assistive technology.
Accessibility is a core part of the skill set of many games user researchers, and is an area that the industry is increasingly aware of - I’d highly recommend checking out their free course to grow your own expertise.
Essential Games User Research Reading
I often get asked by students or career transitioners what reading they should be doing to get up to speed. To help with this, I’ve put together this list of two sets of games user research books. The first is core research skills for games UX designers and researchers to learn what is expected from them. The second set is of broader games UX books to round out your games experience – giving you the context needed to make an impact.
Read the essential reading list for Games UX and User Researchers.
Entry level Games User Research Jobs
Warner Brothers are looking for a temporary user research assistant to help run studies wiith Monolith Productions.
Riot Games are looking for a entry level user research moderator . This is a link to Jimmy Zhou’s discord post - you will need to join the GRUX Discord to view it.
The wonderful new research team at Rockstar games also asked me to include this advert for a UX Researcher. It’s perhaps not an entry level role, but they did ask very nicely! View the UX Researcher role at Rockstar Games.
Happy New Year
2022 is going to be a busy year for me, and so I’m really looking forward to sharing our games user research journey together. Here are some of the things on my mind currently…
Next month is the one year anniversary of the How To Be A Games User Researcher book, and I have been preparing a special gift ready to celebrate - look out for it in the next issue.
It’s also nearly a year of these email lessons. If they, or the book, have helped you get your first games job in the last year, do let me know – I’d love to hear your story!
I’m also sponsoring a few of the games user research and playtesting events this year - I’ll share more as they come around, and their excellent line-ups get announced.
Also early this year I’ll be releasing the playtest kit - aimed at helping developers run better playtests. It’s still at a pre-launch discounted price, so if you’re interested it’s a great time to pick it up now.
Thanks everyone. Do keep in touch on twitter, and I’ll see everyone for more games user research knowledge next month!