Our QA Lead Nadia on how to test a game, analyze user behavior and what to do if a player behaves differently than you expected.
Nadia, QA Lead
Training
A playtest is not an actual test of a game. The users involved should not search for bugs that the testing department missed, but rather focus on actually playing the game as your target audience.
If possible, it’s best to playtest as early as possible when it’s not too late to change the overall dev-direction. This will give you an understanding of whether the balance is set correctly, what the approximate retention of the app may be, whether the mechanics are correctly chosen and whether the gameplay is not too complex.
Check list
1. Google Questionnaires. They help systematize many crucial criteria of the testing group, e.g. help you understand whether the playtest participant falls into the target audience.
2. Signed NDA (Non-Disclosure Agreements). A video filming agreement may also be necessary if you intend to film the playtest.
3. Correctly working devices on which users will test the application. It is important to ensure that the charge level on all devices is sufficient for at least an hour of gaming. Stocking up on extra batteries may also be a good idea.
4. A video camera, an action camera, or another mobile phone for video shooting. Capturing players’ emotions is really important! To capture everything that happens during the game, Flippy Friends AR playtests used standard iPhone screen video and a GoPro camera to capture the behavior of the players, their emotions and reactions.
5. Invitation to a playtest. We advise you to indicate that the application is being developed by a third-party company, and you are only an intermediary and conduct market research. In this case, the players will be more critical while playing and providing feedback.
6. Rapid feedback. Use a separate questionnaire to record the players’ feelings right after they finished the game session.
To capture everything that happens during the game, Flippy Friends AR playtests used standard iPhone screen video and a GoPro camera to capture the behavior of the players, their emotions and reactions. It is important to ensure that the charge level on all devices is sufficient for at least an hour of gaming session. Or stock up on extra batteries.
Whom to invite?
We were looking for people in web-communities with part-time jobs, in students- and themed gaming publics. There were enough people who wanted to take part in a playtest. However, some were turned down, as they responded to the ad just because of the phrase “we guarantee a decent reward.”
Where to play?
The playtest was carried out in a separate empty room so they would not disturb anyone in our dev-team. The players were given test devices with the game pre-installed and a legend: imagine that they themselves were interested in the game and installed it.
Quality playtest rules
1. Do not invite friends or acquaintances. They will more likely tell you nice things about the game in fear to offend you.
2. Don’t call the entire development team for a playtest. Only the lead game designer and/or the chief developer will be enough.
3. You can interfere with the gameplay only if you notice that the player has “broken” the app and isn’t able to play on. In all other cases, do not interfere or prompt the players.
4. Be sure to capture the gameplay on video. The record may indicate lots of usability problems afterwards.
5. The more playtests, the better. Don’t limit yourself to a couple of players, try to let at least twenty people who are part of your target audience.
6. Don’t stop the players. Try to catch the moment when users themselves want to finish — this way you will understand if the game is capable of holding someone’s attention at all.
7. Immediately after the end of the game session, give players a questionnaire with questions about gameplay, interface, sound and general playability, while the user experience is fresh.
8. Don’t focus on paying for the playtest, as there is a chance that people will come to you who will just want that “easy money”.
9. Do not invite people who have already participated in that playtest once. They will already have an impression of the game, and you will not receive fresh information.
What to look out for
1. Any deviation from the scenario planned when designing the game. For example, you thought that the player would destroy all enemies using stealth methods, and he decided to make a real massacre with a shotgun. Or an example from Flippy Friends AR: to throw, you need to hold, pull and release — then the object will fly along the correct trajectory. As it turned out, the players did not understand this and used short swipes.
2. Player reactions. Try to talk to anyone looking shy or strained so that (s)he does not feel awkward and constrained. This happens due to video being shot or a large number of strangers in the room. From the randomly thrown phrases of the players during the game session, one can also draw a conclusion about the emotions that the player experienced at that moment.
3. Emotions and reactions. Boredom, surprise, interest, frustration, irritation — all these emotions affect retention. If the application causes the player not the excitement of the competition, but anger and disappointment, then most likely he will not return to the game, or maybe delete it immediately after another failure.
After the game
At the end of the playtest, we gave the players candy and some merchandise based on the already released game. Prepare nice gifts in advance that will leave a good impression on your company. Useful merchandise such as pens, notebooks, stickers, t-shirts (depending on your imagination and budget) will not only be a good reward, but also an advertisement for your products.
Bottom line
During the Flippy Friends AR playtest, we received a lot of raw data that needed to be processed and, I must say, this is the most labor-intensive part of the playtest. For the team, we wrote a short report with a list of incorrect actions of players and their emotions with screenshots, gifs and videos. After the playtest, it became clear where users made mistakes, where they got stuck and where they managed to “break” the game logic. Watching videos and listening to recordings, writing reports and drawing up charts based on the results took a lot of time.
The project team experienced mixed emotions after the results — it was necessary to redo the tutorial and some game mechanics, because the users did not understand how to start the game session. It is not easy to accept the need for changes, but they benefited the application and the following playtests proved this.
Playtesting needs to be taken seriously. Yes, it can be painful to see how the result of many months of work becomes useless. But it’s better to find out sooner that your game needs redoing than after the release!