Results 1 to 7 of 7
  1. #1
    Community Manager Berserkerkitten's Avatar
    Join Date
    Jun 2017
    Location
    Nottingham, UK
    Posts
    1,088

    Update #43: UIX and Focus Testing: How user feedback lets us make a better game

    Dear Pathfinders,

    The first alpha test is almost here!

    Our first testing stage for Pathfinder: Kingmaker begins this week! Everyone who selected alpha access add-ons in the pledge manager will be sent a Steam key for alpha testing, so keep your eyes peeled for an email from us. As a quick reminder: anyone who backed at least $95 for the beta tier can buy the alpha add-on through the pledge manager. Before you take the plunge, please be aware that our alpha build is just that - an early, unfinished version of the game, which contains bugs, typos, and placeholder assets. It will give you an impression of our work in progress, as well as a chance to help us find bugs and test certain game systems. If you’d rather see the game closer to launch, it's probably a good idea to wait until beta testing and release.

    UIX testing + focus testing + meeting for bug triage

    *Please note that some contents shown in the videos in this update are still subject to approval by Paizo

    As we mentioned before, while developing interfaces we use our UX lab to get the most reliable feedback from the players. Today we want to tell you more about how we do it and what we get as a result.

    Before we get the testing started, we need to decide what we want to learn from our tests - whether certain interfaces work as we planned, if players like to complete quests, or even how long they remain interested in playing within certain time frames.

    After our goals are set, we create a draft script with the help of our colleagues from UX lab, which consists of the following parts:

    1. Introduction – when the player gets acquainted with the game and receives instructions. Here, it’s extremely important for the player to realize that he must play exactly as he would normally play at home. In other words, he can help us only if he forgets he’s helping us. :)
    2. Testing description.
    3. Things we (Owlcats) should pay special attention to.
    4. Questions that arise during testing.
    5. Final interview.

    The next step is selecting the respondents. Usually, for our testing we invite 6 to 9 participants who belong to one of three groups: those who are familiar with the tabletop version of Pathfinder, those who used to play other roleplaying board systems and those who never played any roleplaying board games. At the same time, all the respondents are crpg players and hence belong to our target audience. The number of respondents, that belong to each of these groups, depends on the goal of our testing. In case of our interface tests, we’re equally interested in opinions of representatives of all the groups mentioned above. If we are testing game mechanics, we want feedback from players familiar with RPG systems. And if we’re going to test the learning curve, we want comments from players unfamiliar with RPG systems. It’s simple logic – those who have the most difficulty figuring out what’s going on in the game are the most help to us.

    After we have selected our respondents, testing begins. On average, each test takes three days, with three respondents per day. Each player needs approximately three hours to undergo the testing - introduction takes a half an hour, playing the game is two hours, and the last half hour is the final interview.

    The testing itself looks like this:

    • We meet the respondent volunteer and treat him or her to something yummy. :)
    • We introduce the respondent to our UX lab specialist (a developer is always present during the testing, taking the opportunity to watch the respondent’s game in “live” mode).
    • We give the respondent a quiet room to play in with no distractions – just you and the game. Oh, and a chair. :)
    • We tell the respondent about the game, but do not mention a word about the goal of our testing (This is to keep the results unbiased.)
    • We tune the equipment: eye tracking, cameras, microphones. Using these devices, we record everything that happens in the testing room and broadcast it live to the observation room: the player’s words, their facial expression, gaze direction, and so on.
    • Then the testing starts. During the game, the player can share their experience in any form they see fit.
    • After that comes final interview with the UX specialist who will ask questions about your gameplay experience.

    During the testing, all the player’s comments are written down by both the developer and the lab specialist. Together with the video records, these notes will later be used to create a final report.

    Check out our UIX testing video here

    After the testing is complete, UX specialists compile all the data they’ve collected and make a summary report. Its form depends on the initial goals of the testing. In most cases, they just take the list of these goals and add their extended remarks, screenshots and thoughts on what’s working properly and what’s problematic.

    Here is an example of such a report:



    The final stage comes when the team reviews the reports, discusses new-found problems/inconveniences, and decides how to solve them.

    UIX testing is an extremely useful tool, and it gives detailed feedback on how users perceive our game. But we realize that this method has a substantial drawback – the number of testers is too small, and this testing is extremely time-consuming. As a result, representation and statistical validity of the results we get can be questionable. That’s why we use UIX-testing only to detect some nuances and non-apparent bugs of usability in interfaces that have already passed several iterations of testing.

    To collect more representative feedback data on our game, we use another classic approach – focus testing.

    To perform focus testing, we invite the same categories of players we use for UIX-testing, but the numbers are bigger – 10-20 people in each category. It's important to note that in this case territoriality doesn’t matter much , so we can invite people from other countries to participate in our testing.

    To perform the testing, we assign one developer to each player. It’s important to us that every game developer – and especially every game designer – take part in an iteration of focus testing at least once. Imagining how people would play your game is one thing, but seeing how they do it in real time – is something completely different.

    The process of focus-testing includes these activities:

    • Beforehand, we prepare computers with the game installed. We try to use various laptops, as examples of most potentially “inconvenient” hardware (monitor too small, keyboard not comfortable enough) :)
    • We meet the respondent volunteer or reach them via Skype. If the testing is done via Skype, we ask the participant to adjust the camera so we can see their monitor and – ideally – their face (if there are two cameras available)
    • We lock inside the special room where nothing would distract us from the game We do our best to provide a quiet game environment with no distractions.
    • We tell the respondent about the game – briefly describing the events in the game up to the beginning of the test, explain how the controls work, and so on.
    • Testing begins. The point of focus-testing is to avoid giving hints to the player. Instead, we observe their actions and notice what wasn’t clear enough, or what went wrong/not as it should. To make this easier, the player is invited to comment on their in-game actions in detail as they are playing.
    • Final questions. Usually we ask the player to give us his general opinion on the game (and also, is the game interesting enough to buy it – the criteria of “parting with my one’s own money” is often the most effective appraisal of how compelling the game is ☺), describe the most annoying and the most pleasant elements.

    All the remarks of participants are added to a special table, which looks like this (example):



    During the last test we performed in July, we collected about 1.4K comments. After preliminary sorting and merging, this gave us about 600 suggestions and bugs for discussion. Once we have those, then it’s time for the most interesting activity - issue resolution brainstorming. Some of these discussions were extremely heated, and more yet to come!

    Brainstorming video

    The last focus-test triage resulted in about 340 new tasks.

    Even though we discuss the game’s problems in these triage meetings, these discussions are very motivating for the team, because each decision improves our game, not in some abstract way, but for real people playing it.

    In conclusion, we’d like to remind once again that we can’t wait to start a new, most representative test – our first Alpha test stage – and it starts this week. We hope we’ll gather massive feedback and lots of comments, for it’s important to us to understand how you like playing our game and what we should do to improve your experience!

    Hail to the kings!

    Owlcats.

    To the Kickstarter version
    Ye cannae touch this! - Clan McHammer

  2. #2
    Senior Member Grifta's Avatar
    Join Date
    Jul 2017
    Location
    Victoria, BC, Canada
    Posts
    324
    The video snippets are a great idea. We hadn't seen the team doing their stuff in a while.

  3. #3
    Senior Member Stratagemini's Avatar
    Join Date
    Jul 2017
    Posts
    1,013
    Will there be an Alpha test Forum? since we can't discuss what's in the Alpha on the regular forum?

  4. #4
    Community Manager Berserkerkitten's Avatar
    Join Date
    Jun 2017
    Location
    Nottingham, UK
    Posts
    1,088
    Quote Originally Posted by Stratagemini View Post
    Will there be an Alpha test Forum? since we can't discuss what's in the Alpha on the regular forum?
    The alpha forum section is already up. I dunno whether or not our admins have already made it visible to any non-Owlcats right now, but it's totally there and should work for alpha users as soon as it's all live. Hope I didn't jinx it now. :-/
    Ye cannae touch this! - Clan McHammer

  5. #5
    Member
    Join Date
    Jul 2017
    Posts
    28
    Great update!

  6. #6
    Senior Member HenriHakl's Avatar
    Join Date
    Jul 2017
    Location
    Stellenbosch, South Africa
    Posts
    530
    The update mentions the issue with some testers responding negatively to the micro-alignment shifts; but does not say how the team is addressing this.

    Personally (not having tried it myself) I feel I would like to have that level of feedback on my decisions. I think this would benefit from an alternate way of presenting the micro-alignment shifts rather than not having them at all. A suggestion in that regard: instead of showing hard numbers (embedded into the dialogue output) it would be better to use temporary display of appropriate alignment icons rising on the character portrait. Similar to how damage numers are often shown in games in combat over characters - but showing 3 <L> and 1 <G> icon on the character portrait for a (3 lawful + 1 good) shift.

    Players can still after dialogue (or during?) track the detailed alignment path in their character sheet; but I feel it is important to know that any choice you make has consequences (such as alignment adjustment) and having a little animation to indicate such micro-adjustments seems appropriate to me.

  7. #7
    Senior Member Pathfinder's Avatar
    Join Date
    Jul 2017
    Location
    Stolen Lands
    Posts
    653
    Quote Originally Posted by HenriHakl View Post
    Personally (not having tried it myself) I feel I would like to have that level of feedback on my decisions. I think this would benefit from an alternate way of presenting the micro-alignment shifts rather than not having them at all. A suggestion in that regard: instead of showing hard numbers (embedded into the dialogue output) it would be better to use temporary display of appropriate alignment icons rising on the character portrait. Similar to how damage numers are often shown in games in combat over characters - but showing 3 <L> and 1 <G> icon on the character portrait for a (3 lawful + 1 good) shift.
    That make me think of bit different kind of thing what good DM will deal but in computer games AI can never act like human so:
    IF player use metadata (he as a human know something what player character can not know, let's say he have played game before or read material).
    In Baldur's Gate we do remember how certain dark elf were able to lose all his equipments simply by talking for him and planting HUGE amount of skeletons around him. Then starting to attack and using only bows and running around like crazy.

    So, naturally player can choose to try and kill everything what is moving (and not moving) but I think it would be proper to make shift atleast toward chaotic if not evil even when player (based on meta, not something what game have told that far) choose to start killing innocent looking pesants/farmers and so on just because player (not character) know that later these NPC's will revealed to be party of bandits / some evil cult or so. OR even if player knowing that after party go to the room, talk with "friendly people" get attacked by (let's say by using "reload game after dying" method) them and choose to send plenty of area attack spells there trying to kill them before encounter happen.

    After all party and characher WILL be looked totally different if they let's say encounter bandit village, slay them and take a loot then if they encounter village, slay them all take loot and tell "they might have been bandits, we just acted first"
    "Road to the man's heart go through the chest"

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •