PilotStudy-Group:NGRS

From CS160 User Interfaces Sp09

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participants 3.1.1 Gustavo 3.1.2 Malin 3.1.3 Cecilia 3.2 Apparatus 3.3 Tasks 3.3.1 Easy Task: Logging In/Signing Up 3.3.2 Medium Task: Controlling the Red Blood Cell Character 3.3.3 Hard Task: Changing the Control Settings 3.4 Procedure 4 Test Measures 5 Results 5.1 Task 1: Easy 5.1.1 Gustavo 5.1.2 Malin 5.1.3 Cecilia 5.2 Task 2: Medium 5.2.1 Gustavo 5.2.2 Malin 5.2.3 Cecilia 5.3 Task 3: Hard 5.3.1 Gustavo 5.3.2 Malin 5.3.3 Cecilia 5.4 Qualitative Results Summary 5.4.1 Task One 5.4.2 Task Two 5.4.3 Task Three 5.5 General Summary 6 Discussion 7 Appendices 7.1 Demonstration Script 7.2 Instructions to Users 7.3 Raw Data 7.3.1 Gustavo 7.3.1.1 Task One 7.3.1.2 Task Two 7.3.1.3 Task Three 7.3.1.4 Feedback 7.3.2 Malin 7.3.2.1 Task One 7.3.2.2 Task Two 7.3.2.3 Task Three 7.3.2.4 Feedback 7.3.3 Cecilia 7.3.3.1 Task One 7.3.3.2 Task Two 7.3.3.3 Task Three 7.3.3.4 Feedback

Introduction

Our project is designing an interactive game, called "The Cell's Search", that would teach students (specifically medical school students) about the vascular system. The premise of the game is as follows: the user takes on the role of a deoxygenated red blood cell that starts off in some venous vessel in the body. The user, under time constraints, attempts to navigate through the maze of blood vessels in the body in order to get back "home" to the heart so the cell can be reoxygenated.

Our team has produced an interactive prototype of our game interface that addresses three tasks (of easy, medium, and hard difficulty) that we found to be most important when playing the actual game. This interactive system will require a user to (1) register and log in (easy), (2) navigate the red blood cell character through the maze (medium), and (3) change game settings/controls while in-game (hard).

Because we are in the initial phases of designing our interface, we will be evaluating our interactive prototype of this game. The generated design has only been used by the design team - we have not tested this system on real users. User study is crucial to developing a final product; the design team is biased in terms of interaction with the system because they are familiar with it. Real users might not have the same level of ease in navigating the interface.

Through careful observation of users' experiences with our prototype and through the users' commentary and feedback during and after their experience, we hope to collect useful data that will aid us in improving our design for the final product. We hope to acquire feedback on both positive and negative aspects of our system, especially constructive criticisms that offer us features we can improve and apply in future versions of the design.

Implementation and Improvements

• Instructional alerts appear only when user first logs in after registration
• User no longer needs to re-register/login after losing game
  • Game restarts from beginning once user loses
• Different components are now integrated seamlessly into the interface to create a more consistent look

Method

Participants

Our game is designed to assist users in familiarizing themselves with the human vascular system. As a result, we targeted users who were either interested in medical school (pre-med) or interested in learning more about the human body. For this pilot study, we chose users who are students because students would be most likely to use our game as a study tool. We also wanted to have a mixed selection of individuals in terms of familiarity with and daily consumption of video games because familiarity with games, specifically online flash-based games, would create a confound in our experiment: these are expert users who would be able overcome design flaws because they are familiar with game interfaces. Background information on our users are as follows (names have been changed to respect the privacy of the individuals in our study):

Gustavo

Gustavo is a 21 year old Integrated Biology major who plans on going on to medical school. Although he has never supplemented learning/studying with games, he plays a good deal of flash-based games online whenever he has free time. He is an A-B student. Gustavo likes studying at the library whenever he has exams. His mode of study is usually to prepare flash cards and review his notes that he takes in class. He says if there were a more fun way of studying biology for his exams, that would be excellent.

Malin

Malin is a 22 year old Mechanical Engineering major. Although he does not plan on going into medicine at the moment, he has always been fascinated with the human body, and prior to deciding on Mechanical Engineering, he was debating whether to take the pre-med route. Whenever he is not busy with his engineering classes, Malin says he enjoys reading books that broaden his biology knowledge. He is also an avid video gamer. When pitched the idea behind The Cell's Search, Malin was immediately enthusiastic. He, like Gustavo, would enjoy a more fun way of studying biology. Especially since he has been learning biological concepts for fun, Malin says he would greatly appreciate a game that was not only interactive but also educational.

Cecilia

Cecilia is a 22 year old Microbial Biology major. She is pre-med and has completed all her requirements for medical school, however she does not plan on taking the MCAT until a year later, when she plans on applying to schools. Cecilia says had there been some more enjoyable method to studying the human vascular system, she would have immediately taken it into consideration. Since she has yet to take the MCAT, she says a game such as The Cell's Search may be useful to her in preparation for the test. Cecilia isn't much of a computer buff, but she does use the Internet to aid her in her studies, referencing sites such as Wikipedia. She usually uses her textbook and notes to study for exams.

Apparatus

For our user study, we used Denise's computer, a Toshiba Satellite laptop, that had a browser open (Mozilla Firefox) with The Cell's Search login page up. Connected to the laptop was a handheld mouse (so users didn't have to use the laptop trackpad if desired). The laptop and mouse were placed on a tabletop surface at which users were seated, one at a time. Users were not allowed to be in the room while the study was performed on another user.

Tasks

We chose three tasks to perform in our user study. These tasks are as follows:

Easy Task: Logging In/Signing Up

"Logging In/Signing Up" is an easy task, must is necessary in order to even begin playing the game. This simple task only requires the user to type in his username and password and click the "Log In" button. Once the user clicks the "Log In" button, he is brought to the "Welcome" screen, where he can continue on to play the game. If the user does not yet have his own account in the game, he can simply click the "Register" button and type in his desired username, password, and email address. Upon clicking the "Register" button from within the registration form screen, his account is created and he is brought to the "Welcome" screen, which confirms that his account has been created and an email confirmation has been sent. From here, he can proceed to the game. (Note: The email confirmation is done in a wizard-of-oz manner, where we (the team) are the ones who actually send the email. This is our current method of choice until we implement the actual emailing mechanism.)

• Observed: Ease with registration/log-in process, error messages, clarity of information requested when registering.

Medium Task: Controlling the Red Blood Cell Character

"Controlling the Red Blood Cell Character" is a medium task. Depending on the user's control settings, the user has to help the red blood cell character maneuver through the vascular system while avoiding dangers, such as plaque. In addition, the user must decide which paths are the correct ones to take in order to reach the final goal destination -- the heart. For example, in our prototype, the user has to determine whether to choose the upward diagonal path or the lower diagonal path while avoiding the plaque-covered walls.

• Observed: Ease with controlling the red blood cell character, ease with figuring out controls, comprehension of game maze and goal.

Hard Task: Changing the Control Settings

"Changing the Control Settings" is a hard task. The user has to halt his game play in order to see the pop-up menu, from which he chooses to resume the game, change the control settings, or see game scores. From there, he can choose "Game Settings" in order to customize how he wants to maneuver throughout the game. The user can choose keys on the keyboard to move the blood cell up, down, left, or right. He also has the option to choose from one of the common schemes, which utilizes the arrow keys or the W-A-S-D letter keys. In addition, the user can choose whether he wants the chat, hints, and vein names to remain off/on during the duration of his game play. These are aspects of the game that affect the user's gaming experience and must be thought of carefully during customization.

• Observed: Ease with accessing game settings menu, ease with changing game settings and control settings, clarity of settings on menu.

Procedure

Since our tasks in this user study are the same as our tasks in the lo-fi prototype user study and the prototype is pretty similar other than the fact that this one is actually interactive and demonstrated via computer, we decided to approach the study in a similar fashion as we did the lo-fi prototype user study.

• Greeter/Observer - Siddharth Shah
• Facilitator - Salman Rahman
• Computer/Observer - Denise Ngai
• Observer - Dwij Garg

1. Greeter was responsible for receiving each participant at the apartment, introducing them to group members, making small talk while things were set up, and just ensuring comfort of the subject overall.

2. Facilitator was responsible for conducting the demonstration, explaining tasks to user, probing the user for feedback, and leading the user study overall.

3. Computer was responsible for responding to user input by manipulating the prototype (for features that aren't implemented yet), clarifying wizard-of-oz techniques that were used, setting up the apparatus, and ensuring that everything ran smoothly (technologically speaking).

4. Observers were responsible for recording notes during the study process

5. Greeter asks user:

          * Major
          * Year
          * Career plans 

6. Greeter then carries on light conversation about unrelated topics (i.e. how they are doing, how school is going, background on group members, etc.) while Observer #1 sets up in a convenient location with computer to take notes. Also, the Computer sets up the prototype.

7. When everything is set up and ready, Greeter asks user to sign consent form.

8. Greeter now takes on the role of Observer #2.

9. Facilitator now takes the lead and gives introduction to project

           * Describes our overall project goal and mission statement
           * Details reasoning behind developing interactive prototype
           * Informs why we are doing the user study / why it's necessary
           * Impresses importance of user's feedback to our design goals
           * Eases user's fears about "messing up" or giving negative feedback 

10. Facilitator then goes through the demonstration of how the protoype works and how the user interacts with the system

           * Demonstrates the demo task of turning on chat feature (as an example task)
           * Asks if user has any questions about how the prototype is to be used
           * Lets user know he/she will be probed for thoughts during the tasks 

11. Facilitator gives user three tasks (one at a time): signing in to the game, controlling the red blood cell character, and changing the game control options (specifically, user is asked to change keys used to control the red blood cell character)

           * Probes user for input about what he or she is thinking, feels about usability of the interface, etc. when the user has stopped talking
           * Observers take notes on user feedback / user actions 

12. After study conducted, facilitator conducts wrap-up

           * Asks user for likes/dislikes of interface
           * Asks user for overall impression of the design
           * Asks user which of the tasks was most difficult and why
           * Asks user if he/she has any suggestions for improvements
           * Asks user which things he/she would most like to keep

Test Measures

Observers took careful notes on the actions each user performed on the prototype, keeping in mind the following:

• Time it took each user to execute each task
• "Incorrect" actions performed by user
• Ease with registration/log-in process
  • Did user activate any error messages?
  • Was the information requested on the registration/log-in page manifested in a clear and easily understandable manner?
  • Anything confusing about the registration/log-in screen?
• Ease with controlling the red blood cell character
  • Was it easy to begin moving the cell?
  • Was it easy to figure out how to control the cell and move him in different directions?
  • Did the user understand the goal of the game?
  • Did the user understand the setting of the game and how to navigate through the maze?
• Ease with accessing game settings menu
  • Was the user able to get to the game settings menu without any problems?
  • Did the user understand the labels and instructions in the game settings menu?
  • Was the user able to change game settings without any problems?

Notes were taken to address the above points. In addition, each time a user voiced any commentary (positive or negative) regarding the game interface, observers took note of it. Every time a user experienced confusion or ease (executed something without hesitance), it was taken note of. Observers were also instructed to pay close attention to users' facial expressions in order to assess how a user was feeling even if he/she didn't verbally express anything.

Simply put, the following summarizes our approach to the test measures:

Bottom-line Data

  1. Time taken to complete task
  2. Number of "incorrect" actions 

Process Data

  1. General impression (as assessed by facilitator's questions)
  2. Comments made
  3. Facial expressions

Results

Independent Variables: Menu type, Game type, Apparatus type (device type), Controls

Dependent Variables: Time, Errors, Outcome

Control Variables: Game, Game Interface, Testing Environment, Apparatus (Devices used)

Random Variables: Attributes of subjects (gender, age, intelligence, eyesight, etc.)

Task 1: Easy

Result Summary: Across the board, this seemed to be not only the easiest task for our subjects, but it also was completely error free. No one messed this task up - not even slightly. Overall, people commented about how easy the task was and that it was very self-explanatory and the interface was simple to navigate and understand. One user wanted to see, out of curiosity, what would happen if he input an invalid email address; this proved helpful because he had some useful comments regarding the outcome.

Gustavo

• Time: 25.0 seconds
• Incorrect Inputs: 1 (input an invalid email address on purpose)
• Comment: ""Not hard." - referring to the task
• Comment: Prefers page to say "New user? Register" to clarify what new users should do; did not like placement of "Register" button.
• Comment: Error messages should pop-up and not require user to hover over erroneous field in order to see error message
• Comment: Thought it was good that we outlined erroneous boxes, but we should change error outline box color -- currently red so blends in with background.

Malin

• Time: 26.1 seconds
• Incorrect Inputs: 0

• Comment: Thought this was an easy task.
• Observation: User had no trouble with it whatsoever.

Cecilia

• Time: 45 seconds
• Incorrect Inputs: 0

• Comment: "There's no problem with how you guys did this part, I'm just a slow typer." -- referring to length of time taken to execute task.

Task 2: Medium

Result Summary: Users seemed to not pay close attention to the pop-up instructions regarding controlling the character. Also, users were unable to correctly identify the plaque lining the vessel walls as dangerous and ran into it, killing their character. Users had difficulty understanding where they were supposed to go in terms of navigating the character. Whenever a user attempted to use the mouse, it resulted in a negative outcome -- difficulty controlling the character or immediately running into the plaque. Mouse control seemed too sensitive. As a result of the aforementioned, this seemed to be by far the most difficult task for users.

Gustavo

• Time: 24.8 seconds / 31.7 seconds
• Incorrect Inputs: 3 / 0

• Observation: User ran into plaque in bottom vein
• Comment: "What are you supposed to do?"
• Comment: "What was supposed to happen?",
• Comment: Instructions only tell you how to move but not where to go - no objective/goal described.

Malin

• Time: 47.4 seconds / 19.4 seconds / 19.9 seconds
• Incorrect Inputs: 3 / 1 / 0

• Observation: Technical difficulties with plaque/boundaries.
• Observation: User unsure of controls at first; tried using the mouse but got off track.
• Observation: In second trial, user began with the mouse then switched to arrow key controls; proceeded smoothly.
• Observation: Hit plaque after passing diagonal portion of inferior vena cava.
• Observation: In third trial, used only arrow keys for controls and reached the heart.
• Comment: Once you win game, you can't do anything; pop-up alert just keeps popping up when you attempt to move.
• Comment: Didn't know point of game until played a few times and reached the heart; actual game interface does not state anywhere regarding point of game.
• Comment: No instructions state to avoid plaque.
• Comment: No directions as to what to do in game, only how to move/control character.
• Comment: Doesn't seem reasonable to use mouse or offer that option because as screen goes further, you lose desk space to move mouse.
• Comment: Should have an option to restart game in the middle of playing the game.
• Comment: "This should be the 'hard' task. You should switch this with the 'hard' task."

Cecilia

• Time: 10.3 seconds / 36.2 seconds / 55.2 seconds
• Incorrect Inputs: 1 / 1 / 0

• Observation: Started with the keyboard controls, but hit the plaque pretty quickly.
• Observation: In second trial, ran into plaque in the diagonal part.
• Observation: In third trial, reached the heart but went really slowly through maze so as to avoid plaque.
• Comment: The heart should be more prominent since it's the end goal of the game.
• Comment: Celly is a cutie.

Task 3: Hard

Result Summary: Users had little or pretty much no trouble with this task. Users generally wanted clearer instructions to accessing the game settings menu and clearer explanations as to what the game settings menu consists of. There were annoyances post-saving settings with regards to the mouse control during the game. Curious users were amused by the additional options given in the game settings menu, such as 'Chat' and 'Hints', and seemed enthusiastic to know that these would be in a future implementation.

Gustavo

• Time: 35.7 seconds
• Incorrect Inputs: 0

• Comment: Beginning/instructional pop-up should clarify what menu consists of or word instructions more clearly with regards to what menu will pop-up when you hit ESC or click the mouse during the game.

Malin

• Time: 25.2 seconds
• Incorrect Inputs: 0

• Comment: Mouse control is faulty, not smooth. Also, after editing game settings and clicking "Save and Close," the game detects where mouse is (outside plaque) and you immediately die due to position of mouse upon clicking "Save and Close" button.

Cecilia

• Time: 38.7 seconds / 40.4 seconds
• Incorrect Inputs: 1 / 0

• Observation: User showed confusion trying to access menu.
• Comment: After trying for a while, she confessed that she could not remember how to bring up the controls menu.
• Comment: The controls menu should be more plainly accessible.

Qualitative Results Summary

Task One

• The average length of time it took to complete Task 1 was 32.03 seconds.
  • Standard Deviation: 9.18
• The average number of incorrect actions per user was 1/3.

Task Two

• The average length of time it took to hit an end point (death) in Task 2 (during initial trial) was 27.5 seconds.
  • Standard Deviation: 15.27
• Upon familiarizing themselves with the controls, the average length of time it took a user to reach the heart was 35.6 seconds.
  • Standard Deviation: 14.67
• The average number of incorrect actions per user was 3.

Task Three

• The average length of time it took to complete Task 3 was 33.77 seconds.
  • Standard Deviation: 6.35
• The average number of incorrect actions per user was 1/3.

General Summary

In general, the users did not seem to enjoy the current interface design in terms of aesthetic. Two stated that the prototype was too red. All users had the most trouble with Task 2: Controlling the Red Blood Cell Character. Users claimed that the easiest task was Task 1: Logging In/Signing Up. All three users were not aware of the plaque or the point of it. As with the lo-fi prototype, users want instructions to be more clear. It seems the best way to avoid confusion is to have blunt instructions regarding each feature. User suggested a "How To" feature. Users also want the locator map to be functional.

Discussion

As a result of this pilot usability study, we've come up with a few implementation changes that we will be making in the future release of our game. Several of these changes are directly influenced by user feedback while some changes were considered after observing users interact with our prototype.

First, we noticed that some people had trouble with logging in and signing up. We realized that this is a time-intensive, typing-intensive task that is not actually a necessary part of the game. So, we are going to give people the option to play as a Guest user (no login or signup is required). This will make it easier for people to get started playing immediately rather than first having to struggle through the login/registration process. Also, people looking to just try our game would likely be deterred by the need to register and give out their email address all for a game they might not even like or use again. Besides adding a button to "Play as Guest" to the first screen of our game, we are also going to have a small link that, when clicked, displays the advantages of registering. The main benefits include the ability to have your settings and preferences saved, and you can have your name and scores show up on the high scores screen if you qualify.

Secondly, we don't really have an introduction to our game anywhere. We are planning to include an "About/How To" section that basically just lays out the purpose of the game, why it was created, our target audience, and brief instructions on how to play. This will help potential players to decide whether the game is really for them, and it might even serve to excite or entice some of them to play. Furthermore, this will hopefully clarify users' confusion regarding how to actually play the game and what the goal is.

In addition to the "Play as Guest" and "About/How To" features, we will change how the "Register" button looks. Currently, it is not easily noticeable for new users and blends in too much with the "Log In" button. Therefore, we will take Gustavo's suggestion about having a "New user? Register!" link/button somewhere more noticeable on the front page and in a more noticeable font/color.

Another important realization we got out of this study is that we desperately need to redo our color scheme. Some people hated it, while others loved it. More often than not, however, people did not like the vast amount of red. We need to make our scheme less divisive and polarizing. We want everybody to at least not mind it; they don't have to love it, but they definitely shouldn't dislike it, and it definitely shouldn't get in the way of the user playing the game. The main complaint that we got from the haters was that there was too much red. We were using the red to symbolize the circulatory system, but if too much red is a problem, we will change some of it to whites and pinks and other colors that could be found in the body. We need to retain contrast so that users can still easily differentiate between the foreground and the background (lighter red on darker red is maybe not the best combination), but we must also maintain consistency over our whole interface. In our current design, the title of our game is somewhat hard to read due to lack of contrast. The vein names were similarly hard to differentiate, which is a major problem because they are supposed to help the user navigate the circulatory system. Another problem with the indiscriminate use of red color is that validation errors cause red outlines around the offending fields, but these red outlines 1) are not going to be as visible if the background of the canvas is also red, and 2) might not strike the users as being indicative of an error because after all everything in the game is red. With regards to the validation errors, we need to either change their outline color and keep the background of the login/registration section red or vice versa.

We also noticed that the customization screen was not as accessible as we had hoped. We are going to add a label telling people that they can press ESC or click the mouse to pause the game and get to the customization screen. We are considering removing mouse control (for controlling the red blood cell character) because it seems too complicated for our users. If we do get rid of the mouse option, then we can also add a button that people can click to get to the customization screen. Our users generally didn't seem to have that much trouble dealing with the customization screen itself; the process of getting to the screen is what really gave them the most trouble. Once we reduce or eliminate this obstacle, everything related to control customization and preference-setting should go smoothly.

The instant-death aspect of the plaque is a nuisance that we noticed our users hated. As the "maze" of the game grows in complexity and length, dying from a single run-in with the plaque will become an even more unreasonable and disheartening experience. This frustration may cause our users to no longer want to play our game, which of course would be a disaster for us as designers. To fix this problem, we will be adding a "health" aspect to the game. The user will start out with 5 bars of health, and each time s/he hits the plaque, s/he will lose one half of a health bar. We could just do 10 bars of health, and then each hit costs 1 full bar of health, but we chose to go with the 5 bars of health because it allows us to add a little creative spin on the typically mundane health bar seen in so many games. Instead of using bars, we are going to use little hearts. When the user loses a half bar of health, we can show one of the little hearts breaking down the middle and then one half falls off (and disappears). This is an apt adjustment because the point of the game is to reach the heart; losing a heart is therefore a bad thing. We are going to put this creative health bar in the upper right corner of the game screen. In this location it will be out of the way but always quickly accessible/visible.

Users should also have the option to restart the game if s/he feels that s/he is not performing well or is going about the game incorrectly. Therefore, there will also be either a "New Game" or "Restart Game" button or shortcut that a user can access while in game. Furthermore, addressing how disheartening it is to die from a single run-in with plaque, it'd be even more so disheartening if one were far into the maze. This is why we've decided to also offer users the option to save their game so they can continue from where they left off. As long as users save their game before dying (before running out of "health"), they can continue from where they last saved.

Finally, the high scores screen should be accessible to all users (even unregistered visitors) without starting a game, because 1) it makes no sense to only be able to view high scores during a game, and 2) it will be useful for users wanting to show off their score; they should be able to do this without having to go through logging in and everything. To remedy this situation, we are going to add a button on the first page to let visitors view the high scores. We are not going to allow Guests to record their high scores, so we should include a note on the high scores screen that reminds them of this; that will hopefully encourage them to register. In addition, hopefully seeing these high scores will trigger the competitive nature in people who happen to wander onto our game and that will give them the push to register and engage in The Cell's Search.

Appendices

Demonstration Script

Greeter: "Hello ______________. Thank you for coming and participating in our prototype study. We appreciate your time and your feedback a great deal."

Greeter asks user: - Major - Year - Career plans

Greeter then carries on light conversation about unrelated topics (i.e. how they are doing, how school is going, background on group members, etc.) while Observer #1 sets up in a convenient location with computer to take notes. Also, the Computer sets up the prototype.

When everything is set up and ready, Greeter asks user to sign consent form.

Greeter now takes on the role of Observer #2.

Facilitator: "So you might be wondering what this entire 'user study' is all about. We are all in CS 160 which is a class on user interfaces. For the class, we have been working on a semester-long project that will give us experience and insight in designing and implementing user interfaces. The theme for the project is 'games with a purpose'. Games with a purpose are exactly what they sound like: games that are fun to play, but still serve some beneficial underlying purpose to the user or to society at large. For our GWAP, we are designing a game for medical students that teaches them about the human vascular system. We have come up with a maze-like game where the user takes on the role of a deoxygenated red blood cell that is trying to find its way back to the heart. The user's task is to traverse through the network of vessels to achieve this goal. In doing so, the user learns and internalizes the names, locations, and interconnectedness of the vessels. This is a valuable tool especially when learning from a book becomes tedious and daunting. The game we have developed is a flash-based web application that can be played off a website. We are in our initial design steps so what we are doing today is trying to get feedback from real users, like yourself. We have constructed an interactive prototype that will simulate our actual application. Your job today will be to perform some tasks that real users will be expected to perform in our game. Our prototype is only meant to test certain tasks, so not every feature has been implemented yet. In the early steps, we know there will be a lot of things that don't work so it will be easier to fix all those things before we spend a lot of time actually building the game.

It is really important that you understand that all your feedback is important to us: both positive and negative. Don't feel scared about giving negative feedback or telling us something is confusing/sucks/etc. We promise we won't take it personally! Also, don't feel bad or frustrated if you can't figure out how to perform a task - we aren't here to critique you; we are here to critique our interface. So if you are getting stuck somewhere or doing something incorrectly, it's because we have designed it poorly and needs to be fixed!

The way this test will work is that I will be giving you instructions. Denise will be playing the role of the computer; whenever there are any technical difficulties with the laptop, she will be here to assist. Please feel free to request her assistance if you feel something is malfunctioning with the electronics, such as the mouse or laptop itself. We will be giving you a short demo in a second so you can see how this prototype works.

It is crucial that you think aloud when you are performing your tasks. Just say aloud what you think about how the task is going: i.e. is it easy to navigate to where you want to go, are the menus/options/etc. easy to understand, is something confusing, or are there changes you would make. You can also tell us if you like something / dislike something or if you think it's cool, etc.

OK - we are now going to begin. I will start by performing a task that a normal user would perform while playing the game. The purpose of this demo is to just show you how the interface is supposed to work.

While in the game, users may desire to turn on the chat feature in order to discuss the game with other users. I will show you how I would go about doing just that."

[Facilitator: *Logs into game and brings up game settings menu*]

"OK, so now that this screen is up, you can see there are several in-game options, one being 'Chat'. I will check the box for 'Chat' and save my settings."

[Facilitator: *Checks chat box and hits 'Save and Close'*]

"So that was a quick demonstration about how the interface works and now it's your turn!"

Instructions to Users

• Task 1
  • Your task is the following: Please sign in to the system.

• Task 2
  • Your task is the following: Please move your red blood cell character from his current location through the top right vein.

• Task 3
  • Your task is the following: Please change the keys used to control the character.

Raw Data

Gustavo

Task One

    Time: 25.0 s
    Incorrect actions: 1 (Email input)
    Comments: "Not hard"; prefers page to say "New user? Register" to clarify for new users;
    error messages should pop-up and not require user to hover over erroneous field in order to see error message;
    change error outline box color -- currently red so blends in with background.

Task Two

    Trial One:
    Time: 24.8 s
    Incorrect actions: 3
    Comments: "What are you supposed to do?"; "What was supposed to happen?",
    instructions only tell you how to move but not where to go - no objective/goal described,
    user ran into plaque in bottom vein.
    Trial Two:
    Time: 31.7 s
    Incorrect actions: 0
    Comments: "Why won't it stop telling me I won?"; user found heart

Task Three

    User asked to change keys used to control character.
    Time: 35.7 s
    Incorrect actions: 0
    Comments: Beginning/instructional pop-up should clarify what menu consists of
    or word instructions more clearly with regards to what menu will pop-up when you
    hit ESC or click the mouse during the game.

Feedback

    - Did not like interface design (color scheme, overall design, etc.)

Malin

Task One

    Time: 26.1 s
    Incorrect actions: 0
    Comments: None

Task Two

    Trial One:
    Time: 47.4 s
    Incorrect actions: 3
    Comments: Technical difficulties with plaque/boundaries; unsure of controls; tried with mouse and got off track
    Trial Two:
    Time: 19.4 s
    Incorrect actions: 1
    Comments: Began with mouse then switched to arrow keys;
    figured out controls and proceeded smoothly;
    ran into plaque when past the diagonal portion of the inferior vena cava.
    Trial Three:
    Time: 19.9 s
    Incorrect actions: 0
    Comments: Used only arrow keys for controls; reached the heart.
    "This should be the 'hard' task. You should switch this with the 'hard' task."

Task Three

    User asked to change keys used to control character
    Time: 25.2 s
    Incorrect actions: 0
    Comments: None

Feedback

    - Once you win game, you can't do anything; pop-up alert just keeps popping up when you attempt to move.
    - Design is too red.
    - Didn't know point of game until played a few times and reached the heart; actual game interface does not state anywhere regarding point of game.
    - No instructions state to avoid plaque.
    - No directions as to what to do in game, only how to move/control character.
    - Mouse control is faulty, not smooth. Also, after editing game settings and clicking "Save and Close,"
      the game detects where mouse is (outside plaque) and you immediately die due to position of mouse upon clicking "Save and Close" button.
    - Doesn't seem reasonable to use mouse or offer that option because as screen goes further, you lose desk space to move mouse.
    - Should have an option to restart game in the middle of playing the game.

Cecilia

Task One

    Time: 45.0 s
    Incorrect actions: 0
    Comments: This girl cannot type. Seriously. Cannot type.
    "There's no problem with how you guys did this part, I'm just a slow typer."

Task Two

    Trial One:
    Time: 10.3 s
    Incorrect actions: 1
    Comments: Started with the keyboard controls, but hit the plaque pretty quickly.
    Trial Two:
    Time: 36.2 s
    Incorrect actions: 1
    Comments: Ran into plaque in the diagonal part.
    Trial Three:
    Time: 55.2 s
    Incorrect actions: 0
    Comments: Finally reached the heart, but went really slow so she wouldn't hit the plaque.
    

Task Three

    User asked to change keys used to control character
    Trial One:
    Time: 38.7 s
    Incorrect actions: 1
    Comments: After trying for a while, she confessed that she could not remember how to bring up the controls menu.
    Trial Two:
    Time: 40.4 s
    Incorrect actions: 0
    Comments: Changed to the WASD keys.

Feedback

    - The controls menu should be more plainly accessible.
    - The heart should be more prominent since it's the end goal of the game.
    - Celly is a cutie.

---