LoFi-Group:NGRS
From CS160 User Interfaces Sp09
Contents |
Introduction and Mission Statement
Contributions: Dwij Garg: Prototype creation, write up, conducted user study Denise Ngai: Video prototype creation, write up, conducted user study Siddharth Shah: Intro, Mission statement, Methods write up, conducted user study Salman Rahman: Results, Discussion write up, conducted user study
Medical students taking human anatomy classes are graded based on knowledge of the subject matter and these grades are extremely important for getting into competitive residencies. As such, students are highly motivated to learn the material. However, learning this material can be extremely cumbersome because traditional methods of memorizing names, locations, and functions of vessels are boring and unrelatable. Imagine trying to memorize a road map of New York City, including what roads connect what areas to what other areas, by just studying the map. Without driving those roads and letting your brain form the connections between what roads are connecting what parts of the city, an individual will have a poor mental map of the city. Thus, designing a new way to learn about the human vascular system that is interactive, exciting, and intuitive would be of great benefit to students.
Our project is designing an interactive game, called "The Cell's Search", that would teach students (specifically medical school students) about the vascular system. The premise of the game is as follows: the user takes on the role of a deoxygenated red blood cell that starts off in some venous vessel in the body. The user, under time constraints, attempts to navigate through the maze of blood vessels in the body in order to get back "home" to the heart so the cell can be reoxygenated.
It is the interface for this interactive game that we will be evaluating today. We are in our initial phases of interface design for this game. Up to this point, the generated design has only been used by the design team - we have not tested this system on real users. User study is crucial to developing a final product because the design team is biased in terms of interaction with the system because they are very familiar with it. Real users might not have the same level of ease in navigating the interface.
Because designing a hi-fi interface (i.e. an interactive computer model) is extremely time consuming and more difficult to implement changes into, we have constructed a low-fidelity prototype of our game (using construction paper, tape, glue, and other arts/crafts materials) to study with test users. We will test user interactions with this low-fi representation of our system by giving users three tasks to perform and monitor how they are able to perform these tasks. Using this data, along with user comments, we can make changes to our design that will ensure a more user-friendly final product.
Mission statement: NGRS strives wholeheartedly to develop an interactive maze game that truly helps students learn about the human vascular system. We endeavor to design and implement an interface for this game that is engaging and exciting for the user, while being easy to learn for novices, yet flexible enough to be advantageous to experts.
Prototype Description
Our prototype consists of three tasks that a user will encounter when playing our game. Task One involves the user logging in with his username and password or signing up for a username if he doesn't have one yet. A user is required to have a username and password in order to play the game because his settings and scores will be recorded. Task Two involves controlling the red blood cell character as it traverses through the vascular system in order to reach the heart. The user must use the keyboard controls in order to make sure the red blood cell character avoids plaque while on its journey to the goal. In Task Three, the user changes the game's control settings to match his own control preferences.
We designed our prototype using only paper because of its flexibility. Multiple pieces were made to complete our paper prototype system. Each task had its base layout, which had cut outs for inserting any text boxes, buttons, or other changeable options. These cut-outs were very useful in our prototype. For example, as can be seen in Task One, they allowed for us to simulate button pressing (by replacing the the current button with a grayed out button) or text-typing (by manually writing in text, letter by letter, onto a blank paper that was inserted in the text-box cut out).
Among the multiple pieces in our prototype system were the smaller moving parts, such as the text cursor, mouse arrow, and red blood cell character. These pieces were created to easily simulate movement across the interface/base layout for a task.
Task One
Task One consisted of a base layout that by default had the log in fields and an add-on text area for signing up (see figures 1.1 and 1.4). Each button was its own piece, and each button had it's "clicked on" counterpart (figure 1.2). Plain white slips of paper were placed underneath the text-box cut outs. These slips of paper were written on manually to simulate typing. The mouse arrow was used to simulate mouse movement and clicking on buttons and text boxes. A tiny text cursor was used to simulate a real blinking text cursor as seen on actually computer interfaces (figure 2.2). In addition, a paper keyboard was made to allow the user to simulate using a real keyboard while maneuvering around the interface (figure 1.3).
[Figure 1.1] Task One: Logging In [Without buttons]
[Figure 1.2] Buttons used for Task One
[Figure 1.3] Paper keyboard
[Figure 1.4] Task One: Signing Up
Task Two
Task Two consisted of a base layout that featured the actual game-play interface -- timer slot, vascular path, location map, etc. The timer slot was made as a cut-out so the timer could easily be simulated by writing in different times onto blank sheets of paper that were inserted into the slot. The red blood cell character (figure 2.2) was placed in the vascular path and moved according to keys pressed on the keyboard (figure 1.3).
[Figure 2.1] Task Two: Controlling the Red Blood Cell Character
[Figure 2.2.] The middle object is the red blood cell character featured in the game.
Task Three
Task Three featured a base layout with several cut-outs. The base layout had columns for the changeable settings and on each side of the settings were cut-outs for the selection marker (the tiny red blood cell character) and the actual controls for the settings (figure 3.2).
<center>
[Figure 3.1] Task Three: Changing the Control Settings
[Figure 3.2] Items used in Task Three. The mini red blood cell character is used as a selector. 'W', 'A', 'S', 'D' and the arrows represent keyboard keys for controls.
[Figure 3.3.] Pop-up that took you to the Control Settings while in game-play.</center>
[Figure 4] All prototype paper materials used.
Video Prototype
Our low-fidelity video prototype was made with paper pieces that were designed to simulate the user interface of our game, "The Cell's Search." The prototype consists of multiple paper pieces that were used to simulate the animation that would actually happen in the real game. A video editing program (SAM Animation) and a webcam (provided in class) were used to capture still shots of the paper prototype in order to create a stop-motion video prototype to simulate the movements that would occur in the game. These paper prototype pieces were slowly moved around the interface and pictures were taken frame by frame in order to create a close-to-real-life simulation.
During the creation of the stop-motion video prototype, double-stick tape was used in order to anchor the pieces in place in order to ensure that they remained in the same place while other pieces were moved around the interface. We paid close attention to small details, such as the blinking text cursor in the sign-up and log-in screens and the timer in the actually game screen. The simulation of the blinking text cursor was simple but tedious; a tiny piece of paper with the drawing of a text cursor was placed and removed repeatedly in order to create the blinking animation. Voice-over was added at the end to supplement the video and help viewers understand what is going on.
The main difficulty in the production of this video prototype was moving the little paper pieces around the interface, especially the mouse cursor and blinking text cursor. These little pieces were tedious and difficult to remove or keep in place. Another difficulty was changing multiple items on the interface at once while keeping them steady. Pieces had to remain in the same position or close to the same position in order to look realistic in the animation. When moving multiple pieces, it was difficult to keep stationary the other pieces that weren't supposed to move. Lighting and shadows were another difficulty. It was hard to capture an image without having some sort of shadow from the production equipment appear in the shot.
Aside from the difficulties of the production, several things worked well. For instance, the turn out of the mouse cursor moving across the interface was pretty realistic. In addition, the simulation of pressing a button also came out very well. Dealing with the interface when only one item was in motion was easy to do. The program used to capture and edit the stop-motion videos, SAM Animation, was actually very easy to work with. Its interface is simple and the program could handle all simple edits that were necessary to the video, such as copying frames and changing speeds. It didn't take too much time to become familiar with the program, and actually putting the images together to create a video was fairly simple. Adding voice-narration on the videos was also very simple with this program.
Introduction
Easy Task
Logging In
Signing Up
Medium Task - Controlling the Red-Blood Cell Character
Hard Task - Changing Control Settings
Method
Participants
For this study we wanted to select users that were in our specific target user group: pre-med and medical school students. Because we are designing an application for this target base, it is the feedback of these users that would be most directly beneficial in improving our interface design. We also wanted to have a mixed selection of individuals in terms of familiarity with and daily consumption of video games because familiarity with games, specifically online flash-based games, would create a confounding in our experiment: these are expert users who would be able overcome design flaws because they are familiar with game interfaces. Two of the three subjects, "Mike" and "Jim", are pre-medical students at Berkeley who are currently taking human anatomy. We were able to recruit Mike and Jim for the study by relying on one of our group members who is taking an MCB class with them. The third subject, "Jane", is a first-year medical school student who is also taking human anatomy along with an anatomy lab which involves human cadaver dissections. We were able to recruit Jane for the study because one of our group members works at a lab at UCSF along with Jane.
We collected some background information about the users and present it here.
Jane
Jane differed from our other interviewees in that she is currently a medical school student and the amount of material in her anatomy classes is much more vast. She has never supplemented learning with any games and doesn't usually play games. When she does, she enjoys puzzles and riddle-style games. She gets most of her studying done at home and spends 4-5 hours a day studying. She usually gets a B+ in her classes and is looking to do better. She uses flash cards, textbook, and online research to supplement her learning. Specifically in anatomy, she supplements her learning with the lab portion that involves human cadaver dissection. This helps her learn more and visualize more but cadavers are high in demand and short in supply. It's hard for her to do all the dissections and visualizations as she has group mates whom she has to share time with. She would be open to supplementing her knowledge with a game.
Jim
Jim has never used a game to supplement knowledge in a course before but often plays video games at home. He enjoys RPG style games and avoids FPS. He spends 3 hours a day studying for his biology classes and nearly 5 for a midterm. He prefers to study at home. He ranges from B+ to an A in his biology classes and wishes he could do better. He finds the amount of memorization for classes to be daunting and would like a way of learning that is less intensive on memorization. He finds that he has to re-read the material over and over and over in order to get a firm grasp. In studying for the vascular system he reviews slides, reads the textbook and traces pathways. He internalizes the information through rote memorization. He supplements learning through online research and attending office hours. If given the option to play a game that would help solidify his knowledge he would like to do so and consider it studying.
Mike
Mike revealed that he prefers studying alone and at home/library but he had most of his studying done so he decided to do a study group. He doesn't play very many video games, but when he does he plays them at home. He has never used an educational game before to supplement his knowledge of course material. When he does play games, he enjoys games that have a competitive aspect to it. He typically spends 3 hours a day studying for his anatomy class and about 12 hours studying for a midterm. He typically receives a 'B' in biology classes and would like to do better than that. In anatomy, he finds the amount of memorization excessive but not daunting. In order to supplement his book learning, he looks up figures, diagrams, and explanations on the internet of concepts he finds difficult. Usually after performing these functions he feels he has a firm grasp on the material.
Environment
This user study was conducted in the living room of one of our group member's apartments. This location seemed to be most convenient for all parties involved as it was close to campus on South side. Users were interviewed one at a time and only one user was at the apartment at any given point. This is for obvious reasons (so as not to unfairly give advantage to other users). There were refreshments and snacks set up for the user in the kitchen and he/she was notified that they could take a break at any time and get back to the tasks later. We set up a large table where the interface would be laid out. The user would be sitting on one side of the table and the computer would be sitting on the other side. The computer had all the prototype screens/menus/etc. laid out on her side of the table so as to be easily accessible, yet not visible to the user. Also, the computer was given index cards / post-it notes / tape / pencils / pens / markers in case he/she needed to make messages/menus/etc. on the fly in response to user input that was unforeseen (though we had anticipated nearly all possible interactions, it doesn't hurt to be prepared). For more information about the prototype, read "Prototype" section above. The facilitator stood opposite the user, standing next to the computer. The two observers were about 4-6 feet away from the table on the ends so as not to crowd the user and make him feel at ease and relaxed about the study. Observers were recording information on laptop computers and were required to stay quiet and not react to any of the users behavior.
Procedures
Greeter - Siddharth Shah
Facilitator - Salman Rahman
Computer - Denise Ngai
Observer - Dwij Garg
The detailed procedure that we went through is discussed in the appendix (i.e. demo script) but we will recapitulate the information here.
- 1. Greeter was responsible for receiving each partcipant at the apartment, introducing them to group members, getting consent form signed, making small talk while things were set up, and just ensuring comfort of the subject overall.
- 2. Facilitator was responsible for conducting the demonstration, explaining tasks to user, probing the user for feedback, and leading the user study overall.
- 3. Computer was responsible for responding to user input by manipulating the prototype
- 4. Observers were responsible for recording notes during the study process
- 5. Greeter asks user:
- Major
- Year
- Career plans
- 6. Greeter then carries on light conversation about unrelated topics (i.e. how they are doing, how school is going, background on group members, etc.) while Observer #1 sets up in a convenient location with computer to take notes. Also, the Computer sets up the prototype.
- 7. When everything is set up and ready, Greeter asks user to sign consent form.
- 8. Greeter now takes on the role of Observer #2.
- 9. Facilitator now takes the lead and gives introduction to project
- Describes our overall project goal and mission statement
- Details reasoning behind developing low-fidelity prototype
- Informs why we are doing the user study / why it's necessary
- Impresses importance of user's feedback to our design goals
- Eases user's fears about "messing up" or giving negative feedback
- 10. Facilitator then goes through the demonstration of how the protoype works and how the user interacts with the system
- Lets user know that computer is "dumb" and only reacts to user inputs
- Shows how clicking / typing is achieved
- Shows how new windows appear based on input
- Asks if user has any questions about how the prototype is to be used
- Lets user know he/she will be probed for thoughts during the tasks
- 11. Facilitator gives user three tasks (one at a time): signing in to the game, controlling the red blood cell character, and changing the game control options
- Refer to appendix for exact instructions given to user
- Probes user for input about what he or she is thinking, feels about usability of the interface, etc. when the user has stopped talking
- Observers take notes on user feedback / user actions
- 12. After study conducted, facilitator conducts wrap-up
- Asks user for likes/dislikes of interface
- Asks user for overall impression of the design
- Asks user which of the tasks was most difficult and why
- Asks user if he/she has any suggestions for improvements
- Asks user which things he/she would most like to keep
Test Measures
For our dependent variables, we wanted to quantitatively measure something that we could compare across users: time and number of incorrect inputs. These are easy to measure and variability between users would suggest familiarity with the system. Variability between tasks will give an indication of the difficulty of the task as well as possible flaws in the design. Qualitatively, we measured the user's comments, impressions, and any gestures made. These would give an indication as to how difficult the user is finding the current task and will be helpful in pointing out problems with the current design.
Bottom-line Data
- Time taken to complete task
- Number of "incorrect" actions
Process Data
- General impression (as assessed by facilitator's questions)
- Comments made
- Facial expressions
Tasks
Task One: Logging In/Signing Up
"Task One: Logging In/Signing Up" is an easy task. This simple task only requires the user to type in his username and password and click the "Go!" button. Once the user clicks the "Go!" button, he is brought to the "Welcome" screen, where he can continue on to play the game. If the user does not yet have his own account in the game, he can simply click the "Sign Up" button and type in his desired username, password, and email address. Upon clicking the "Create" button, his account is created and he is brought to the "Welcome" screen, which confirms that his account has been created. From here, he can proceed to the game.
Task Two: Controlling the Red Blood Cell Character
"Task Two: Controlling the Red Blood Cell Character" is a medium task. Depending on the user's control settings, the user has to help the red blood cell character maneuver through the vascular system while avoiding dangers, such as plague. In addition, the user must decide which paths are the correct ones to take in order to reach the final goal destination -- the heart. For example, in our prototype, the user has to determine whether to choose the upward diagonal path or the lower diagonal path while avoiding the plaque-covered walls.
Task Three: Changing the Control Settings
"Task Three: Changing the Control Settings" is a hard task. The user has to halt his game play in order to see the pop-up menu, from which he chooses to resume the game, change the control settings, or see game scores. From there, he can choose "Control Settings" in order to customize how he wants to maneuver throughout the game. The user can choose whether to use letters on the keyboard to move up (W), down (S), left (A), or right (D) or he can use the arrow keys. In addition, the user can choose whether he wants the chat, hints, and vein names to remain off/on during the duration of his game play. These are aspects of the game that affect the user's gaming experience.
Results
Task 1: Easy
Across the board, this seemed to be not only the easiest task for our subjects, but it also was completely error free. No one messed this task up - not even slightly. Overall, people commented about how easy the task was and that it was very self-explanatory and the interface was simple to navigate and understand.
Mike
- Time: 40 seconds
- Incorrect Inputs: 0
- Comment: "All the fields are self explanatory"
- -I don’t agree this is a usability problem.
- Comment: "Oh now I have to check my email" (after signing up - he thought this because it asked for his email address)
- -Major Usability Problem - clarify wether confirmation email is being sent or not
- Comment: "Oh wow thats great" (In reference to the fact it automatically logged him in after sign up - he didn't have to check email)
- -I don’t agree this is a usability problem.
Jane
- Time: 45 seconds
- Incorrect Inputs: 0
- Comment: Liked that it auto logs in after signing up
- -I don’t agree this is a usability problem.
- Comment: "Very easy to do / fields easily navigatable"
- -I don’t agree this is a usability problem.
- Comment: Liked that it doesnt take you to a different web page - all within the same interface
- -I don’t agree this is a usability problem.
Jim
- Time: 20 seconds
- Incorrect Inputs: 0
- Comment: "This is super easy - give me something harder to do"
- -I don’t agree this is a usability problem.
- Comment: "Everything is self explanatory i know exactly where to go and what to do"
- -I don’t agree this is a usability problem.
Task 2: Medium
Result Summary: YIKES! None of the users knew how to control the character and had to guess that you use the arrow keys on the keyboard (some were able to guess correctly on the first try, while others struggled to try different options). The users also posed wonderment as to why the mouse would not control the character. Also, while some users were able to correctly identify the plaque lining the vessel walls as dangerous, others were not sure and ran into it, killing their character. This seemed to be by far the most difficult task for users.
Mike
- Time: 90 seconds
- Incorrect Inputs: 4
- Observation: User wasnt sure how to start controlling the guy
- -Usability catastrophe: imperative to fix
- Observation: He ended up killing the guy by pushing down - he wanted to see what happens (for fun)
- -I don’t agree this is a usability problem.
- Comment: "Oh so when you don't push anything the guy just sits there ... I can use this to my advantage"
- -I don’t agree this is a usability problem.
- Comment: "Hm.. I'm not sure if I could have used the mouse to move either ... I just guessed to use the keyboard"
- -Usability catastrophe: imperative to fix
Jane
- Time: 35 seconds
- Incorrect Inputs: 0
- Comment: Assumed you use keyboard
- -Usability catastrophe: imperative to fix
- Comment: "LOL low-fi is hillarious"
- -I don’t agree this is a usability problem.
- Comment: wondered what happens if you use the mouse
- -Major usability problem: important to fix
- Comment: "Stuff on veins looked dangerous so avoided it"
- -I don’t agree this is a usability problem.
- Comment: "Map helps to decide where to go"
- -I don’t agree this is a usability problem.
- Comment: "This game looks fun!"
- -I don’t agree this is a usability problem.
Jim
- Time: 120 seconds
- Incorrect Inputs: 6
- Observation: User tried moving character with mouse - didn't work
- Comment: "Not sure how to move character ...since mouse doesn't work im guessing its the keyboard"
- -Major usability problem: important to fix
- Observation: User tried using "W A S D" controls like counter strike, didnt work
- Comment: "WTF"
- -Major usability problem: important to fix
- Observatoin: User tried using arrow keys and it worked
- Comment: "Finally...you need to give instructions or something on how to make the character move"
- -Major usability problem: important to fix
- Observation: User hit the plaque and died
- Comment: "How was I supposed to know not to hit that? You need to make this stuff clearer"
- -Major usability problem: important to fix
Task 3: Hard
Result Summary: Overall users seemed to do OK on this task but nearly all users got caught on the same point. They thought that they could customize the movement options to any key of their choice (i.e. change the key for turning right to the letter "L"). However, this is not the case in the current implementation of the interface: users can only toggle back and forth between arrow keys and the "W-A-S-D" layout keys. Also, it is significant to note that one of the users mentioned there is no instruction as to how to leave the control menu (i.e. users guessed to hit the ESC key) and the menu does not note that changes to the options have been saved. Users did seem to understand what the options were referring to, though, citing that the labels and layout were easy to understand and read.
Mike
- Time: 60 seconds
- Incorrect Inputs: 2
- Observation: He first tried spacebar then enter key to change the option but it doesn't work since its controlled by mouse so he wants that to be put in
- -Major usability problem: important to fix
- Comment: He doesn't like the fact that the selection doesnt wrap when you use the keyboard and move off the page
- -Major usability problem: important to fix
- Comment: He doesn't know that it saved his changes
- -Major usability problem: important to fix
- Comment: He guessed that you have to hit ESC to exit the menu
- -Major usability problem: important to fix
Jane
- Time: 80 seconds
- Incorrect Inputs: 2
- Comment: Menu pops right up and pauses game! Awesome!"
- -I don’t agree this is a usability problem.
- Comment: "Could i have used the keyboard to select control setting?"
- -I don’t agree this is a usability problem.
- Observation: Tried to type something in but it didn't work
- -Major usability problem: important to fix
- Comment: "Assumed you could have selected any key you want to move, not just selecting between two options"
- -Major usability problem: important to fix
- Comment: "Easy to understand what options are saying"
Jim
- Time: 60 seconds
- Incorrect Inputs: 1
- Comment: "Very obvious how to pull up menu ... says right on the screen"
- -I don’t agree this is a usability problem.
- Comment: "That's pretty wack that you only have two options for movement ... it would be better if user could decide their own"
- -Major usability problem: important to fix
Discussion
This user study was immensely successful in providing us with feedback to guide our design. As designers, we have become experts in our interface and thus are able to overcome deficiencies in the system through our expert knowledge. It is really helpful to have a fresh mind use the system in order to determine flaws. Let's go through, task by task, and discuss what we have learned.
Our first task of having the users sign up and login went very smoothly. All users cited that it was easy to determine what to do and that fields were labeled intuitively. All users liked the fact that the program automatically logged you in after you sign up, so you don't have to worry about opening up your email and looking for a confirmation link, etc. Also, users liked the fact that the sign-up form was in-game and you were not redirected to another site. This prevented having to re-navigate back to the game and start over. One of our users, Mike, did point out something significant to keep in mind though. In our current interface design, we had forgotten to realize that we need to send a confirmation email to users after the sign up. We will implement a change in our design that, upon signing up, users will see text that directs them to check for a confirmation email in the near future. Users will have 24 hours from signing up to click on the link to ensure activation of the account. They can begin play right away, as they will be logged in automatically; however, if they don't activate the account within 24 hours, the login will no longer be functional. This will prevent users from giving fraudulent email addresses while still not inconveniencing users during game play by having to check their email and activate the account.
The second task seemed to go pretty disastrously for all users - a direct result of our poor design. None of the users knew how to begin moving the character. Some tried moving the mouse, to no avail. Others tried using the "W-A-S-D" keys which is so common in modern games, like Counter-Strike. Only after trial-and-error did users discover that the arrow keys are used to control the user. After this, as users tried to navigate the player, they hit another snag point: the plaque lining the vessel walls. Some were able to understand that the plaque was to be avoided, while others did not and ended up dying. In response to the user confusion about how to move the character and plaque, our group has decided to include a pop-up that has a quick and dirty game-play guide which lays out the controls and basic premise of the game. Novice users will be able to read the guide and quickly determine how to play the game, without going through a long online manual. Expert users will be able to close the pop up quickly. We might even have an option on the pop-up that says "Do not open in the future" that when selected, does not open the pop-up menu for that user log-in. Furthermore, all users cited that they would like to have the ability to control the character by both mouse and keyboard. Thus, we will update our design to allow for that. Once users got going, they were able to figure out that at a junction, you must either go up or down and selecting one route will take you into that new vessel and into a new frame. Many users also cited the fact that the map seems to be helpful, but they were not sure that it was a map of the body at first, or they failed to notice it. One user, Mike, had a wonderful suggestion that we could use to solve this problem. We can change our navigation design to have the heart beat (a simple animation where the heart goes smaller then larger, like in cartoons). This will attract the eye of the user, so they pay attention to the fact that the map is there to help them and it makes it pretty self-explanatory what the map is of.
The last task also received a large number of complaints from our users. All users were able to figure out how to open the in-game menu and select the control settings. They cited that this was easy to do because it had been labeled on the game screen. They all found the automatic pausing of the game and pop-up nature to be "cool" and easy to use. All users earnestly expressed the desire to be able to customize their own control settings. As the system currently stands, users only have two movement layout options: the arrow keys vs. the W-A-S-D letters. However, users assumed the system was set up to allow them to select their own layout, as most games allow you to do this. Thus, in our redesign, of the control options, we will include this feature. On a positive note, users stated that the options were laid out in an easy to read manner and were easily understandable. Nothing was ambiguous about the options. They also liked that you know which option you are editing because a small red blood cell character appears next to the selected option. This makes it easy to use keyboard keys to navigate the menu. One user did express that if you use the keyboard and move the selection of the page, it should wrap around, which it currently does not do. We will also implement this change because we think it's a good idea. There were two other major usability problems with this page. One was that there was no button or instruction as to how to quit from the control setting menu. Users had to guess to hit the ESC button to get back to the game menu, from which they could resume the game. This was a fault of our design, as we had overlooked exiting the menu. Next, users did not know if by hitting ESC their changes would still be saved. They expressed that it would be nice to have some kind of confirmation that allows them to know that their changes are permanent and stored.
Though this low-fi prototype user study was able to reveal a lot of design flaws and design strong-points, there are a few things that it could not reveal. The first is the effectiveness of this game in improving students' understanding of the human vascular system. This experiment was not designed to measure test score differences when users supplement studying with game play vs. just studying via traditional book learning. That is a more complicated experiment and would require much thought in setting up and executing properly. It is an important test to conduct however, because if this game offers no advantage, or worse, offers a disadvantage compared with learning from the book, then our game is purposeless! Furthermore, this test was not able to reveal whether or not this game is fun to play for users. Because of the low-fidelity nature of the prototype, the interactivity is choppy and slow. Thus, users do not get a feel for the rush and excitement of performing these tasks under timed conditions. Also, the prototype was not set up to be a maze, like the real game is, so users did not have to try finding the heart. Since that is the entire premise behind the game, we would have to set up a prototype that allows users to do that in order to gauge the "funness" response from users.
Appendix
Consent Forms
We have not included user consent forms here in order to protect subject confidentiality. They are included in our printout.
Demonstration Script
Greeter: "Hello ______________. Thank you for coming and participating in our prototype study. We appreciate your time and your feedback a great deal."
Greeter asks user: - Major - Year - Career plans
Greeter then carries on light conversation about unrelated topics (i.e. how they are doing, how school is going, background on group members, etc.) while Observer #1 sets up in a convenient location with computer to take notes. Also, the Computer sets up the prototype.
When everything is set up and ready, Greeter asks user to sign consent form.
Greeter now takes on the role of Observer #2.
Facilitator: "So you might be wondering what this entire 'user study' is all about. We are all in CS 160 which is a class on user interfaces. For the class, we have been working on a semester-long project that will give us experience and insight in designing and implementing user interfaces. The theme for the project is 'games with a purpose'. Games with a purpose are exactly what they sound like: games that are fun to play, but still serve some beneficial underlying purpose to the user or to society at large. For our GWAP, we are designing a game for medical students that teaches them about the human vascular system. We have come up with a maze-like game where the user takes on the role of a deoxygenated red blood cell that is trying to find its way back to the heart. The user's task is to traverse through the network of vessels to achieve this goal. In doing so, the user learns and internalizes the names, locations, and interconnectedness of the vessels. This is a valuable tool especially when learning from a book becomes tedious and daunting. The game we have developed is a flash-based web application that can be played off a website. We are in our initial design steps so what we are doing today is trying to get feedback from real users, like yourself. We have constructed a very low-tech paper prototype that will simulate our actual application. Your job today will be to perform some tasks that real users will be expected to perform in our game. The purpose of making this prototype very low-tech is that it saves us the hassle of having to spend a lot of time making huge revisions. In the early steps, we know there will be a lot of things that don't work so it will be easier to fix all those things before we spend a lot of time actually building the game.
It is really important that you understand that all your feedback is important to us: both positive and negative. Don't feel scared about giving negative feedback or telling us something is confusing/sucks/etc. We promise we won't take it personally! Also, don't feel bad or frustrated if you can't figure out how to perform a task - we aren't here to critique you; we are here to critique our interface. So if you are getting stuck somewhere or doing something incorrectly, it's because we have designed it poorly and needs to be fixed!
The way this test will work is that I will be giving you instructions and Denise will be playing the role of the computer. She won't be able to talk to you or give you any advice, help, etc. She will just react to your input like a computer would. She will layout the screens in front of you, just as you would see them on your computer. You have the paper keyboard here to use, like you would your normal keyboard and you should use your finger as a mouse. So if you want to click something, just take your finger and press down on the button. If you want to type something, type it on your paper keyboard and say what you are typing aloud so Denise can receive your input and react accordingly. We will be giving you a short demo in a second so you can see how this prototype works.
It is crucial that you think aloud when you are performing your tasks. Just say aloud what you think about how the task is going: i.e. is it easy to navigate to where you want to go, are the menus/options/etc. easy to understand, is something confusing, or are there changes you would make. You can also tell us if you like something / dislike something or if you think it's cool, etc.
OK - we are now going to begin. I will start by performing a task that a normal user would perform while playing the game. The purpose of this demo is to just show you how the interface is supposed to work, how you interact with the "Computer" (aka Denise).
When users select to play in single-player mode, they are required to choose the difficulty level for the game between easy / medium / hard. I will show you how a user selects the difficulty level once the screen comes up for that."
[Computer: *Places difficulty level screen up*]
"OK, so now that this screen is up, you can see the easy option is currently selected because there is a red dot next to the easy button. Now if I want to select the hard option, I can either take my finger (mouse) and push on 'hard' or I can use the keyboard down arrow key to move to 'hard'."
[Facilitator: *Hits down arrow key twice*]
[Computer: *Places red dot next to 'hard'*]
[Facilitator: *Hits ENTER key*]
[Computer: Changes screen to 'Loading Game' screen]
"So that was a quick demonstration about how the interface works and now it's your turn!"
Instructions to Users
Task 1
Your task is the following: Please sign in to the system.
Task 2
Your task is the following: Please move your red blood cell character from his current location through the top right vein.
Task 3
Your task is the following: Please change your control options - you can change any option you wish.
Raw Experiment Data
The following data is raw and unedited. Please refer to the results section for properly formatted and edited data.
task 1 user 1
comment: all the fields are self explanatory comment: he's thinking there should be a confirmation email comment: now he thinks he has to check his email (after conformation) comment: oh wow thats great
post task analysis
it was tight
what if he doesnt remember his password(auto login) confirmation email? it automatically logged u in so that was tight -- he likes the fact that the confirmation wasnt required yet look at the thing above
time: 40 seconds
- of incorrect: 0
task 2
comment: user wasnt sure how to start controlling the guy comment: he ended up killing the guy by ppushing down - he wanted to see what happens (we started over)
comment: he realizes that when u dont push anything nothing happens - and he will use that to his advantage comment: it wasnt clear wether he should use the mouse or keyboard - he thinks the guy should move on his own and u should tell him to go up or down
post - game issues:
he realises now that he missed the map which would have told him where to go..... label issues
time: 1:30 seconds
- of incorrect presses: 4
task 3
noticed: he clicked esc to exit to get the menu
noticed: he first tried spacebar then enter key to change the option but it doesnt work since its controlled by mouse so he wants that to be put in comment: he doesnt like the fact that the thing doesnt wrap when u use the keyboard comment: he doesnt know that it saved comment: he guessed that you have to hit esc
time: 1:00
- of incorrect presses: 2
POST THING FEEDBACK:
throbbing heart he thinks its tight that we did it low fi clarify mouse / keyboard interactions in the menu (allow you to control character with the mouse) signing in and setting up user name was clear
User 2:
task 1
comment: liked that it auto logs in after signing up comment: very easy to do / fields easily navigatable comment: liked that it doesnt take you to a different web page - all within the same interface
time: 45 seconds
- of missed clicks: 0
task 2
comment 1: low-fi prototype kind of hard to work with here comment 2: assumed you use keyboard comment: lol low-fi is hillarious comment 3: wondered what happens if you use the mouse comment 4: stuff on veins looked dangerous so avoided it comment 5: map helps to decide where to go comment 6: this game looks fun!
time: 35 seconds
- of missed clicks: 0
task 3:
noticed that user clicked mouse to pull up menu used mouse to click menu option
comment: menu pops right up and pauses game! awesome comment: could i have used the keyboard to select control setting? noticed: tried to type something in but it didn't work comment: assumed you could have selected any key you want to move, not just selecting between two options comment: easy to understand what options are saying
time: 1:20
- of missed things: 2
user 3
task 1
comment: this is super easy - give me something harder to do comment: everything is self explanatory i know exactly where to go and what to do
time: 20 seconds missed clicks: 0
task 2
user tried moving character with mouse - didn't work comment: not sure how to move character ...since mouse doesn't work im guessing its the keyboard user tried using "W A S D" controls like counter strike, didnt work comment: WTF user tried using arrow keys and it worked comment: finally...you need to give instructions or something on how to make the character move user hit the plaque and died comment: how was i supposed to know not to hit that? you need to make this stuff clearer
time: 2:00
missed clicks: 6
task 3:
comment: very obvious how to pull up menu ... says right on the screen user hit esc key, toggled with keyboard comment: that's pretty wack that you only have two options for movement ... it would be better if user could decide their own
time: 1:00 missed clicks: 1
