LoFi-Group:MechanicalSquirrel

From Cs160-sp08

Contents 1 Team Members 2 Introduction and Mission Statement 3 Prototype 3.1 New Character (B) 3.2 Home Screen (A) 3.3 Quests Screens (C) 3.3.1 Individual Quest Screens (C1A, C2A, C3B) 3.3.2 Quest Search Screen (C3) 3.4 "You" Screens (D) 3.4.1 Edit Avatar Screen (D2) 3.4.2 Edit Room Screens (D3) 3.5 Recent Events Screen (E) 3.6 Friends Screens (F) 4 Method 4.1 Participants 4.2 Environment 4.3 Tasks 4.4 Procedure 4.5 Test Measures 4.5.1 Process Data 4.5.2 Bottom Line Data 5 Results 5.1 General observations 5.2 Task results 5.2.1 Task 1: Finding a quest and "completing" it (medium) 5.2.2 Task 2: Viewing recent events (easy) 5.2.3 Task 3: Editing the room/avatar (hard) 5.3 Game Comments 6 Discussion 7 Appendix 7.1 Files 7.2 Raw Process Data 7.2.1 General comments 7.2.1.1 Participant 1 7.2.1.2 Participant 2 7.2.1.3 Participant 3 7.2.2 Task 1: Finding a quest and "completing it" (medium) 7.2.2.1 Participant 1 7.2.2.2 Participant 2 7.2.2.3 Participant 3 7.2.3 Task 2: Viewing recent events (easy) 7.2.3.1 Participant 1 7.2.3.2 Participant 2 7.2.3.3 Participant 3 7.2.4 Task 3: Arranging the room and editing the avatar (hard) 7.2.4.1 Participant 1 7.2.4.2 Participant 2 7.2.4.3 Participant 3 7.2.5 Summary of Issue Average Severity 7.3 Bottom Line Data

Team Members

• User:Eric Cheung - Eric wrote the results section and the appendix
• User:Alex Choy - Alex wrote the discussion section
• User:Hsiu-Fan Wang - Hsiu-Fan wrote the prototype section
• User:Glen Wong - Glen wrote the method section

Everybody made parts for the low fidelity prototype, worked on the introduction and mission statement, and went to all the experiments.

Introduction and Mission Statement

The system being evaluated is the interface for our Android application, a real world role playing game. The purpose behind this project is to encourage computer gamers to accomplish goals that involve real world tasks. To evaluate our game's user interface, we have created a lo-fi prototype and have tested it with three users. We introduced the system to the user, demonstrated a basic task in the system, and then had the user perform three tasks, observing any complaints, comments, reactions, etc. that the user had. By doing this experiment, we hope to improve the interface of our game so that the user spends more time having fun by doing tasks and obtaining items and less time figuring out how to get started.

Mission Statement: Many self-improvement systems and ideas exist now. However, they do not cater to every audience. Specifically, many "hardcore" computer game players lack the motivation and incentive to go outside. The mission of our project is to re-create features of online games in an outdoor and real-life setting that motivates gamers who play games at home to go out and do fun things to better their life.

Prototype

Low-fidelity user testing was conducted with a grid of base screens upon which we laid additional UI elements (for spinners, menus and simple art assets), the screens were then framed by the Android hardware buttons (see Phone Cutout). There were pieces of yellow cellophane cut to mimic the Android focus indicators that moved as the user selected items by "pressing" buttons on the Android phone cutout.

The following is a diagram showing the flow from screen to screen of our project. (Stacks of images signify that they are the same screen but differing only slightly due to scrolling or from task progress)

New Character (B)

These two screens ask the user to create their account (which consists of requesting for account information as well as creating their "character" which are divided amongst B1 and B2 respectively).

As can be seen in B2, we implemented spinner selection by overlaying paper strips with text, as well as updating the avatar with their selections. (The avatar they create then persists throughout the application, and can be seen in A and D1-D3 unless they choose to edit it)

Home Screen (A)

From the home screen, the user is presented with their avatar and a simple directional menu to access the four basic categories of screens available to them. (Pressing the direction on the d-pad takes the user to that screen). As can be seen, the character avatar that was created in B2 is seen in A to provide continuity.

Quests Screens (C)

There are quite a few "top-level" quest screens (C1, C2, C3), which provide different lists of quests (C1 is quests that the user is actively engaged in, C2 is quests that the user has completed, and C3 provides a search interface while C3A lists quests that match those search criteria). Users could switch from C1, C2 and C3 by pressing the left and right d-pad keys, and select different quests by pressing d-pad up and down.

C1 demonstrates how the user can "scroll" through a list of quests (by pressing d-pad up/down, which also shifted focus from item to item), and C2 and C3A operate identically in terms of presentation and button functions. D-pad center selects a quest and takes the user to the respective quest view (C1A, C2A and C3B for C1, C2 and C3 respectively).

The currently selected quest displays some additional contextual information (task requirements for a quest for Active and Find New, and the given reward for Completed)

Individual Quest Screens (C1A, C2A, C3B)

The individual quest screens display an individual quest, with a small label specifying the status of the quest (Ongoing, Complete and New for quests in Active, Completed and Find New, respectively), as well as all task requirements and their reward. There is a different menu for each quest depending on status, ongoing quests provide options to either view a map of task locations or to abandon a quest (which removes it from the active quest list), while new quests provide options to either view a map of task locations or accept the quest (which adds it to the active quest list).

C3B is an example of an individual quest screen with the menu displayed. The user can then use the d-pad to select the menu option and then follow that action. Pressing the back button will return the user to the quest list that they were viewing previously.

Quest Search Screen (C3)

The quest search screen is a simple text input field with instructions, and a submit button. Users would type in their query on the keyboard and then move the selection to the button and perform a search.

"You" Screens (D)

The "You" screens provide a way for the user to view their avatar and room as well as the ability to change the appearance of their avatar (by selecting new clothing) and rearranging their room (by selecting what earned trophies to display). This serves as the publically visible representation of their in-game achievements.

D1 is the starting view upon entering the "You" section, and character-specific customizations can be accessed by pressing the menu button (and then selecting either "Edit Avatar" or "Edit Room")

Edit Avatar Screen (D2)

Much like the character creation (B2), avatar editing allows the user to select from a collection of different clothing options using spinners, with changes immediately saved to the server.

Edit Room Screens (D3)

Redecorating a room is a two-step process, first the user navigates to an existing trophy via the d-pad (D3A), and then selects it. Upon selection, the left half of the screen displays a list of all trophies that are currently not on display (D3B), allowing the user to select the trophy that they wish.

Recent Events Screen (E)

This screen is a view of events that have recently occured, and provides a way to view recent events (as "quest completed" notifications disappear from the screen relatively quickly and can be missed if the user is not running the application or is not looking at the screen.)

Events can consist of more than just quest completion, but also other events such as an avatar levelling up. The user can scroll through the list of events by pressing the d-pad up/down keys (though this is unavailable since there are only two events displayed.)

Friends Screens (F)

The friend menu item takes the user to F1, which is a simple listing of all friends that are also participating in the game (the list of friends is built from a user's contact list so as to prevent the game from becoming "yet another social network"). Users can then select the name of a friend (using d-pad up/down and center) which will take them to a read-only version of the D1 display, F2.

F2 is simple, only allowing the user to view their friend's statistics and currently worn clothing and currently displayed trophies. The title specifies that they are viewing a friend, as well as the friend's name so as to minimize confusion.

Method

Participants

As our product is intended to target gamers, all the participants in the experiment were gamers as well as individuals who are avid cell phone users. We selected these participants by finding individuals who were using their cellphones, approaching them, and asking them if they considered themselves gamers. To get an accurate representation of the market we attempted to diversify the participants as much as possible. As such, our participants included a female and individuals from three distinct majors (mass communication, electrical engineering and computer science, and molecular cell biology).

Environment

All of our tests took place in the unit lounges, as these provided a quiet, non-threatening atmosphere. The prototype elements were placed on a table with the computer sitting opposite of the participant to allow for quick and unobtrusive updates. The facilitator sat to the left of the participant so that they could guide him/her through the tasks and offer help when he/she had difficulties completing the task via the interface. Observers positioned themselves as close as possible with the space remaining so that they had a good view of how the user interacted with the interface and where they had struggles, but took care to avoid crowding the participant in an effort to avoid performance anxiety.

Tasks

The group first planned out a "perfect run through" of the tasks, deciding on the path that we hoped the user would take through (which is detailed below). We then looked for deviations from the task, or instances when the user had difficulty deciding on their next step. The expected "perfect run" is listed below.

1. Find a quest in "Cupertino" and "completing" it (medium)
   • After briefly demoing the application and Android (by explaining buttons) the facilitator walked the user through creating a character (screens B1 and B2). Following character creation, the user was presented with the home screen which displays a diamond made up of four menu options with each menu position accessible by its respective equivalent on the D-pad (Screen A).
   • From the home screen the user was expected to select the correct button (pressing up on the D-pad) to go to the quests menu.
   • The quest menu presented the user with three tabs: active (C1), completed (C2), and find new (C3). When the quest screen is first brought up the active quests tab is selected.
   • We expected the user to correctly realize that since the active tab is highlighted, they can utilize their D-pad left and right buttons to navigate to adjacent tabs. The tab we wanted them to end up on was the one labeled "Find New."
   • Upon reaching the find new tab the user was presented with an interface with search box and search button (C3). Some text on the page mentioned that the user could try searching for something that interests them or they could enter their zip code for nearby quests.
   • At this point, the facilitator asked the user to search for quests in Cupertino.
   • We expected the user to use the D-pad to navigate down to the search box, enter in the zip code (given by the facilitator), press down on the D-pad one more time, then select the search button via the central D-pad button.
   • They were then presented with a list of quests in Cupertino (C3A) and expected to select one (via the central D-pad button) to expand the task and view its details (C3B).
   • From there, they were expected to press the central D-pad button again to switch to a screen giving more details on the quest, such as what the reward was and what the subtasks were.
   • On this screen, we wanted the user to press the menu button to bring up a menu with two options: accept quest and view map.
   • The user was then supposed to highlight the accept quest option via the D-pad left/right buttons and select using the central D-pad button.
   • The user was supposed to "complete" the quest by fulfilling each subtask in the quest.
   • The quest that was chosen involved two subtasks: going to a popular Cupertino destination known as Cupertino Square and going there with no one else.
   • Since Cupertino is not within easy walking distance, we expected the user to verbally say that they would go to the location mentioned in the subtask.
   • The computer would then pop up a notification letting them know that the task was now complete.
2. Check recent activity (easy) (note: this was called "view stats" in the contextual inquiry assignment)
   • We expected the user to press the back button to return to the home screen (A).
   • From the home screen the user was supposed to press the D-pad down button to access the recent events screen (E).
   • The recent events screen would show them a list of recent activity on their account. The resulting screen would show the user as having created a character and having completed the quest they had chosen.
3. Edit room/avatar (hard)
   • The user was supposed to press the back button to return to the home screen where they would use the D-pad right button to go to the "You" screen. (D1)
   • On the "you" screen the user was presented with their avatar on the left and four blank boxes on the right.
   • We expected the user to push the menu button which would bring up a menu presenting them with two options: edit avatar and edit room.
   • The user was told to customize their room (D3) and avatar (D2) however they wanted.
   • If the user chose to customize their room, they were expected to highlight the correct menu option via the D-pad left/right buttons and select it via the central D-pad button.
   • This would then hide the avatar, replacing it with a bunch of items the user had collected which could be used to decorate their room.
   • The room decoration process was expected to proceed as follows:
     • One of the blank boxes on the right side of the screen was highlighted when the user first selected edit room. The user was supposed to navigate via the D-pad buttons to the desired box and then select it via the central D-pad button. At this point the left side of the room was replaced with a brief help message. (D3A)
     • Then the highlight would move to an item in the users list of collected items on the left side of the screen (D3B). Here the user was supposed to once again navigate via the D-pad buttons to the desired box and then select it via the central D-pad button.
     • After selection the item would then appear in the blank box that was originally selected and the user could repeat the process to fill all the boxes or modify boxes that were filled already (at this point the screen would again resemble screen D3A).
   • If instead the user chose to customize their avatar, they were presented with a screen with three spinner elements for articles of clothing as well as a live image of their avatar (D2).
   • The user could choose different headgear, shirt, and pants via the spinner. They were expected to do this by navigating to the desired spinner element via the D-pad up/down buttons and changing the value of the selected spinner via the D-pad left/right buttons.

Procedure

1. The greeter welcomed the participant and explained to them what would be going on during the usability study. This involved introducing the participant to our product which is the creation of a game on a cellphone where the user plays by doing things in real-life and gaining virtual rewards. At this time the participant was presented with a consent form and was allowed to read through it to make sure they agreed with all the terms laid forth.
2. After signing the form the facilitator took over the session and explained to the participant that he/she would be asked to go through a series of tasks to explore the interface. The facilitator also took time to mention that the participant should feel free to also explore around the interface as he/she felt the need and that any struggles he/she faced while navigating through the interface should be mentioned as to help us improve our interface.
3. The facilitator then explained some basics of the Android system. To allow the participant to become familiar with the basic button interaction provided by the Android interface, the facilitator walked the participant through the character creation process. This had the added purpose of giving the participant a demo of the interface to our product.
4. The facilitator then proceeded to ask the participant to complete the series of tasks given above. When the participant had questions or struggles with the interface the, the facilitator would offer him/her help to the degree that the facilitator felt appropriate. A transparent yellow sheet was used to signify highlighted elements in our interface and this was appropriately updated by the computer. Also multiple reward and clothing elements were made which the participant was able to access during the customization of their avatar and room. These things were done to allow the user to engage better with the interface than if we had just provided the basic elements.
5. After completing all tasks, there was a debriefing session where the participant was asked to voice any concerns with the interface and system that didn't explicitly come up while running through the tasks.

Team roles:

• Alex: Computer
• Eric: Observer
• Hsiu-Fan: Facilitator
• Glen: Greeter/Observer

Test Measures

Process Data

• Ease of navigation to and from the home screen (we were curious as to how the directional menu would fare in practice, and paid attention to user reaction when first confronted with the home screen)
• Ease of use when switching between tabs in the quests screen (while this paradigm is used in the Android contacts app, we opted not to demonstrate that to users due to time limitations, so this served as a good indicator for familiarity with Android UI conventions)
• Clarity/Intuitiveness of the quest search and selection screens
  • How well the user could grasp the concept of a quest and how to complete it
• Use of hardware menu button for bringing up menus
• Clarity/Intuitiveness of the avatar room and customization interface
  • Whether or not the user could tell what the options were for customization
  • How easy it was for the user to customize their room and their avatar the way they wanted

Bottom Line Data

• Time - the amount of time it took the user to accomplish a task or a subtask
  • Unfortunately because our participants were encouraged to be vocal about their concerns and confusions with the interface, there were gaps between the user interacting with the interface and discussion with the facilitator. The times in the appendix are our estimates for the time spent on each task using our interface.
• Number of errors - the amount of tries it took the participant to complete a task or a subtask
• Number of button presses - the number of button presses it took a user to complete a task or a subtask from the end point of the previous task (or home menu for the first task)
• Focus of attention - participants would often look to the facilitator for help when struggling or unsure what to do next.

Results

In our experiment, we tried to gauge the users’ responses to the more important aspects of our game: finding/accepting quests and editing one’s room. We also were curious about how well they would navigate between screens, both from the home screen and from each sub type (quests, you, etc.)

General observations

All of our users figured out that the four buttons in the diamond on the home screen corresponded to the buttons in the center of the phone. However, participants 2 and 3 suggested that if they were arranged in a square, the connection would be stronger. Two of our participants (1 and 2) had trouble grasping the mechanics of the Android cell phone interface, mainly because it was different from the phone they normally use. Participant 1 tried to bring up the menu by pressing the center button instead of the menu button and participant 2 was initially unaware about how to bring up a menu. Participants 2 and 3 were confused about what the home button was used for and thought it would bring them back to the home screen of the game instead of the phone. Despite that, participant 3 said that she thought it was easy to learn how to use the phone. Participant 1 suggested labeling the buttons on-screen similar to how his current phone was laid out.

Task results

Task 1: Finding a quest and "completing" it (medium)

This is were our participants had the greatest number of errors and spent the most time on. Participant 1 did not have any problems navigating to the find new tab from the active quests tab, but the other two participants had trouble. They could not tell which tab was currently selected (which is perhaps a limitation of the low fidelity of our prototype). Once each participant got to the "find new" tab, they had problems figuring out how to accept the quest they wanted. They were confused by the fact that one quest was expanded and the other ones were not and did not know why some of the quest's details were on both the initial find screen and on the subsequent accept screen. After they managed to select the quest two of the participants could not figure out how to accept it, as there was no "accept" button on the screen. Only participant 3 correctly used the menu to accept the quest. All of the participants did not know what the check boxes underneath the quest title were for and how they corresponded to the quest title and quest completion. It was also not apparent to our participants how the quest title was related to the quest itself. Participant 2 remarked that he wasn't sure was a quest was, as it was not obvious. From the length of time spent doing this task, participant 3 took the most time. However, participant 3 used only 8 button presses. Participant 3 was mainly confused (the errors) about the interface and slowly thought about which button press would get to the next screen.

Task 2: Viewing recent events (easy)

None of the participants had particular trouble doing the task itself, as it was meant to be simple and easy. However, participant 3 was not sure how to get to the home screen from the previous task, pressing the home button on the phone and exiting the application, which explains why participant 3 took the longest time and most number of button presses to complete this task. Since there was no direct link back to the home screen, our other participants pressed the back button to get back to the home screen and then to the recent events page. Overall, this task was completed with few or no problems.

Task 3: Editing the room/avatar (hard)

Initially, all of our participants were confused about what their room was for until the facilitator told each of them. Additionally, none of them got the distinction between trophies and posters, as it was difficult to tell from our prototype. Each of them generally had little trouble editing their avatar and found the spinners intuitive. Participant 2 mentioned that while he thought the process was fun, he would have liked a confirmation button or screen to announce that the changes were saved. Once participants 1 and 2 got to the room screen, they were not sure how to select items to be placed or place the items themselves. They were also initially unsure of what items they had to select from, as they were initially presented with a blank room. Only participant 3 figured this screen out quickly, which can be verified by the length of time that she spent performing this task. She agreed with the design limiting the number of items you could show in a room, as this allowed changes based on your current mood. Despite spending the least amount of time performing this task, participant 3 had the highest number of button presses because she enjoyed the modifications and ended up modifying her room (and avatar) more so than the other users.

Game Comments

Additionally, some of our participants had some comments about the game itself. Participant 2 thought that the idea of a real world RPG was cool, but participant 3 was not sure whether it would actually motivate her to do things outside. Both of them agreed that obtaining items would be an important motivating factor, as it would give them a tangible reward. In particular, participant 3 preferred this over a currency system that would require users to buy their own items. Participant 2 did not initially understand the purpose of items, but once it was explained to him, he considered it an important part.

Discussion

From our results, we learned that we have many elements that are confusing to the user or hard to use. Not all users shared the same comments and/or problems. As such, based on the needs, comments, and concerns of all participants in the study, we will need to change a number of aspects of our interface design, as detailed below.

• Navigation to/from the home screen: One thing two of our users suggested changing was the diamond on the home screen. Instead of using a diamond on the home screen, we should reflect the physical layout of the D-pad arrows and instead use a square so there is a more direct mapping between the two (however, this may be dependent on the actual design of the physical phones). In addition, users had frequent difficulty with navigating back to the home screen via the back button, this is an issue that is probably best evaluated with subjects who are given more time to acclimate to the Android UI (particularly with emulator time), as the metaphor is fairly pervasive.
• What quests are: Another thing that needs to be more clearly defined is what a quest is and how to complete one. The quest titles themselves were unclear and some felt like they were too "far-away" from the actual quest itself. A description such as: "A quest is a task or a list of tasks to complete in order to gain a certain reward. They usually involve going to one or more specific locations." will be displayed in the C1 and C2 screens when they are in their empty state. An example quest from our prototype, "Soda Hall Spartan", did not make it clear that the objective was to visit Soda Hall and stay there for an extended period of time. More careful copy-editing may be necessary to maintain the friendly tone of quests while keeping them accessible. Users also found that the check boxes next to each task (which listed the requirements for a quest) in a quest confusing and unhelpful. As suggested by participant 2, we've decided to remedy this by more clearly separating each subtasks into two sections: completed and incomplete, so that the user has a better notion of what they still have to do.
• Selecting and accepting a quest: Scrolling between quests in our prototype was also confusing. Users found the list of quests and expanding quests confusing, mainly because when scrolling between different quests, a few major details would be expanded and then hidden when they selected another quest. Further compounding their confusion was that these same details would be repeated when they pressed the center button to move to the view quest screen. A simple non-expanding list would fix this problem, perhaps with the expanded data always visible (so that users are not forced to glance back and forth between the list and individual views). Users also did not realize that they had to press enter or the center D-pad button in order to view more quest details followed by navigating a menu to accept the selected quest. (One interesting note is that while users all struggled with knowing to press the menu in the first task, they had no such difficulty in the last task). This can be mitigated by adding menu options for the currently selected quest to the quest lists, as well as indicating to the user the availability of additional data for that quest. We also plan to add selectable accept or decline buttons to the individual quest view so that users are not forced to open a menu to accomplish a relatively common task.
• Switching between tabs on the Quest screen: The way that users switch between the "Completed," "Active," and "Find Quests" tabs was a continual source of participant errors. Participants 2 and 3 tried to move between tabs using the menu button instead of pressing the left and right buttons on the phone. This is particularly troubling as this UI metaphor is used in other places within the standard Android applications. We plan to add small left/right arrows to the tab display (as the arrow icons in the directional menu were specifically mentioned as being helpful), and test again with users who have more familiarity with other Android applications. Another option was to add navigation to the menu, but due to Android restrictions on the number of items that can be initially displayed in the menu, this may squeeze out other options (such as accept/decline of quests), and will require split testing with the iconic tab approach.
• What you can do in your room: Our design was not especially clear as to what the user could do in a room. It was hard to view what items were available and users were confused as to the difference between trophies and posters (subsequently, there will not be a difference between the two in the future). Another problem was saving the changes to the room/avatar that a user has made. Two users were comfortable with simply changing their room/avatar and going back to the room change home screen. but one (participant 2) commented that a "save changes" button would be nice to ensure that the changes actually made it through. We will provide some on-screen notification (via the Toast notification system) that the changes were saved, so the user is sure that an error did not occur in the process. We are also considering adding a button to save changes and exit the screen or make a menu tab with that option.
• List sort orders: There were a number of list displays in the interface where problems and questions participants voiced would have been mitigated by choosing different sort orders for list items. The recent event screen will be changed to display recent events in chronological order after comments from users that it was reverse to their expectations. Also, the order of the trophies that are displayed when editing a room will be in the reverse order they were acquired, as users expressed confusion as to where their most recently acquired trophies went.

The experiment could not reveal a few things.

• It could not show us how users would react if they had a real smart phone or had previous exposure to the Android platform. We observed that some things we thought would be obvious with a real phone were found to be confusing during the experiments. In particular, the tabbing system for navigating between quests was generally found to be confusing, but our tabbing system was taken from the contacts application that comes standard with the phone, and every user struggled with the difference between back and home (and appropriate times to use those buttons) despite their pervasive use throughout the system.
• Actually going outside and going to a goal location to complete a task or quest could not be tested in the experiment for practical reasons. Therefore, the ease of use in an outside environment could not be tested. It was also impossible to guage the effect that the difficulty of tasks would play and how it should tie into the reward system.
• We could also not measure how effective it would be as a motivating tool over time since the experimental subjects were only able to use the game during one sitting. Participant 3 said she wasn't sure if the game would be a strong motivating factor, while actually excited at having a numerical score for her social history.
• It is difficult to guage the discoverability of features as participants were given tasks. In particular certain activities such as editing rooms and avatars were usually quickly accessed, but participants were told about the feature's existance. In particular, as mentioned above, users struggled with pressing the menu the first time they were required to do so (when accepting a quest), but did not hesitate to do so when in the You section. The discoverability of features hidden in menus is particularly troublesome (as revealed by the errors in the first menu usage), while the differential in performance between the two menu usages suggest that a certain degree of training has occured in participants which may require a separate test format to understand more.

Appendix

Files

• Consent form
• Demo script
• Phone Cutout

Raw Process Data

General comments

Participant 1

• Thought the diamond at the home screen was intuitive
• The menu button was confusing since most regular phones don't have it
  • Would be helpful if the menu button was labeled on the screen
  • Instead, this person expected the center button to pop up the menu
• After filling out forms, the "done" button should be selected, not the "back" button

Participant 2

• A little unsure about home screen diamond mapping to the cell phone buttons since it was a diamond and the phone buttons are arranged in a square
• Expected browser-like behavior (underlined words = links)
• Didn't know what the "home" button would be used for in Android
• Generally thought the idea of a real-world RPG was cool

Participant 3

• Figured out how to use diamond mapping on home screen, but mentioned would have preferred that buttons were laid out mimicking the phone hardware buttons
  • Said arrows helped a lot in figuring out how things worked
• Generally easy to learn, didn't need to learn too much about how to use the cell phone
• Highlighting makes things easier to see (focus highlighting via cellophane/Android helps maintain task focus)
• Not sure if it would be a strong motivating factor to go out and do things
  • Mentioned that she enjoyed the ability to view friends, and that combined with numeric scores provided a motivation to be more social than her friends.

Task 1: Finding a quest and "completing it" (medium)

Participant 1

• Thought the left/right navigation between tabs on the active/completed/find new quest screen was easy to figure out
• Not really easy to select the quest you want

Participant 2

• Liked quest aspect and variety
• Not really sure what quests are and what you should be searching on
• Not sure how to tell which tab is currently being used when switching between active/completed/find new
• Tried to use the menu button to switch between tabs
• Confused that pressing the center button does not submit a search on the "Find new" screen
• Quest titles are unclear and it's unclear what the sub-objectives are
• Not sure how to accept a quest since there is no accept quest button
  • Also not sure what the check boxes are for
  • Instead of check boxes, maybe have % completed or last completed tasks
• Expansion of details is confusing since the details are also on a separate screen anyway (when choosing to accept or not) (4)
• List format of active quests confusing (wasn't sure if it was scrollable)

Participant 3

• Thought the sub-objectives were confusing, didn't know what the check boxes were for
• Had trouble figuring out how to switch between the tabs
  • Pressed back first, then menu
• Found list of quests and expanding quests confusing
• Figured out how to use the menu to accept a quest
• Preferred receiving an item per quest instead of receiving money to buy things
• Suggested having a menu with the same options as the tabs so that people can use either

Task 2: Viewing recent events (easy)

Participant 1

• Expected to see the "name" from the initial character creation

Participant 2

• Confused about what the page's purpose was for. Recent events = quests?

Participant 3

• Had trouble navigating back to this page from the quests screen
  • Had to guess how the back button would go back
  • Tried to use the home button of the phone to go to the homescreen

Task 3: Arranging the room and editing the avatar (hard)

Participant 1

• Generally confused about what you could do in a person's room
• Paused to figure out moving and selecting items

Participant 2

• Figured out how to use the spinners in the "edit avatar" screen quickly (similar methods are used for customization in current games)
• Liked the aspect of avatar customization, thought it was fun
• Would have liked a confirmation button or screen for avatar changes (otherwise not sure what would happened if you navigated away)
• Confused about what happened to item obtained after completing a quest
• Purpose of room not initially clear
• Not clear what the distinction is between trophies and posters
• Not clear what items are used for

Participant 3

• Figured out how to bring up edit menu
• Mentioned that a limited selection was okay, so you could change the room based on your current mood
• Mentioned that she preferred rewards that are quest-specific, and felt that this improved her valuation of room customization.

Summary of Issue Average Severity

During user testing, we noted negative events and later individually assigned them severity. The average was rounded and is collected below in order of severity. (Similar events were combined)

• Severity 5
  • Hard to select wanted quest
  • Difficulty switching from tab to tab
  • Confusion about purpose behind quests
  • Confusion about purpose behind character personalization
• Severity 4
  • Work/reward ratio may be too high
  • Don't know where trophies go upon quest completion
• Severity 3
  • Expected link behavior from underlined words
  • Center button behavior is largely similar to enter on PC... but not always
  • Quest task requirement checkbox meaning unclear
  • Didn't realize quest lists were scrollable
  • No feedback of successful saving of character customization
• Severity 2
  • Own character name is never seen
  • Unclear of difference between events and quests
  • Unclear of difference between trophies and posters
  • Done button should have precedence over back in selection order
  • Menu button confusion
  • Home button confusion
• Severity 1
  • Unclear which tab is currently selected

In general, we decided that we needed to more clearly present the application as a game where users move from place to place and can accrue items to use in decorating their virtual room. While this can be seen from the features available to users, this is never "sold" to the user, and users responded much more positively towards the application once the core concept had been explained (where before there were many questions about what the nature of a quest was, or what items were for). This also led to comments about work/reward ratios, and such. Beyond that there are a number of smaller issues that spawned the numerous changes mentioned above, as well as a number of UI complaints that the team wishes to revisit in tests with higher fidelity mockups and better educated users to see if they continue to be issues.

Bottom Line Data

Time

• Each participant took roughly 20 minutes to complete the three tasks
  • Participant 1 took about 10 minutes to do task 1, 1 minute to do task 2, and 8 minutes to do task 3.
  • Participant 2 took about 11 minutes to do task 1, 1 minute to do task 2, and 7 minutes to do task 3.
  • Participant 3 took about 13 minutes to do task 1, 2 minutes to do task 2, and 5 minutes to do task 3.

Number of errors

• Participant 1: 2 major errors during the finding a quest task and room customization
  • Finding a quest (1 error): error on how to actually add a quest from the quest details page
  • Room customization (1 error): error in how to bring up customization menu
• Participant 2: 2 major errors also during the finding a quest task and room customization
  • Finding a quest (3 errors): error with the tab interface usage, confusion over quest objective check boxes, and how to actually add a quest from the quest details page
  • Room customization (1 error): error in how to bring up customization menu
• Participant 3: 1 major error during the finding a quest task
  • Finding a quest (3 errors): error with the tab interface usage, how to switch between tabs, and confusion over the quest objectives checkboxes

Number of button presses

• Participant 1:
  • Finding a quest and completing it: 8 (not including search field entry)
  • Viewing recent events: 2
  • Customizing room: 16
• Participant 2:
  • Finding a quest and completing it: 10 (not including search field entry)
  • Viewing recent events: 2
  • Customizing room: 12
• Participant 3:
  • Finding a quest and completing it: 8 (not including search field entry)
  • Viewing recent events: 3
  • Customizing room: 19

---