PilotStudy-Group:O

From CS160 User Interfaces Sp09

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participants 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 4.1 Task 1 (Easy) 4.2 Task 2 (Medium) 4.3 Task 3 (Hard) 5 Results 5.1 Participant #1 5.2 Participant #2 5.3 Participant #3 6 Discussion 7 Appendices 7.1 Materials 7.2 Raw Data

Introduction

The system being evaluated is the interactive prototype for our groups's game project: Glyph Fighter. The game is designed to help students of East Asian languages (such as Chinese and Japanese) learn to write the language. We chose Chinese in our current implementation for convenience, but the game can be extended to in fact any logographic language. The player takes on the role of a sword fighter, who must face a stream of virtual enemies. When the game starts, the player is presented with the English translation of the character she is expected to reproduce. For example, for the character in Fig. 1, the player will be prompted with the hint "big". She must then input the character (with the mouse) in the correct sequence of strokes, and the correct direction for each stroke, as shown in Fig. 2, 3, and 4, in order to score points and defeat the enemy.

Fig. 1

Fig. 2

Fig. 3

Fig. 4

The pilot usability study aims to smooth out design flaws and bugs in this still rudimentary version of our design. Through observation of test users' interaction with the game as well as users' feedback, we will be able to establish the overall structure and primary elements of the game. For instance, we will be able to tell if our control mechanics are intuitive and feasible, and whether or not our tutorial serves its purpose. The study can also provide a less biased gauge on issues such as game difficulty (in terms of gameplay), ease of navigation within the interface, and aesthetics.

Implementation and Improvements

• Improved the animation/stroke matching. (Strokes will now generate a better matching animation)
• Changed the framerate to make the animations slower.

Method

Participants

Participants' names left out in order to preserve anonymity.

• Participant #1 - Female Chinese American student. Currently taking Chinese 1BX. This person knows how to speak Chinese and only knows how to read and write simple words.
• Participant #2 - Male American student. Currently taking Chinese 1B. This person only knows what has been taught to him in Chinese 1A. He has limited speaking and reading/writing ability.
• Participant #3 - Male Chinese American student. He is not enrolled in a Chinese class but is trying to learn the language on his own. He knows how to speak Chinese and has limited reading/writing ability.

Apparatus

The objects used in this study were a laptop with our game running on it, a USB mouse, and USB tablet. Studies were performed in a study room at the Doe Library. The environment was a very quiet room with each user coming individually. Users were situated in front of the laptop and asked to complete the three tasks laid out for them. The user was first presented with a brief demo of what the general idea of the game was and how the interface worked. The demo was extremely brief and gave no specific or detailed instructions of how to use/play the game. The user was told to think aloud as they performed the three tasks and was given minimal help on how to use the interface.

Tasks

• Task 1 (Easy) - Learn how to play the game. This task is considered easy because the user must simply click on the "Tutorial" button from the main screen and sit through the video tutorial.
• Task 2 (Medium) - Start a game using the custom difficulty settings. This task is harder than the first because it requires the user to navigate through to the right screen and also requires the user to understand what the rules of the game are.
• Task 3 (Hard) - Start and play through a game. The hard task is actually playing through and winning the custom game. Players must actually play through the game prototype and win. This can be of varying difficulty depending on the user's custom settings.

Procedure

The user was first given a short demo of the system. The script was extremely short and is as follows:

• "I'm going to give you a quick run through of the game"
• Click Play
• Click Medium
• "So I'm just going to go ahead and play a medium game and hit the enemy a few times"
• Perform three strokes
• "Okay that's all I'm going to show you, now here is a list of the tasks you are to perform.."

Task 1

The user was first told to learn how to play the game. Users were expected to press the tutorial button and watch the tutorial video on how to play the game. From the tutorial screen, the user could either watch the tutorial or exit the tutorial if they became impatient. When the user was finished with this task, he was asked to explain the general rules and was also asked this series of questions:

• How do you damage an enemy?
• Where is your health bar and the enemy's health bar located?
• When are you attacked?
• How do you win the game?

The user was gauged on how well he answered the questions and how well he understood the game.

Task 2

The user was told to start a game using the custom options. Users were expected to navigate to the play screen and choose "Custom Game" instead of the Easy, Medium, or Hard options. The user could choose from the following options:

• Character Difficulty - Determines which character library the user is given. If hard is chosen, transparent outlines of the characters do not appear.
• Time Limit - Allows the user to change the number of seconds he is given per stroke. Allows for 3-10 seconds per stroke.
• Player Health - Determines player starting health. Ranges from 50-100.
• Enemy Health - Determines enemy starting health. Ranges from 50-100.

Time spent on each option was measured to determine how clear these options were. The user was asked to think aloud what he thought the options were and also why he chose his options.

Task 3

The user was told to play through the game. The user either lost the game or won the game and was measured on how well they performed the task. Users used a USB mouse to input strokes the first time through and then used a USB tablet the second time through and were asked to compare between the two. Users were asked to think aloud the different sensations as they inputted strokes, as well as anything else that came to their minds during gameplay. At the end of the task, the users were asked to compare the two methods of inputs and which one they preferred, and were also asked to give general feedback of the game screen.

Test Measures

Task 1 (Easy)

• Total time to complete the task
• Time spent before clicking the tutorial button
• Full tutorial watched
• Test the user on what the rules, procedures, and victory conditions are

Reasons: Total time to complete the task includes how long it takes for the user to find the tutorial button in order to see if the text is too small or if the font is unclear. The time spent watching the tutorial is also measured because some users may get bored and press the "back" button thinking they have learned all of the rules. At the end of the task, the user was asked to explain the game and the games' rules.

Task 2 (Medium)

• Total time to complete the task
• Time spent finding the custom game screen
• Total time spent on the custom game screen
• Time spent editing each component on the custom game screen

Reasons: Total time spent to complete the task was an accumulated value of the time spent on the custom game screen, and time spent finding the custom game screen. Time spent finding the custom game screen was used to measure whether or not it was confusing on how to get to the screen, or if our font/styles were too small or unclear. Total time spent on the custom game screen was the accumulation of time spent editing each component on the custom game screen. This measured how clear the options were and included the users playing around with the options in order to get a sense of how the games' rules worked.

Task 3 (Hard)

This task was done with a mouse and a tablet

• Total time to complete the task
• Number of mistakes per character
• Number of correct strokes per character
• Number of times the timer reached 0
• Player's health between characters and at the end of the game
• Enemy's health at the end of the game (if the player lost)

Reasons: Total time to complete the task (winning or losing) measured how well the user was able to complete each character and win the game (or how fast they were able to lose). Number of mistakes per character allowed us to get a sense of how picky our stroke comparison algorithm was and also allowed us to measure the user's frustration levels if they made multiple mistakes. The number of correct strokes per character had the same purpose. Number of times the timer reached 0 allowed us to see if the time given per stroke was reasonable. The player's health at the end of the game was measured (the enemy's if the player did not win) in order to gauge how hard the game was. The health was also measured between characters in order to gauge the learning curve to see if players found the game easier toward the end.

For the last task, users used both a USB mouse and a USB tablet for stroke input.

Results

Participant #1

Task 1

• Total Time - 50 seconds
• Time before going to tutorial - 5 seconds
• Full tutorial watched? - Yes

This user first asked the question "How do you know which stroke to do first?" The user also did not notice that there was a timer in the top right of the screen and stated that you are attacked only when you mess up a stroke. Other than that, the user understood where the health bars were located and also when he could attack the enemy. User expressed that the tutorial went by too fast and should have been slowed down. She also felt that the tutorial's content was adequate because she felt she was able to explain the game well enough.

Task 2

• Total Time - 24 seconds
• Time to find custom game - 4 seconds
• Time in custom game menu - 20 seconds
• Approximate time per component - 4-6 seconds

User was asked to think aloud what she thought the different options meant. User thought that the characters option allowed her to change the difficulty of the enemy. She thought the term "characters" was a bit ambiguous and would have preferred some kind of description. She stated that the options "Enemy Health", "Player Health", and "Time" were all fairly straightforward and was able to describe each of these options correctly.

Task 3

With Mouse

• Total Time - 35 seconds
• Mistakes - 1
• Correct - 8
• Timer Reached 0 - 0
• Player Health - Hit once
• Enemy Health - 0

With Tablet

• Total Time - 25 seconds
• Mistakes - 0
• Correct - 8
• Timer Reached 0 - 0
• Player Health - Full
• Enemy Health - 0

This user was able to win the game on her first try. She felt that using the tablet was much easier than using the mouse. She complained that the feedback was unclear and she was not sure when she hit the enemy or when the enemy hit her. User felt there should be more noticeable feedback whenever the user made a correct stroke or missed a stroke. She also felt that winning was unclear, graying out the screen and having text with "Winner!" in the corner was not big enough and she expressed that she would want some large letters flashing "You WIN!"

Participant #2

Task 1

• Total Time - 1 minute 20 seconds
• Time before going to tutorial - 60 seconds
• Full tutorial watched? - Yes

This user did not press the tutorial button first and instead began playing a game first in order to learn. Eventually he navigated to the tutorial screen and watched the tutorial. User expressed that he wanted some controls for the tutorial to play, rewind, stop, or go forward. User also expressed that the tutorial went by a little too fast and he could not take in all of the rules. He also noted the bug that you cannot play the tutorial more than once.

Task 2

• Total Time - 40 seconds
• Time to find custom game - 5 seconds
• Time in custom game menu - 35 seconds
• Approximate time per component - 8-10 seconds

User felt that the term "characters" was ambiguous and at first said that it referred to the difficulty of the enemy. After saying this, he corrected himself stating that it probably meant how complex the Chinese characters were. User preferred more explanation of this feature and wanted mouseover tooltips. The user stated that the other three options were straightforward and had no problem explaining what altering these options did. The user also wanted a wider range of variables to choose from within the options (for example expanding number values on the timer to have a greater range).

Task 3

With Mouse

• Total Time - 30 seconds
• Mistakes - 7
• Correct - 4
• Timer Reached 0 - 0
• Player Health - 0
• Enemy Health - Killed 1 enemy

With Tablet

• Total Time - 50 seconds
• Mistakes - 7
• Correct - 3
• Timer Reached 0 - 0
• Player Health - 0
• Enemy Health - Killed 1 enemy

This user felt that using the tablet was harder than using the mouse. User was unable to win the game. The user noted that he wanted strokes to stay on screen when he made them if they were correct, he felt strokes should only disappear if they were incorrect. User also wanted more feedback when strokes were right or wrong. He also expressed disapproval in the fact that stroke orders were not preserved and wanted this feature to be implemented. User also noted that he thought he finished writing a word but apparently was not and his character subsequently killed because of it. User did not know where he made a mistake so he began writing the character from the beginning, however rewriting strokes results in a wrong stroke which is most likely what killed this user. Finally, the user expressed that he wanted the timer to be more visually significant because he felt it blended in too much with the game screen.

Participant #3

Task 1

• Total Time - 40 seconds
• Time before going to tutorial - 20 seconds
• Full tutorial watched? - Yes

This user also went straight to the game screen when asked to learn how to play the game. When the user died, he decided to go to the tutorial screen. This user did not get what the requirements were for stabbing and felt the tutorial went by a little fast. He also felt the tutorial was vague and at first did not understand what the transparent outlines of the words were for. He felt this element of the game should have been included in the tutorial. When asked what the rules of the game were, the user was able to explain the game fairly well but did not realize the timer running out also lead to the player getting attacked.

Task 2

• Total Time - 24 seconds
• Time to find custom game - 4 seconds
• Time in custom game menu - 20 seconds
• Approximate time per component - 4-6 seconds

This user did not attempt to actually drag the drag bars but instead tried clicking on the values he wanted. He said in normal applications the drag bars had a "snap to" feature in which the bar would recognize where he was clicking and snap to that value. This user was also confused on what the "Characters" option meant and thought it meant it indicated the characters' different skill levels. This user was able to correctly explain what the rest of the options meant.

Task 3

With Mouse

• Total Time - 39 seconds
• Mistakes - 7
• Correct - 2
• Timer Reached 0 - 0
• Player Health - 0
• Enemy Health - Hit an enemy twice

With Tablet

• Total Time - 30 seconds
• Mistakes - 3
• Correct - 8
• Timer Reached 0 - 0
• Player Health - Hit 3 times
• Enemy Health - 0

The user felt the outline of the characters was not entirely clear at first because he did not see the English translation of the Chinese character on screen. He also felt that the stroke comparator was extremely picky but performed much better with the tablet than with the mouse. He felt it needed to be more clear that one was to trace over the characters, as well as more clear when the user attacked or the enemy attacked. He also felt it would be a good idea to have already correctly inputted strokes remain on screen and have the wrongly inputted strokes disappear from the screen.

Discussion

Through the pilot useability study, we have zoomed in on the following areas of possible improvements:

• Better correct/wrong stroke feedback: All three test users commented negatively on the feedback design. Users were unclear when they performed a correct stroke and equally uninformed when they performed an incorrect stroke. A planned improvement is to implement more dramatic on-screen feedback (such as flashing health bars) as well as auditory cues (such as groaning and slashing). Another crucial change that can improve the feedback has to do with the following point: correct strokes staying on screen.
• Correct strokes should stay on screen: Two out of the three test users suggested that strokes that were correctly performed should stay on screen. We agree. Not only will this drastically improve the abovementioned feedback problem, it will also reduce recognition errors by allowing the previous strokes to serve as anchors to position the next stroke. It will also allow users to see which strokes they've already completed when making a mistake. In fact, we tried but failed to implement this feature in the development of the prototype. This will definitely be a key area of work in future revisions.
• Mouseover tooltips for the custom game options: Again, all three test users had problem recognizing the true meaning of "Character Difficulty" in the Custom Difficulty Menu. A mouseover tooltip would help the user understand the meaning of each option without cluttering up the limited space.
• Tutorial controls (play, pause, rewind, forward): According to two out of the three test users, the tutorial went by too quickly. This will be a problems for players, especially when their attention is not fully focused on the game. Missing a part of the tutorial may force them to view the entire thing all over again. Controls that allow them to pause and continue, as well as to skip to the previous or next instruction would be helpful.
• Correct stroke order: two of the users complained about stroke order. Currently, the player can enter strokes in any order. At the very least, correct stroke order should be an option on the custom game screen. In order to indicate which strokes come first, perhaps numbers could be added to the strokes on the game screen.
• Make game screen elements more conspicuous: several users complained about the timer being inconspicuous, and one user said the same thing about the definition. These elements are crucial and should be more visible, for instance by making them bigger or by making them contrast more with the background. One user complained about the victory screen being underwhelming. We should make victory more visible as well, for instance by using a flashing "YOU WIN!" sign, along with sound effects.
• Make stroke recognition better: our current stroke recognition is very picky and caused several users problems. We will continue refining it.
• Use the term "glyphs" instead of "characters": one user confused the term "characters" with the player and enemy, not the written Chinese. We should stick to the word "glyphs".

Appendices

Materials

• Consent Form
• Short Game Demo Script shown to Participants (or download the .avi file here)

Raw Data

• Task measurement comparison charts

---