PilotStudy-Group:WAAM!
From CS160 User Interfaces Sp09
Contents |
Introduction
"Bible vs. Rock" is a computer game that is designed to help users memorize Bible verses in an effective and enjoyable way. In the game, the user is displayed as a figure that is running to the right, across platforms that continually scroll to the left. Each platform displays a word (in consecutive order) from the Bible verse that the user is currently playing the round with. The user's goal is to type in the word contained in the current platform before the platform disappears off the left side of the screen. If the user does not correctly input the word before the platform hits the left side of the screen, the figure falls off the platform and down onto the rocks below, causing him to lose a life. If the user types in the word correctly before the platform hits the left side of the screen, the figure jumps from the current platform and word to the next platform and word. The user must continue to type in each consecutive word correctly until the figure jumps to the final platform, which is the Bible reference of the verse. Upon arriving at this platform, the user has completed the current level. If the user does not successfully reach the end of the verse from the beginning, and loses three lives, then they fail the game, and must restart. If the user completes the verse, he moves on to the next level of the verse, in which a certain percentage of the words are replaced with blanks, promoting the memorization of the verse rather than simply typing in displayed words. If the user completes all the levels and successfully finishes the level in which every word is a blank, he finishes that verse and can move on to a different one.
The goal of our pilot usability study is to discover flaws in our user interface, and to find ways to improve usability. We hope to find tasks that cause user errors, and cause users confusion regarding how to best proceed. We will focus collecting data that will help us improve the game in terms of both intuitiveness and minimizing time required to complete tasks. After completing this this pilot experiment, we hope to have enough new data to find ways to make using "Bible vs. Rock" faster and more intutive.
Implementation and Improvements
Browsing Statistics and Recommendation
- Changed color scheme of Top Scores and Verse Suggestions (many complaints about the colors being too loud).
- Made sure the Green Font of Verse Added notification had a white background for easier view ability. Made color darker and font bigger.
- Changed “Add Verse to Account” to “Add to MyVerses” for buttons.
- Made popups appear on the center of the screen instead of the corner.
- Made View Popup disable the background, so that it has to be closed to continue, to make sure users close the popup, and disables the ability to view multiple verses at once (which no one found useful in the first place).
- Made several fonts bigger and more readable.
Game Play
- Changed the colorful thing on the button to rocks.
- Made it so that it ignores whitespace when you are inputting text.
- The guy doesn't float at the end when you win.
- Variable word speed - made it so that when you go faster than the words scroll, it catches up.
Method
Participant
Since our game is targeted towards Christians who are interested in memorizing more Bible verses, we decided to target the same user group and demographic as our previous usability tests. In addition to choosing participants who regularly attend church, we also selected participants who were familiar with using computers, since these are the types of people who might adopt a computer game such as "Bible vs. Rock" to aid in their memorization.
UserA: Is a 21 year old mathematics senior at the University of California, Berkeley. He has attended church regularly for four years, and also owns a laptop. He was a randomly selected member of Gracepoint Fellowship Church, of Berkeley.
UserB: Is a 22 year old mechanical engineering senior at the University of California, Berkeley. He has attended church regularly since childhood, and is highly proficient in using computers. He was a randomly selected member of Gracepoint Fellowship Church, of Berkeley.
UserC: Is a 20 year old philosophy junior at the University of California, Berkeley. He has attended church regularly since childhood, and regularly uses a laptop computer. He was a randomly selected member of Gracepoint Fellowship Church, of Berkeley.
Apparatus
For the pilot study, we set up our base of operations in Michael's living room with three laptops running our updated interactive prototype for the users to use. One person acted as facilitator and led the study by explaining the interface, while another demonstrated alongside on the prototype, using each of the laptops. The other two group members took notes using paper and pencil/pen; one of them used the stopwatch tool in their cellphone to accurately measure the time taken by our participants to complete each task.
Tasks
- Use the web interface to find the most popular verse and add it to your account. (Easy)
- Some users may not have a verse in mind to memorize, so they can look at the most popular verse tab to see which verse is the most popular. This task is fairly simple (and thus labeled as "easy") as the buttons, sections, and information are self explanatory. When our users carried out this task, we watched to see how long they took to accomplish the task, and how many times they clicked on the wrong item/link.
- View end-game results and see which word you lost on. (Medium)
- After losing or winning in a game, users usually want to see their results. They want to see how far they got, how well they were doing, and exactly what word they messed up on so that they could focus on memorizing that particular word. There is a lot of information in the results screen and new players might not understand the significance of some of them at first, which is why this was rated as a Medium difficulty task. We observed the time taken for the users to figure out how to find the word they lost on.
- Play the game and successfully make it to the end of the level. (Hard)
- This is the actual game itself so naturally it is a Hard task as users need to correctly input a verse word for word, more and more portions out of memory as the level of difficulty increases, and all of this under a time constraint. Mainly, we watched to see how many times users lost while playing.
Procedure
First, we grabbed an empty room and set up everything needed for the user surveys: one laptop setup for the first three tasks (using the web interface), another laptop set up with the fourth task (playing the game), and one more laptop set up for the last task (maneuvering around the end results screen). one person acted as the moderator. After asking our participants to sign the consent form, he proceeded to give them a bit of background on our game and what they needed to do. Another one of our group members demonstrated a few sample tasks using our interface, then we revealed the five representative tasks to our users, and instructed them to accomplish them one after the other. In the meantime, the rest of our group members were observing the users and taking notes on various benchmarks, like total time taken to accomplish a task, the comments that the user made while trying to performing the task, and the number of errors made during the process. At the end, we debriefed the users on how they did, and cleared up any misconceptions they might have had. In return, the users gave us final comments about what they thought about our interface, and gave some opinions on what they thought could have been improved in our design.
These were our roles:
- Alex: Greeter, facilitator
- Aaron: note-taker
- William: note-taker
- Mike: note-taker, time-taker
Test Measures
- Time taken for a task
We wanted to find out the amount of time each of the tasks we had the users complete took. This was intended to help us find slow tasks that could potentially be bottlenecks, and find ways to make those tasks easier and faster to complete.
- Number of errors per task
We also aimed to record both the number of errors made during each task, and what exactly those errors were. We did this in order to find areas and specific places in our user interface that could be improved to help users avoid making mistakes.
- Questions per task
For this measure, we recording both the number of questions users had and what the question was regarding. This was intended to help us determine how intuitive the task was, and how we could improve the user interface to make the sections that were unintuitive easier to user.
Results
Task 1: Add the most popular verse to your account
Task 2: View the profile of the second highest scoring user
Task 3: Randomly generate verses and add one you like to your account
Task 4: Complete two levels of the gameplay successfully
Task 5: Discover which word you died on
UserA
Task 1
Time: 23.53s
Errors: 0
Questions: 1
Task 2
Time: 10.32s
Errors: 0
Questions: 0
Task 3
Time: 10.32s
Errors: 0
Questions: 0
Task 4
Time: 2m 22.91s
Errors: 0
Questions: 0
Number of losses: 3
Task 5
Time: 5.96s
Errors: 0
Questions: 0
UserB
Task 1
Time: 7.25s
Errors: 0
Questions: 2
Task 2
Time: 5.59s
Errors: 0
Questions: 0
Task 3
Time: 12.07s
Errors: 0
Questions: 0
Task 4
Time: 1m 26.7s
Errors: 2
Questions: 2
Number of losses: 1
Task 5
Errors: 0
Questions: 1
Time: 7.1s
UserC
Task 1
Time: 3.22s
Errors: 0
Questions: 0
Task 2
Time: 2.37s
Errors: 0
Questions: 0
Task 3
Time: 5.07s
Errors: 0
Questions: 0
Task 4
Time: 3m 31.64s
Errors: 1
Questions: 2
Number of losses: 7
Task 5
Time: 3m 31.64s
Errors: 0
Questions: 0
Discussion
What we learned from our user study:
As usual having test users unfamiliar with our user interface test our prototype was very enlightening. Features and components that one would assume to be quite obvious with clear labels still posed some obstacles or hesitations by users. For example two users asked about clicking buttons for the first task of adding the most popular verse to their account.
We also learned that there are a variety of users out there with a variety of preferences and skill level. One user kept saying that they had trouble typing fast and that our game was too challenging for them. We learned that our user interface needs to be able to playable for a variety of skill levels and we need to do this by adding more customizable features such as block passing speed.
Also we assumed that our game was pretty intuitive: blocks have words, you type them in, you progress. You see a blank word and since we are trying to have users memorize the verse we would assume the user would see it as a blank for the user to fill in, however one user typed in the underscores that represented the blank “____”. In our class readings we were told not to assume that this was an isolated special case and assume it represents a group of possible users out there. We see that our game does require some instructions or tutorials.
Overall this pilot usability study helped reveal the dangerous assumptions we made while designing our interface. It also helped show the variety of users we might encounter in the future.
Changes we plan to implement for our next prototype iteration:
- So that we can make the gameplay more intuitive, in order to progress to the next word block we will have users press either <Enter> or <Spacebar>. <Enter> because it is the intuitively key to press when entering information and <Spacebar> because it maps out directly to how people usually type out verses on the computer.
- We will give the option to users to press <Enter> instead of clicking on the button, when there is only one button on the screen available to press (like start, retry, next level, etc.) for faster gameplay where users won’t have to change from mouse to keyboard (homing).
- Some users complained that the blocked scrolled to quickly and that the speed should be customizable. We plan to have an option of entering your words per minute, or play a mini-game to calculate your words per minute in order to better tune the gameplay to each user.
- We plan to make the end screen more consistent with the in-game graphics. Also we plan to better the graphics in general (animation, clouds, and more). Also we want the buttons to highlight when the mouse is over the buttons because right now the buttons just look like images.
- We may make it so that when a user types the correct word for a blank block, the block will materialize the correct word, in order to help players keep track of where they are.
- When the user fails the level on a word, we will show the word they failed.
- Maybe have the word fade out as it comes across the screen, to indicate the user should memorize the word. Also explore different ways to help facilitate memorization.
- We will add the option of viewing an instructions “how to play” screen or tutorial before playing the game. Many users were confused of how to play in the beginning.
- We will add some lag time before the user has to start typing.
- The Top Scores screen will show cumulative scores first, because this is what is commonly viewed as a more intuitive ranking system.
- We plan to make certain fonts larger in general to make them more readable. Also we will choose better color combinations to make font easier to read.
- We also plan to implement our login and account features and connect our components together without the use of wizard of oz if possible.
Appendices
Demo script
This is our game: Bible vs. Fire, a game intended to both entertain and assist in memorization of Bible verses.
- Start from the Login screen.
- Users can log into the website to access their personalized list of verses. But before they can do that, first-time users must create a new account. Enter desired username and password, then click on 'create new account'. A confirmation window pops up, and users can now log into their newly created account.
- Show the various tabs logged-in users can view.
- Users can see a list of their saved verses in the 'MyVerses' tab. Here, they can chose to add a new verse, edit the existing verses, or remove verses from their list.
- Users can also edit their account information.
- Show how to start playing the game
- In the game, click on 'start'. Users have to get their character to the end of the level by correctly typing in the words inside the blocks the character is standing on to move to the next block. In doing so, the users are typing (and committing to muscle memory) the verses from start to end. If the user doesn't type quickly enough, their character will fall into a rocky pit of doom and perish; the player loses a life and starts again from the beginning.
Task instructions
- Find the most popular verse and add it to your account.
- Find the 2nd highest-scoring user's profile.
- Randomly generate a verse, and add one that you like to your account.
- Play two rounds of the game without losing all three lives.
- Lose the game, view the end game results, and find the word on which you lost.
Comments, questions, and errors
Raw data (i.e., entire merged critical incident logs)
User A
- “Do I just click it?” (during the “add the most popular verse” task)
- “I thought this was a game…” (referring to the verse suggest and top scores feature)
- (while playing the game) <Squints>, “I can’t see the words.”
- <Died on level 1>
- <Died twice on 2nd level>
- <Never died until completion>
User B
- “Wait 2nd or 1st most popular verse?” (didn’t hear the question well)
- “Where’s my account man?” (not yet implemented)
- “Do I have to type spaces? No I don’t.” (Answers own question)
- “Feels weird to type without spaces.”
- “What is that?” (refers to blank blocks) <Tries to type actual underscores> “I get it”
- <Wants to press enter to continue to next level instead of clicking on the button.>
- “Ah I hit enter again.”
- “What about punctuation?”
- “What this?” (The yellow block on the end screen)
- “Font is too small on results screen” (the key for the marks on the verse).
- “Make result screen look better.”
User C
- “Oh sick hahahaha” (refers to picture of Will)
- “When I click this button it lies to me?” (referring to Add to My Verses)
- “How about an intro game to test for how fast you type.”
- “What are there underscores for spaces?”
- “You misspelled Corinthians!”
- “Do I have to cap?”
- “I’m getting nervous”
- <Presses enter after every word>
- <Didn’t know it automatically progresses without having to press enter or space.>
- “If I mess up what happens?”
- “You guys should have a lag time in the beginning of the game.”
- “Why don’t you guys have instructions?”
- “Hey this is very small font” (the key for the marks on the verse).
- “It’s like when you like get like insurance like…” (?)
Task Time Results
Task 1: Add the most popular verse to your account
| Participant | Time |
| UserA | 23.53s |
| UserB | 7.25s |
| UserC | 3.22s |
Task 2: View the profile of the second highest scoring user
| Participant | Time |
| UserA | 10.32s |
| UserB | 5.59s |
| UserC | 2.37s |
Task 3: Randomly generate verses and add one you like to your account
| Participant | Time |
| UserA | 17.61s |
| UserB | 12.07s |
| UserC | 5.07s |
Task 4: Complete two levels of the gameplay successfully
| Participant | Time |
| UserA | 2m 22.91s |
| UserB | 1m 26.7s |
| UserC | 3m 31.64s |
Task 5: Discover which word you died on
| Participant | Time |
| UserA | 5.96s |
| UserB | 7.1s |
| UserC | 3.53s |
