PilotStudy-Group:O
From CS160 User Interfaces Fa06
Contents |
Introduction
This study was undertaken to measure the effectiveness of our system for interpreting the results of pen-and-paper games with GAMESMAN. The system is given as input a set of strokes representing one or more games played by a user. After docking the pen, the user is shown each game that he played and prompted to supply the name of the game and any important details regarding how the game was played (for example, whether a game of Tic-Tac-Toe was played with the objective to make three-in-a-row, or whether the objective was to force one's opponent to do so). The system combines these details with the strokes recorded by the pen to produce a file that records the game. This file can then be read by GAMESMAN to load the game and apply the analysis tools it provides, such as Value History.
This particular experiment was conducted as a preliminary usability test of our system. Users were asked to play a sequence of games using the Anoto digital pen, upload these games into a file, and use the Visual Value History provided by GAMESMAN to determine at which point (if any) the player in the lead made a play that took him from a winning position to a losing one. Users were observed to determine the speed and accuracy at which they could perform these tasks. The results were then examined to analyze the ability of this type of experiment to provide useful feedback for improving our interface.
Implementation and Improvements
At the time of the interactive prototype, our project was capable of reading a game of Tic-Tac-Toe and loading it into GAMESMAN. Since then, there have been several improvements to our project's functionality. The most significant of these is the addition of Dots And Boxes to the games that our system can recognize and interpret. The user can now play a game of Dots And Boxes with any size board up to 3x3 squares (width and height can be independently set) and load it into GAMESMAN. The Dots And Boxes module is not perfect (it occasionally fails to recognize strokes, and it will not recognize the board unless the dots are drawn left-to-right, top-to-bottom) but it is available for users to try out, and it was used extensively in our interviews. The Dots and Boxes algorithm, as well as Tic-Tac-Toe, will be improved in future iterations.
Another significant feature that has been added is an animated display of the user's strokes. When the user first brings up a game in the Upload File utility, the strokes that make up this game are displayed in a panel on the left side of the window. Previously, only the finished game was shown. Now, the panel starts out blank and the strokes appear upon it in the order that they were drawn by the user. We also added a button that, if pressed while the animation is being displayed, stops the animation and displays the finished game; if the finished game is already displayed, this button starts the animation over from the beginning (the label on the button changes appropriately to describe each action). This feature is intended to aid the user in recalling the details about an individual game.
Finally, there have been several smaller changes to the interface. There is a button that can be clicked to bring up a graphical file chooser to assist the user in specifying the location and filename of a GAMESMAN save file. Each game is analyzed and saved immediately after clicking Upload, and clicking the Finish button now brings up a dialog asking the user if he wishes to launch GAMESMAN immediately. Responding with Yes or No will quit our Upload utility, clicking Cancel will get back to the Upload window. There have also been some changes to the codebase, specifically replacing all previous R3 code with our own custom classes. This isn't visible to the user, but adds certain features that we need to analyze the games easier.
Method
Participants
As we discussed very early in the project, there are two major target user groups for our product: those involved with the GamesCrafters research group and gamers who are interested in using the GAMESMAN system in order to analyze the games they play to formulate strategies. As such, we chose our participants in the study from these two groups of people. The first two participants we selected (User 1 and User 2) were both selected to be in our study because they were members of the GamesCrafters research group and are already very familiar with the GAMESMAN system. As an added bonus, they would also be interested in seeing our results, as it is related to something that they work very closely with.
The third user (User 3) was selected as he represented the other major target user group for our product, gamers who are interested in analyzing the games they paly for formulate strategies. He was selected because he was unfamiliar with the GAMESMAN system, and thus would be valuable in providing insight from someone who is not already intimate with the GAMESMAN system and may not already have certain skillsets or assumptions involved with operating GAMESMAN.
Apparatus
The equipment we used were merely a Windows laptop that had the Anoto Digital Pen suite installed, as well as the GAMESMAN system and our code for the upload UI. We also used the Anoto digital pens, the notebook that was provided for us in order to play games on, pens and paper to take notes on any notable events during the interview, and stopwatches to take careful note of how long tasks took to complete and at what time of performing each task any notable events occurred.
Tasks
Playing a Game
The first task (easy task) that players were asked to do involved simply playing games of their choice based on the games that were implemented with the Anoto digital pen system (in this case, Tic-Tac-Toe or Dots and Boxes). One of our team members was selected to be the designated opponent for each participant. Participants were encouraged to play at least one game of each, but were free to select whichever game they wished to play and however many number of games they wished to play. While participants performed this task, we paid attention to the way they made their strokes with the pen and any potential "unexpected" strokes that could cause our stroke, board, and space recognition algorigthms to be thrown off (for example, lines that go into other spaces in Tic-Tac-Toe).
Uploading a Game
The second task (medium task) that participants were asked to do involved uploading the games they played via the upload UI that we created. The main goal is to successfully navigate the UI to pick specific games and upload them into a format that can be read by GAMESMAN. After inserting the pen into the cradle, the software will automatically parse the pen data, then detect and separate each game, and then present the user with the Game Upload interface. It is up to the user to provide information via the upload UI, such as what type of game was played, what rule variants were used, and what is the filename that the file will be saved as. The user may choose to skip certain games, but those that are chosen will be recognized and saved out to a file that can be read by GAMESMAN. While participants performed this task, we paid attention to any questions or points of confusion that the participant had while working through the upload interface, as well as any mistakes the participant made while trying to upload games.
Loading a game in GAMESMAN and analyzing with VVH
The third task (hard task) involved participants starting up GAMESMAN, loading a game that they saved, and using Visual Value History to examine the values of their moves and identify any critical moves in the game. Users must first launch the GAMESMAN application and pick the game they want to use. Then they must load a game file that was saved previously by the upload step. After loading the game, the user is able to see all the moves made in that game, undo and redo the moves, and view the Visual Value History. It's possible to see the eventual outcome of the game (Win/Lose/Draw) for every possible move, and the user can choose a different move at any point to see the mistakes and learn better strategies. We think that both new and experienced users with respect to using the GAMESMAN system will spend most of the time on this task. While participants performed this task, we paid attention to any questions or points of confusion that the participant had while trying to analyze their game with GAMESMAN, as well as any mistakes the particpant made while trying to load the game.
Procedure
The procedure for this assignment was similar to the procedure we followed when testing our lo-fi prototype, except that this time the actions were carried out using the Anoto pen and our interactive prototype. During the interviews, we had one facilitator and two observers. The facilitator began by welcoming the subject and explaining the purpose of our interview. At this point, we obtained the subject's permission to continue and had him sign a consent form. We also used this opportunity to collect some statistics about the user, such as age, gender, major, and year in school.
The next step in the process was to explain our project in detail to the interviewee. The interviewee was asked whether he was familiar with the technologies we used in our project; namely the Anoto Pen and GAMESMAN. If he was not, we explained the technology and, in the case of GAMESMAN, performed a brief demonstration. We also took this opportunity to ask the subject whether he knew the rules of Dots And Boxes; if not, we explained them and played a sample game to demonstrate gameplay.
Afterward, the user began completing tasks. There was one observer situated on each side of the user, and as the user performed the tasks, the observers took notes and recorded critical incidents. The observers also ran a timer to see how long the user took to perform each individual task, and to note at what time the user performed actions that were deemed to be "errors." The observers counted these errors and noted how many were made during each task.
- Actions considered "errors" in task 1 included anything that would cause the pen to behave in an undesired manner; that is, strokes outside the range of where they would be expected by our interpreter, single moves consisting of multiple strokes, strokes that extended past the point at which they were intended to, etc. Gameplay errors such as the user making an unwise move were not considered "errors," as we were not evaluating the user's ability to win at games. (However, if such a gameplay error led to a pen error - the user attempting to scratch out a half-completed bad move, for example - this was noted.)
- Actions considered "errors" in task 2 included any actions that generated events in the Upload File interface that the observers deemed were not what the user intended. Uploading a game that wasn't intended to be uploaded, skipping a game that was intended, or uploading a game with incorrect details were all considered "errors" for this task. Unintended actions that produced no consequences (typos, clicks on non-active sections of the interface, etc.) were not counted as errors.
- Actions considered "errors" in task 3 followed many of the same criteria as actions in part 2. Actions that brought the user farther away from his goal were considered "errors." Starting up the wrong game, loading the wrong .gcs file, or incorrectly interpreting the Visual Value History were all considered errors.
In addition, any time the user became stuck and asked for assistance was generally considered to be an error. There were a few exceptions to this, such as situations that were caused by facilitator error. In these situations, the facilitator provided an explanation of what the problem was and how to resolve it. In addition, the facilitator provided assistance in locating the directory in which save files were stored; this is because the default directory into which our Upload File interface saved files was nested about 6 levels deep in a rather obscure location. Not being able to locate this directory was not considered an error, although not being able to determine which files within that directory belonged to which games was an error.
After the user finished all three tasks, the facilitator thanked them and asked for general feedback regarding the overall system. These comments were noted by the observers. Finally, the user was thanked again and the interview was concluded.
Test Measures
There were several things that we measured during our pilot study interviews. The first thing that we measured was the time that it took each user to complete each task. This was measured because we wanted to make sure that each task we had designed for the users could be completed in a reasonable time, particularly the uploading phase. If the UI is too confusing and the process of uploading each game takes too long, users may consider using the system we have designed to be too cumbersome and not worth the benefits it offers in bridging pen and paper and the GAMESMAN system. Therefore, we measured time in order to make sure that users were not taking too long for each phase.
We also measured the number of errors that users encountered upon completing each task, including stopping to ask questions if the user was too confused as to what to do next (this was considered an error, as the UI should be designed in a manner that it is apparent what to do next, because it is essentially non-functional otherwise). The number of errors associated with each task provide evidence with how problematic each part of our product may be. Also taken into consideration was the nature of each error and how often it came up with each user (whether it was a consistent problem or whether it was a problem that only a small population of users are likely to make). Obviously, a consistent problem is on a higher priority to fix than a problem that only one test user had with our interface.
Results
User 1
For the first task, the user spent a total of 5:15 playing three games, one game of Tic-Tac-Toe (0:50) and two games of Dots and Boxes (2:15, 2:10). No errors were observed while playing the games, but later it was determined that a mistake was made in playing the games of Dots and Boxes, due to the way that the dots were drawn while setting up the board. The current implementation requires that the dots be drawn in a certain order (from left to right for each row).
For the second task, the user spent approximately 1:55 to navigate through the upload UI and upload the games he just played on the dot paper. One major error was observed during this process, as the user clicked on the "next" button after filling out the information for each game, instead of the "upload game". In order to properly observe the user, he was not notified of his mistake until after he had gone through all the games, at which point he was stopped and was told that the correct way to upload the games is to select "upload game", and that in the current state, no games had been uploaded. The user then went back and properly uploaded all the desired games in a timely fashion.
For the third task, the user spent approximately 4:30 loading the saved games into GAMESMAN and trying to find the critical move/"turning point" in each game. As the default path had not yet been properly set, the team member leading the interview (Sean) had to assist the user to find the correct directory in which the game was saved. Upon loading the third game (Dots and Boxes), it was discovered that the space parsing algorithm for Dots and Boxes did not work correctly, as the loaded game did not look anything like what the game on paper looked like (as mentioned above, the reason for this was discovered to be due to the way the board was initially set up). A small bug was also discovered during this task in GAMEMAN, in which the saved game logic for Tic-Tac-Toe seemed to always load the game as a Misere game (3 in a row loses), a strange but notable bug. This would later be brought up to the GamesCrafters group.
During the debriefing and asking questions phase, the user commented that he really enjoyed using the Anoto Digital pen. He also commented that during the upload phase, watching the animation was pretty cool, but he felt like he had to finish waiting for it to finish animating before he could start uploading his games. He therefore suggested that the animation start with the final frame, with the option of re-animating it with the click of a button. He also commented on the confusion over the next/upload game button and wished that the correct GAMESMAN game module would launch immediately after the UI closed.
User 2
For the first task, the user spent approximately 5:15 playing three games: once again, one game of Tic-Tac-Toe (1:05) and two games of Dots and Boxes (1:57, 2:13). Though playing the games went fairly smoothly, there was an incident involving the Dots and Boxes game where the user accidentally failed to draw a line that fully connected two dots and thus drew a line with two strokes, with the second stroke continuing the first stroke to the dot. This was noted, as the algorithm for space recognition was still fairly minimal at this point and may not have read that stroke correctly.
For the second task, the user spent 3:05 navigating through the upload user interface. Once again, the user made a mistake in confusing the "next" button for the "upload game" button, and once again, the user was allowed to go through all the games in such a fashion before Sean intervened and informed the user of the correct way to upload a game. It is interesting to note that the user actually wondered aloud about the difference before hand, but went on to make the wrong choice. The user also asked a question about some of the options, when he hit the previous button to the Tic-Tac-Toe game, but the option selected was still on Dots and Boxes.
For the third task, the user had some trouble finding the right game to open in GAMESMAN (almost clicking on Connections, instead of Dots and Boxes). Once again, Sean had to inform the user of the default path, as that had not yet been fixed. Though the Dots and Boxes games came out better this time due to the correct setting up of the board, there were still some bugs to work out, as the very last move of the game, for some reason, had not registered in the saved game. Also, the Misere-instead-of-Standard bug came up again when trying to open the Tic-Tac-Toe game. The total time for the third task was approximately 7:00.
Overall, the user commented that playing on paper was straightforward and that he appreciated that we were trying to bridge the gap between GAMESMAN and standard pen and paper. He shared several comments and critiques as User 1, particularly about the wording issues of "next" and suggested it be changed to "skip" as to avoid confusion.
User 3
For the first task, the user spent approximately 4:50 playing three games: once again, one game of Tic-Tac-Toe (0:45) and two games of Dots and Boxes (1:56, 1:49). For the most part, the games went smoothly, with the exception of the third game (the second game of Dots and Boxes), in which the board was set up incorrectly. The game was aborted and a fourth game was started instead.
For the second task, the user spent approximately 2:30 navigating through the upload UI. For the third time, the user made the mistake of confusing "next" with "upload game" and was allowed to go through all the games before being informed of the correct method and told to try again. It was interesting to note that due to aboring the third game, the upload UI read nearly almost all of the dots that were created in the aborted game as separate "games" in and of themselves. Thus, the third game to be uploaded was actually the seventh "game" listed in the upload UI.
For the third task, the user spent approximately 8:00 loading the game through GAMESMAN and using the Visual Value History to determine where any crucial moves in the game were made. This user was unfamiliar with the GAMESMAN system, due to not being involved with the GamesCrafters research group, so before starting the task, he had to be explained some of the mechanics and the backgrounds of the GAMESMAN system. Despite his inexperience, the user had no trouble navigating through the GAMESMAN system, but had some trouble differentiating between the different saved game names, as he had not paid attention to what he had named the games earlier during the uploading phase and couldn't tell which ones were his, as there were several earlier saved games in the folder.
Overall, the user also had a good time experimenting with the pen and noted that the pen felt natural, like a normal pen. This user brought up a unique issue about the window in which the user selects the filename that the game will be saved under, as he thought that the filename box was just to select the folder the game would be saved under. He expressed concerns that this wasn't made clear enough and that we should probably include a way to make it clear what folder the game was being saved under and that the user was entering the name of the file.
Discussion
We learned about several big design flaws and found a few bugs during our pilot run. As noted above the biggest design flaw was illuminated by the fact that every participant clicked the “Next” button instead of the “Upload Game” button. This is one of the most critical parts of the application and not a single test user was able to figure out the correct way to do it so there is obviously something wrong with the design. This will be changed immediately and without this change a real experiment would not be as productive. We have discussed a few options, including changing “Next” to “Skip” or “Skip Game”.
Another thing we noticed is that none of the users changed the save filename of the game. We feel this is because it looked confusing and/or complicated and already seemed to have a decent default value meaning it didn't require attention. However, once the users tried to load the games they had to think harder to figure out which file corresponded to which game. We think that by changing the text around the filename box as well as changing the button text to read “Browse” instead of “Save As” that users might be more inclined to use it because they are more used to it. Another issue with this that the third user noted was that the entire path of the filename showed in the box and made it appear confusing and deterred him from wanting to deal with it. Indeed we only want the filename itself to show so we will make sure this is so in our final version. This will serve to lessen the confusion around that particular textbox.
Also dealing with the filename, we were aware before conducting the study that the default save location from the importer was not the same as the default load location in GAMESMAN. This meant that we had to help the users find the directory in which the files had been saved when they were trying to load games. This is something we want to address for our next iteration, but we will need some sort of configuration file describing the environment because where our tool is installed and where GAMESMAN is installed is specific to each person's computer.
Even with the major flaw and a few minor bugs here and there, all three users really liked the system. Much of this, we suspect, is due to the “cool factor” of the Anoto system since this was all of their first time using it. Specifically for the GamesCrafter's users though, they were obviously excited to see a new way to interact with their system which had probably never crossed their minds, but that they found very interesting. All three of the users appreciated the fact that it bridged a format of playing games that they were familiar with (pen and paper) with a tool that could help them understand where they (or their opponent) went wrong in playing games and formulate strategies to help them win.
Aside from fixing the above issues, we do not believe there are any major changes that need to be made in order to conduct a "real" experiment. Our protocol and routine worked very well. We were well prepared with a script and forms for the observers to fill out. If we were going to do a "real" experiment it might be advantageous to film the interviews so that we could watch them over later to catch things the observers missed. However, with our current system of having two observers, we feel we caught most items of interest and without the extra cost of setting up video and then having to watch it again later.
Workload breakdown
Percentage and nature of contribution from each group member (1 page)
Alex Wallisch (25%)
- Acted as observer during all experiments.
- Implemented functionality updates to Upload File interface.
- Contributed to report (Introduction, Improvements, and Procedure)
Jason Lee (25%)
- Created board detection and move detection logic for Dots and Boxes
- Acted as observer during experiments
- Contributed to report (wrote Results, parts of Method, and Test Measures)
Michael Udaltsov (25%)
- Changed all previously used code from R3 to custom code with new functionality
- Added support for line detection to help with game recognition
- Added animation playback functionality for the game strokes
- Contributed to the write-up (Implementation and Improvements section)
Sean Carr (25%)
- Created PDF forms for study
- Was interviewer for all 3 interviews
- Wrote discussion section of writeup
- Got software setup on demo laptop
- Fixed a couple bugs in GAMESMAN
- Tic-Tac-Toe always being misere
- Game not solving (having bad values)
Appendices
Image:Group O pilot consent form.pdf
Image:Group o pilot user info form.pdf
Image:Group O pilot per task results form.pdf
 |
Script for Interviews
Interviewer: Hello. My name is _________ and this is ____ <introduce others> ____. We've working on a project for our user interfaces class, and we'd like to have the opportunity to test it out. Would you be willing to participate in a usability experiment? All results of the experiment will be kept completely confidential, and you may stop the experiment at any time, although we don't anticipate that we'll be doing anything that will make you uncomfortable.
If interviewee agrees: Cool. Can you please sign this consent form. It basically covers the same stuff I just said: this is all confidential and you can stop at any time. We just need a record of the fact that you were informed of this and you agreed.
After interviewee signs form: Great. Before we start, let me describe a little bit about what we're doing. Have you heard of the Anoto digital pen?
If interviewee says no: Well, the Anoto pen is a pen with a camera inside it. There's a special kind of paper with a dot matrix that the pen can recognize; if you write on top of this paper, the pen can tell exactly where on the paper it is and can store this information. Then, you can upload your strokes to your computer later and get a record of what you drew or wrote. <Go to "Describe project">
If interviewee says yes: Well, that makes things easy. Let me tell you about our project then.
Describe Project: Our goal with this project is to allow people to play games on paper and then use a computer to analyze the game afterward. Are you familiar with GAMESMAN?
If the interviewee says no: GAMESMAN is a research project developed by a student group here at Berkeley. It has the ability to solve any finite, two-player, perfect-information game. At a simple level, it means we can solve games like Tic-Tac-Toe and Connect Four. In theory, the system could also solve Chess and Go, but these games are way too big for any computer to solve. However, for any game in the system, GAMESMAN can provide information at any point concerning who is winning and which moves will lead to a win, a tie, or a loss. GAMESMAN also has a few tools to analyze games after they've been played. For example, there's a feature called "Visual Value History," that displays the list of moves each player made and shows which were good moves and which weren't. Think of it as an explanation for "who messed up where?"
Regardless of how the interviewee answered: A large part of GAMESMAN is the user interfaces for playing the games in the system. GAMESMAN comes with a nice graphical interface for playing the games on the computer. However, a limitation of the system is that it *requires* the user to play on the computer. We've noticed that this isn't the most natural thing for everybody. A lot of people play games in situations where there aren't computers handy, and there are plenty of people who just prefer the feel of playing a game on paper.
Our project allows people to play games on paper using the Anoto digital pen and then upload the games into GAMESMAN to be analyzed. We'd like to take the opportunity to have you test out our system by performing some tasks with it. Is that alright?
If interviewee says yes: Great, thanks. Keep in mind that your performance doesn't reflect on you so much as it does on the quality of our interface. In other words, if you have difficulty figuring out what to do, that's a problem with our end rather than a problem with you.
<Interviewer brings out the paper> The first thing we'd like you to do is play few games of Dots And Boxes with ________. Are you familiar with the rules? <If not, we explain them.>
After user finishes playing a few games: Great. So you've completed the first task we had for you. Congratulations.
The next thing we'd like you to do is load this game into GAMESMAN. You'll do this by placing the pen into this dock and using the application that pops up to save the game into files that GAMESMAN can read. Just go ahead and save everything to the default directory. I'll be here to help you if you get stuck, but you should try to get as far as you can on your own. Please think aloud whenever possible so we get an idea of what components of the interface are unclear. When you're done uploading the games, we'll have you play around with GAMESMAN a bit, but we'll tell you about that when we get there.
After the user finishes uploading all of the games that they played: Alright, good job. You're done with part two. The last thing we'd like you to do is use GAMESMAN to analyze your game. Remember how in the 1st <or 2nd or whatever> game we played, you beat _____? Can you go into GAMESMAN and use it to tell us where he made the play that caused him to lose? Again, if you get stuck I'll give you a prod in the right direction, but you should try to use the tools GAMESMAN provides you to figure it out on your own. Once again, let us know what you are thinking and what about the interface is unclear.
After the user shows us the critical move: Good! That's the move where ______ messed up. You did great.
That's the end of the tasks that we have for you today. Before we finish up, can you just give us some general feedback on the interface we've provided. We're especially interested in the interface used for uploading strokes from the pen into .gcs files, as we have the most control over that aspect of the interface.
After listening to user feedback: Alright, that's great feedback. It seems like we're pretty much done. Thanks a lot for helping us out.
