PilotStudy-Group:Coach Class
From CS 160 User Interfaces Sp10
Contents |
Introduction
Our group performed a pilot usability study to gauge user performance on our iPhone app: the Coach's Clipboard. We designed the app to help basketball coaches in their everyday tasks, with a focus on simplicity and speed. After receiving verbal feedback on our interactive prototype through interviews, we wanted to empirically measure the qualities of our app. Thus, we designed a usability experiment, conducted on three basketball coaches, to determine not only the success of our current design, but also what changes need to be made for future iterations.
Implementation and Improvements
- Added alternative interface for tracking stats during a game (noun-verb interaction vs. the original verb-noun interaction)
- More detailed stats (average per game of all categories) and more types of stats in stats tab
- Added arrows to lines following a player when player is moved in a play
- Undo button when creating a play
- Better default starting positions for players when creating a play
- Cancel button in Choose Starters and Substitution views
- Clearer wording in Substitution view
Method
Participants
Participant 1
The first participant is a 31-year-old high school basketball coach. He has used an iPod touch a few times in the past and so is somewhat familiar with the interface. We interviewed this coach for our contextual inquiry assignment and asked him to participate again at this stage of the project.
Participant 2
The second participant is a 23-year-old basketball coach for an intramural team. He does not own an iPhone but has used one before and is thus relatively familiar with the interface. Contacted through the Berkeley Intramural Sports website, he is a new participant who knew nothing about our app before participating in the study.
Participant 3
The third participant is a 24-year-old part-time basketball coach. He owns an iPod touch and is familiar with the iPhone/iPod touch interface. Referred to us by a previous participant in the low-fidelity prototype, this coach was a new participant and had never seen any part of our app before his interview.
Apparatus
In order to achieve maximal similarity between our three interviews, we standardized the equipment and locations that we used. For each interview, the participant performed the tests on an iPhone with our app installed on it. Additionally, for our third task, we used a laptop to display a video of a basketball game. Lastly, each interview was conducted in a confined room with no other people and minimal distractions.
Tasks
Our tasks for the pilot usability study remained mostly the same as for the previous stages. However, we made the wording a bit more specific so that all users would perform exactly the same thing. This eliminates some confounding variables that could occur if users were, for example, typing different text or creating different plays. We also included a video for coaches to watch for the third task, so that they would all record the same statistics in a real game scenario.
When the tasks were performed we recorded the time of any critical incident (such as a user telling us what he did or did not like) as well as the occurrence of anything related to the dependent variables we measured.
Task 1 - Taking notes (easy)
Create and save a new note in which you write that "Kobe Bryant isn't passing the ball enough in practice." Then, pretend some time has passed and go back to read the note.
Task 2 - Drawing a play (moderate)
Create a new play called "Two Passes" consisting of the following steps: 1) Have the shooting guard start near the top of the 3-point line, a few feet away from the point guard. All other players should stay in their default positions. 2) The first step involves the shooting guard moving to the left corner of the court and the point guard passing the ball to him. 3) The center moves a few feet to the right and the shooting guard passes him the ball. Then, pretend some time has passed and go back to view the play, going through all of its steps.
Task 3 - Player stat tracking (hard)
This task will be repeated with two different versions of our stat tracking interface (the old one and a new one we are testing) selected in random order, to see which one is more effective. Pretend you are coaching the Lakers and they are starting a game right now. Begin a new game in the app, and select a starting lineup of Bryant, Gasol, Fisher, Bynum, and Vujacic. Then, watch this video (http://www.youtube.com/watch?v=v0aSYZs3kK0) and record as many statistics during the gameplay as you can. As substitutions happen in the video, we will pause the video and instruct you on who to substitute in and out. The substitution times and resulting lineups are provided below:
Start (Bryant, Gasol, Fisher, Bynum, Vujacic), 1:04 (Farmar, Brown, Odom, Bynum, Bryant), 1:09 (Gasol, Walton, Brown, Farmar, Odom), 1:14 (Gasol, Farmar, Bryant, Artest, Odom), 1:30 (Gasol, Bynum, Bryant, Artest, Fisher), 2:14 (Brown, Farmar, Odom, Gasol, Bryant), 2:44 (Brown, Artest, Bynum, Odom, Bryant), 2:59 (Fisher, Odom, Bryant, Artest, Bynum), 3:03 (Fisher, Bynum, Bryant, Artest, Gasol)
Procedure
Although all three interviews were conducted at different times and locations, each interview followed the exact same format. We started by greeting the participant, asking him to sign the consent form, and then giving a general overview of our app. We then briefed the participant on the first task and waited for the user to complete it, keeping track of any errors and how long it took. We repeated this process for the second and third tasks, thanked the user for participating, and concluded the test.
Test Measures
For each task, we measured:
- Time to complete the task - to see if any tasks take the user much longer than expected to complete
- Number of errors - to see if the task is particularly error prone
For the drawing a play task, we measured:
- Time spent after a mistake while drawing (if any) until the user is back on track - to see if correcting mistakes takes too long
For the stat tracking task, we measured:
- Number of stats not tracked - to see if our interface was fast enough to allow a user to capture all of the statistics
Results and Discussion
Results
After doing a statistical analysis of our raw data from our interviews, we found the following statistics:
There was only 1 drawing mistake from 1 user in the play drawing task, so we only had one data point for this section which is not statistically significant.
In the table, interface #1 represents our original version of the Game tab, and interface #2 is the new version which we created in an attempt to reduce the number of screens required and time taken to track stats. The order in which our users tested the interfaces was chosen at random to eliminate biases from learning and fatigue. We also tested to see if the difference between the average completion times of these two tasks was statistically significant, and found that t=2.06 and p=0.11. This leads us to believe that the difference is fairly statistically significant and that the new version performs faster than the old version. We also tested to see if the difference between average number of total errors of these two tasks was statistically significant. We found that t=3.32 and p = 0.029, and therefore we were able to reject the null hypothesis and conclude that the new version significantly reduced the number of errors that users make.
Discussion
Although it is difficult to judge how well our app works with data from only three users, it seems relatively clear that for the game section of our app, the new version works better than the original. The mean time to complete the task was shorter with the new interface, and the average number of errors was much smaller as well. Thus, based on the results, we intend to replace our old game interface with the new interface in the next version of our app.
Additionally, we received suggestions to display players sorted by last name to make them easier to find and select. Another suggestion related to the list of players was to add player numbers, so that players could found in that way as well if desired. Finally, it was suggested that we use buttons instead of a table view for listing the players as there are only 5 of them, and we agree that this would likely make the interface a bit easier to use. We plan on implemented all of these suggestions in a future iteration.
Although the focus of the study was on the game tab, there are other additional changes we plan to make based on the results. One change in particular is to rename the Game and Stats tabs. At the beginning of task three, 1 of our participants mistakingly went to the statistics portion of our app first, as opposed to the game portion. This needs to be addressed to make the use of our app less confusing for beginners.
Also, there was an error during one of our tests where the user tried to save a note after entering the note text but not a title. We will remedy this by allowing notes to be saved without a title and simply displaying the first few words of the note in the note list.
Lastly, another suggestion we heard several times for the Playbook tab was to draw arrows without moving players at first, and then move players to the ends of the arrows when a new step is created to better mimic a real coach's clipboard. We will also consider implementing this in a future iteration.
Appendices
Raw Data
Participant 1
Notes:
0:00 - start
0:38 - hit save before entering title (error)
0:49 - done (1 error)
Draw play:
0:00 - start
1:10 - done (no errors, no time taken to correct mistakes)
1:11 - comment (create arrow without moving player, then move player when starting next step)
Track stats (old version):
0:00 - start
0:37 - confused about whether to use stats tab or not (error)
0:48 - pushed bryant for both shot and assist (missed stat, error)
1:55 - tapped new player without deselecting old player during substitution (error)
5:27 - mistake substituting (error)
6:25 - missed stat (missed stat, error)
6:30 - mistake, hit back button (error)
7:17 - done (6 errors, 2 missed stats)
Track stats (new version):
0:00 - start
0:07 - comment: should sort players in abc order by last name
0:35 - at first confused about why there are player names in this interface
1:10 - comment: i like this interface better
2:27 - comment: have 2-pointer and assist as first two menu items, they're used the most
5:45 - done (no errors, no missed stats)
Participant 2
Notes:
0:00 - start
0:05 - tapped "notes" (error)
0:11 - tapped plus button
0:42 - done (1 error)
Draw play:
0:00 - start
0:18 - hit save after setting start positions instead of next step (error)
0:52 - suggests an undo feature in case he made a mistake
1:25 - done (1 error, no time taken to correct drawing mistakes)
Track stats (old version):
0:00 - start
0:12 - is unsure of enter roster button and why it’s in game tab
0:44 - was looking for where to enter blocked shot
1:32 - this time wondered why there was no turnover button
1:45 - hits foul first, before hitting shot for Gasol’s layup (error)
4:29 - forgot to set Bryant as assist to Odom (error, missed stat)
5:33 - forgot a rebound (error, missed stat)
6:05 - comments that we should have player numbers
7:12 - records wrong player taking shot (error, missed stat)
8:21 - done (4 errors, 3 missed stats)
Track stats (new version):
0:00 - start
0:09 - suggests to sort players by last name
2:21 - forgot an assist (error, missed stat)
3:49 - substitution error (was getting tired) (error)
5:05 - comments that he prefers this interface
6:32 - done (2 errors, 1 missed stat)
Participant 3
Notes:
0:00 - start
0:43 - done (0 errors)
0:44 - User expresses preference for voice notes, with the audio being automatically transcribed. User also states that typing is too slow on the iPod touch.
Draw play:
0:00 - start
0:08 - moved the SG to the wrong original position (error)
0:10 - moved to SG to the right position (corrected error)
0:52 - done (1 error, 2 seconds taken to correct drawing mistakes)
0:53 - User expressed view that the basketball blocks the view of the player, that the players were too big with respect to the court, and that the next step feature was difficult to figure out when using it for the first time.
Track stats (new version):
0:00 - start
2:48 - recorded incorrectly a stat belonging to Gasol (error, missed stat)
3:45 - incorrectly records an Artest 2 pointer (error, missed stat)
7:03 - misses a Fisher rebound (error, missed stat)
7:06 - done (3 errors, 3 missed stats)
7:07 - User comments that the enter roster feature seems confusing, that the buttons in this interface are too small, and that there is no point in using a list view for selecting the players. He also expresses his desire for separate defensive and offensive stats and for, ideally, stats to be automatically recorded to his phone without him entering them.
Track stats (old version):
0:00 - start
0:23 - comment: the interface seems much more complicated
1:08 - Missed recording a Fisher assist (error, missed stat)
2:25 - Marked wrong player for assist (error, missed stat)
3:24 - Kobe assist recorded incorrectly (error, missed stat)
4:32 - Did not record a shot by Brown (error, missed stat)
6:02 - Recorded a stat even though no stat occurred (error, missed stat)
6:43 - Accidentally clicked play success (error)
7:08 - done (6 errors, 5 missed stats)
7:09 - User believes this version was better because the buttons were bigger making them harder to miss. He believes that in general play tracking is very difficult. However, he believes this version was easier to use though slower. He believes that it could be improved by removing confirmation buttons (e.g. done) and also by not using list views for selecting players, but instead using buttons.
Demo Script
There are four main things you can do with the app, and there are four tabs for them.
First is the playbook tab [playbook tab is already selected and has a pre-made play]. Here you can create new plays and view saved plays. You tap the plus button to create a new play, [tap plus button], and then you can move the players and the ball around on the screen, [move around players and ball a bit in some random fashion]. [Tap done button and do not save the play]. After you save a play, it will appear in the list of plays as you can see with the one we already have saved.
Second is the game tab [tap on Game tab]. Here you keep track of in-game statistics for players. When a game begins you tap the New Game button [tap the button], and then choose your starters [choose 5 random starters from the already-input Lakers lineup and press Done]. Then, when events happen in the game you tap the corresponding buttons on this screen and input appropriate information on the following screens to record the statistic [show an example of a player committing a foul by tapping the Foul button, selecting a player, and tapping Done].
Third is the stats tab [tap on Stats tab]. Here you can view the various statistics that have been recorded for each player. Tapping on a player brings up that player's statistics, and the back button will go back to the list of all players [tap a player, show the stats view for a few seconds, then tap the back button].
Finally, there is the notes tab [tap on Notes tab which contains a pre-made note]. Here you can write and read notes. Tap the plus button to add a note [tap plus button]. Tap on the title field to write in the title, and tap on the note field to write the note [point to title and note fields]. Tapping save in the top right corner will save the note. [tap back button] You can tap on a note in the note list to read it.
While you are performing the tasks, please point out anything you come across that you consider good or bad about the interface, along with any other comments you may have.
Actual Statistics in the Video in Order
| Stat Number | Type | Player |
|---|---|---|
| 1 | 2 Pointer | Bryant |
| 2 | 3 Pointer | Bryant |
| 3 | 2 Pointer | Gasol |
| 4 | Assist | Fisher |
| 5 | 2 Pointer | Bynum |
| 6 | Assist | Bryant |
| 7 | 2 Pointer | Bynum |
| 8 | Assist | Fisher |
| 9 | 2 Pointer | Farmar |
| 10 | 2 Pointer | Gasol |
| 11 | Assist | Odom |
| 12 | 2 Pointer | Gasol |
| 13 | Rebound | Bynum |
| 14 | 2 Pointer | Bynum |
| 15 | 2 Pointer | Artest |
| 16 | 2 Pointer | Artest |
| 17 | 2 Pointer | Bryant |
| 18 | 2 Pointer | Bynum |
| 19 | Assist | Bryant |
| 20 | 2 Pointer | Brown |
| 21 | 2 Pointer | Odom |
| 22 | Assist | Brown |
| 23 | 2 Pointer | Bryant |
| 24 | 2 Pointer | Farmar |
| 25 | Assist | Gasol |
| 26 | Rebound | Bynum |
| 27 | Rebound | Brown |
| 28 | 2 Pointer | Brown |
| 29 | 2 Pointer | Bynum |
| 30 | 2 Pointer | Artest |
| 31 | 2 Pointer | Bryant |
| 32 | Rebound | Fisher |
Consent Form
CONSENT TO PARTICIPATE IN A USER INTERFACE STUDY
Title: Usability of Basketball iPhone Application
PRINCIPAL INVESTIGATORS:
Name: Alex Sydell, Brian Chin, Jonathan Hirschberg, Jordan Klink, Wei Wu
Email: asydell@berkeley.edu, brianchin@berkeley.edu, jonathan.hirschberg@gmail.com, jklink@berkeley.edu, weiwu@berkeley.edu
Dept.: Electrical Engineering and Computer Science
INTRODUCTION:
You are being invited to volunteer as a subject in an evaluation being conducted at the University of California, Berkeley. This consent form provides you with the information you will need when considering whether to participate in this evaluation. All evaluation and research studies carried out at UC Berkeley are governed by federal and state laws regulating human subjects’ research. If you decide to participate, you will be asked to sign this consent form which states that you have read this document, that any questions you have about the evaluation have been answered, and that you agree to participate.
STUDY PURPOSE:
Through this study, we will better understand the usability of an iPhone application developed by UC Berkeley students. Your expected duration of participation is 1 hour.
You qualify as a possible participant in this evaluation you have experience in coaching basketball, and are currently actively coaching basketball. Both males and females are eligible to participate, but must be 18 years of age or older. The University of California, Berkeley is the only institution participating in this evaluation.
STUDY PROCEDURES:
If you decide to participate in the evaluation you will be asked to complete three tasks on an interactive prototype of an iphone application. The three tasks are:
1) Taking notes - where you will be asked to take notes using the application.
2) Drawing a play - where you will be asked to draw a play using the application.
3) Player stat tracking - where you will be asked to keep track of statistics using the application while watching a video of a basketball game.
Each of these tasks will done using the interactive prototype provided and will take less than 20 minutes to accomplish. In addition, at the end of the study, you will be asked for your opinion on the prototype and any other insights you can provide.
To ensure confidentiality of all of your responses, you will be assigned a Study ID number that will become your Study ID number throughout this study.
STUDY RISKS:
Your participation in this evaluation involves no physical risk. There is the possibility of psychological risk if your results of using the interactive prototype were made public. Because of that risk we are maintaining strict control over all data. The reason for assigning you a Study ID and using that number to code all of your answers is to reduce the risk that any answer you give can be tracked back to you. The master list of Study ID numbers will be kept in a password-protected electronic file.
STUDY BENEFITS:
Benefits to you may include a better understanding of how to use iPhone applications. Also you will be involved in the creation and improvement of an application that you may use in the future. As such, you may find it beneficial to shape its development to suit your needs as a coach.
Benefits to society may include a better understanding of how well certain designs work. Also, the successful development of the application could substantially improve the work of basketball coaches everywhere by providing an hi-tech tool to replace many of the outdated technologies currently used.
COSTS TO THE SUBJECT:
There are no costs for participating in this evaluation.
COMPENSATION:
There is no compensation for participating in this evaluation.
CONFIDENTIALITY:
If you consent to participate in this evaluation, your personal information will be kept confidential.
FUNDING:
There is no funding for this evaluation.
VOLUNTARY PARTICIPATION IN, AND WITHDRAWAL FROM, THE STUDY:
The decision whether to be in this evaluation is entirely up to you. Participation is voluntary. You can refuse to participate, or withdraw from the evaluation at any time, and such a decision will not affect your relationship with the University of California, Berkeley, either now or in the future. Nor will a refusal or withdrawal of participation result in the loss of any other benefits to which you are otherwise entitled. Signing this form does not waive any of your legal rights.
STATEMENT OF CONSENT:
I have reviewed the evaluation design outlined above and have had any questions I have about the evaluation answered to my satisfaction. I understand that my participation is voluntary and that I can withdraw from the evaluation at any time without prejudice. Signing this form does not waive any of my legal rights.
By signing below, you are indicating that this form has been explained to you, that you understand it, and any questions you have about the evaluation have been answered. You are indicating that you understand the ways the evaluation data may be used and how your privacy will be protected. By signing this form, you are agreeing to participate in the evaluation at this time only.
I ACKNOWLEDGE THAT I HAVE READ THE ABOVE EXPLANATION OF THIS EVALUATION THAT ALL OF MY QUESTIONS HAVE BEEN SATISFACTORILY ANSWERED, AND I AGREE TO PARTICIPATE IN THIS EVALUATION.
______________________________________ Signature of study volunteer
______________________________________ Printed name of study volunteer
Date _______________
I CERTIFY THAT I HAVE EXPLAINED FULLY TO THE ABOVE SUBJECT THE NATURE AND PURPOSE, PROCEDURES AND THE POSSIBLE RISK AND POTENTIAL BENEFITS OF THIS EVALUATION.
______________________________ Signature of principal investigator/designate
Date ________________
