PilotStudy-Group:The A-Team

From CS 160 User Interfaces Sp10

Contents 1 Introduction 2 Implementation & Improvements 2.1 New 2.2 Improved 3 Method 3.1 Participants 3.2 Apparatus 3.3 Tasks 3.3.1 Easy: Watching Video 3.3.2 Medium - Adding a Spot 3.3.3 Hard - Adding/Sharing Video 3.4 Procedure 4 Test Measures 5 Results & Discussion 5.1 Participant 1 Results 5.2 Participant 2 Results 5.3 Participant 3 Results 5.4 Discussion 5.4.1 Aesthetics 5.4.2 Usability 5.4.3 Organization 6 Appendices 6.1 Latest App Version 6.2 Consent Form 6.3 Demo Script 6.4 Task Instructions 6.4.1 Task 1 6.4.2 Task 2 6.4.3 Task 3 6.5 Screenshots of Major Improvements 6.6 Raw Data 6.6.1 Participant 1: 6.6.2 Participant 2: 6.6.3 Participant 3: 6.7 Schedule

Introduction

• The system being evaluated is our skater application, called "Skatemaster". It is an app designed to help skaters keep a checklist of all the tricks they can do, and upload videos of themselves performing those tricks onto the app (and consequently, onto YouTube). Users can log into the app using their YouTube User ID. Users can watch other users' videos to try to learn new tricks. Users can also check out cool skate spots on the map posted by others or add a new skate spot. Also, users can share their favorite videos with their friends by emailing their friends YouTube video links.
• The purpose of this study is to test our interface on some target users, particularly to test the changes that we have made based on our interactive prototype study. We have made several changes to the application since our lo-fidelity prototype, and we wanted to make sure that these changes improved our user experience. Also, it is a big change to test our interface on paper versus on the actual iPhone. By conducting pilot study with potential users, we hope to find additional usability problems with our interface and make improvements for the final product based on the interview responses.

Implementation & Improvements

Our previous iterations of this application had problems such as confusion about the applicability of the app, issues with the simple aesthetics, complaints about the limited functions, and questions about what the complete functionality of the app. We have since made many improvements and changes and hopefully solved those problems, and also answered possible questions in the sections below. (Images of major improvements can be found below in the Appendix)

New

• Progress bar on the List screen that shows what percentage of tricks the user has completed. Adds to the usefulness of the checklist function.
• The ability to add a skate spot (the user's current location or a spot chosen via a map) has been enabled since the interactive prototype.
  • The user can take a picture of the skate spot from within the app, or add a photo already in the Camera Roll
  • True location-awareness so the user can mark his/her current location as a new spot
  • User can also use a map to choose the location of the new spot
• Implemented logging into YouTube in order to upload your tricks so that each user can be uniquely identified.

Improved

• Darker green for the checkmarks and other aesthetic changes (unified color theme, borders for information boxes, etc.)
• Can now check off a trick from the Trick Info screen.
• Reorganized the home screen to better serve the needs of the user and make it less bland.
• Eliminated the Favs tab in favor of a Video tab that allows the user to easily access all video functions (browsing, uploading, and sharing)

Method

Participants

We decided to get participants from different starting skating experience levels to get a better idea of which functionalities were most useful. We also mixed up the familiarity with iPhones, independent of skating experience, to get a better understanding of how that factor influenced our teat measures of time and errors. We did not use any of the subjects from the lo-fi tests.

	Participant 1	Participant 2	Participant 3
Age:	21	18	21
Sex:	Male	Male	Female
Skateboarding Experience:	limited - a few times	a year, skating occasionally	none, but interested
iPhone Familiarity:	Owns iPhone, familiar with standards	Owns iPod Touch, can navigate around the iPhone	limited, don't know the standard navigations

Apparatus

Since some of the functionality of our application does not function on the iPhone Simulator, we decided that in order to get a better feel for the whole user experience we needed to do this study with the application installed on the iPhone. This also gave us the ability to easily bring the test to our participants, so as to not ask too much of them or take too much of their time having them meet us somewhere where our laptops can have internet access for use with the iPhone simulator (internet access is needed for our app to function properly).

• Limitation: iPhone 3G still did not have video recording functionality that could be used with our app, instead just used the videos already in the library

A stopwatch was used to measure the time taken to complete each task as well as keep track of when critical incidents occurred during each scenario. We also used an audio recorder to record each task so we could later go back and review the interviews to make sure we did not miss any important incidents.

Tasks

Easy: Watching Video

This is the main function the skateboarders requested in a mobile application, so that they can view their previous videos and other users' videos as well. This task is directly accessed from the 'Video' tab. Under "Browse Videos", the user selects "View Video categories", and then the user is taken to the trick list so that they can find the video of the specific trick they want to watch. This trick list can be sorted by Difficulty or Type.

Medium - Adding a Spot

This is a new functionality that was added for this prototype, so it was very crucial that this be one of the tasks tested by the user. Adding a new skate spot onto the app's map is a feature that many interviewees wanted us to implement. To add a spot, the user selects the "Map" tab. The user is taken to a world map, which shows the user's current GPS position on the map. On the upper right hand corner, the user can select "Add Spot". The user can then name the spot, take a picture of the spot, and select coordinates for the spot, whether it is the user's current location or selecting a new location by dragging a pin on a map.

Hard - Adding/Sharing Video

The functionality of a trick checklist is the distinctive service that only our application offers based on the competitive analysis. The checklist features a simple interface with either a check or an X and videos associated with each of the tricks. The user selects the trick that they want to upload a video to, and edits the location that the trick was performed at - similarly they can upload a video via the "Video" tab. Adding a video (and consequently checking off that trick from the checklist) is somewhat more difficult because in order to check off a task, the user need to properly add the video and tag the location where the trick was performed. Sharing a video was also one of the tasks that interview subjects had the most difficulty performing during the low-fidelity prototyping. We streamlined the process because sharing their accomplishments or just a neat video was an important function the interviewees specified. To share a video, the user taps the "Video" tab, selects "Email a video link", selects one of the video's they have uploaded, then taps "Share" and types in their friend's e-mail address. The friend will receive a link to view the video on YouTube.

• An important clarification is why the use videos rather than other methods to check off a trick. Pictures do not do a trick justice or may not even be distinctive enough to pinpoint what specific trick is performed. Simply tagging locations does not provide the proof that a trick was actually performed, and contributes nothing to other users. The beauty of videos is that it adds utility for the user as well as other nearby users who may learn from local videos.

Procedure

When performing the pilot study with our users, we first introduced ourselves and explained what our project was about. We asked the user some demographic information, and proceeded to introduce them to the application, showing them a basic task based on the demo script to get them somewhat familiar with the application. We followed our script and described each task that the user had to perform. For each task, we kept track of our test measurements and a log of critical incidents. Since we asked our participants to think aloud, we noted all their comments and their actions while using the application. We took a lot of notes including things the users particularly liked, any issues they had while using the app, and any suggestions they have. Finally, we thanked our users for their valuable feedback.

Test Measures

• Time taken to complete each task
  • Since our app is designed to help skateboarders improve their skills, users are likely to use this app at actual skate spots. Such users will want to be able to rapidly access the functions of our app so that they can get right back to skateboarding. By measuring completion time for the benchmark tasks, we can gauge whether or not we have done a good job of making the interface simple and quick to use.
• Number of touches to complete each task
  • We want our app to be efficient, so users should not have to go through too many screens or tap an excessive number of times in order to perform a task. If our measured number of touches is too high, then we will need to redesign certain screens to optimize the interface.
• Number of errors during each task
  • By measuring the number of errors made by users, we can discover which aspects of the interface are confusing or unclear for users; we can then redesign these aspects to make them more user-friendly.
• Number of fatal errors during each task
  • We define "fatal error" as either a bug that crashed the app or a mistake made by the user that causes them to unexpectedly exit the app. By measuring this statistic, we can separate minor issues from major usability disasters that we must immediately fix.
• User satisfaction for each task (based on a Likert scale of 1 through 5)
  • We will ask the user to rate their agreement with the statement "I am satisfied with the ease of completing this task." This metric gives us a better understanding of how user-friendly (or not) our design is.

Results & Discussion

Participant 1 Results

Task	Time	Touches	Errors	Fatal Errors	Likert	Result
Task 1	1:26	5	0	0	4	Liked: video distance from current location; use of app for urban environment; simple and easy, not too cluttered with colors
Task 2	2:24	8	2	0	4	Liked: integrated ability to take photo with iPhone camera; use of Google map and knowing current location Need: loading prompt; street level zoom instead of upper level zoom; too white, kinda bare, add texture to background
Task 3	3:34	Add Video: 15 Share Video: 5	1	0	4	Liked: difficulty ratings Need: multiple location tagging for each trick; adding more than one video per trick; more prompts
Overall Impression/Suggestions Need a confirmation screen or another new screen to put on top of home page once logged in Coolest thing: smoothness of map, distance to nearest spot (maybe miles and # of blocks) Improvements: default zoom to street level, allow multiple locations for single trick, a bit more aesthetics

Participant 2 Results

Task	Time	Touches	Errors	Fatal Errors	Likert	Result
Task 1	1:21	6	0	0	5	Liked: tabs remember where it was last left off; the trick list
Task 2	1:31	10	2	0	5	Liked: control with the map Need: share spots with friends to meet up -> unique ID for each spot on the map
Task 3	2:02	Add Video: 9 Share Video: 5	2	0	4	Liked: integration of iPhone library and mail client Need: to show the uploaded videos within the video tab menus
Overall Impression/Suggestions Applicable to other uses, such as "Tricks", performing spins and jumps at different spots Improvements: make aesthetics not so bland, make the spots and map functions more user friendly

Participant 3 Results

Task	Time	Touches	Errors	Fatal Errors	Likert	Result
Task 1	1:18	7	1	1	4	Liked: YouTube video view Need: add definitions for beginners; change the page titles to make navigation clearer; hard to see the distinct areas, add borders or colors Fatal Error: didn't know where to quit the YouTube video, pushed home button (unfamiliar with standard navigation)
Task 2	1:01	12	2	0	3	Liked: good map interface Need: "Add Spot" doesn't stand out; border off input screen to make it look like a form
Task 3	2:05	Add Video: 10 Share Video: 6	1	0	2	Liked: clean list of tricks Need: "back" button unclear/confusing; video tab page has too many options, too much going on
Overall Impression/Suggestions Need an alphabetical sorting option for the tricks for the beginners who don't know any better Improvements: more bordering and clarifying; confusing for first time iPhone user; clean up video tab

Discussion

Aesthetics

With our interface prototype we have learned a couple of new things about our app that the lo-fi prototype did not show very well. Many of our users had qualms about the aesthetics of our app since it looks quite bare and does not have too much color. Also some of our forms for input such as inputting the name of spots was not that large and some users had suggested that having the name be shown in a bigger font and instead of a textfield just having an edit button would be much better. In addition, the picture that was displayed when the user took a picture was a little small for some users when they took it in a portrait view and they mentioned that having the gradient background would help alleviate the feeling of empty/unused space on the app. One thing that users enjoyed was being able to add videos using a list and having the checkmarks for the tricks was something that our users enjoyed since it was a visual representation of their updates.

Usability

In general, our users were most impressed with the usability of our application. Being able to upload to YouTube and download videos off of YouTube was something that our users liked a lot. Also, integrating the Google maps into the app was another thing that users enjoyed. However, some small quirks about usability came up with some users. For example, Google maps is initially zoomed out to the global scale, some users mentioned that they would like the map to be zoomed in to the street level so that they don't have to keep zooming in to find their spot on the map. To fix this system we would simply have the initial view be zoomed into the current location and on a street level view. Another usability feature that some users suggested was to have a loading icon since a lot of our features are loading from web servers, which can be quite slow depending on the user's current WiFi or 3G signal.

Organization

Having the ability to list tricks by difficulty and type was something that our users liked but they also wanted to be able to organize the tricks by name so that looking for a particular trick wouldn't be so difficult. So how we can fix this is by just creating a new tab that sorts alphabetically. Also, the users didn't like the fact that there was so little real estate on the tricks list page, since there were large tabs for the sorting that could be real estate used for displaying the tricks we decided we could fix this by moving the tabs into a sort button on the top right that would pop up a menu as to which way they wanted to sort the information. Another thing that one of our test users mentioned that we felt was a good point was that there should be a unique identifier for each skate spot. This way friends who also use the app could just say "let's meet at spot 34" today so that people will know which spot it is on the app rather than trying to guess from location and trick type. We already have a skate_spot id so we just need to display it for the users convenience.

Appendices

Latest App Version

• Skatemaster v2.0
• Note that this is quite large when uncompressed because it contains all of the YouTube API as well

Consent Form

• Consent Form

Demo Script

Thank you for participating in this usability study. This is our iPhone skater app, which allows skaters to track which tricks they can perform, improve their skills by watching other users' videos, and find fresh skate spots added by other skateboarders. I will first show you a basic function, and then I will ask you to perform some tasks.

The first thing I’m going to show you is how to browse the trick list: After opening the app, you can browse the trick list by switching to the List tab. Here, you can sort tricks by difficulty in terms of stars (one star for easiest and five stars for hardest), or you can sort by trick type, such as jumps, flips, or grinds. Tricks you have completed are marked by a green check, whereas tricks that you have yet to complete are marked by a red X. Also, the progress bar at the bottom of the screen tells you how many tricks you have completed so far.

Now it’s your turn to interact with the app.

Task Instructions

Task 1

Imagine you're at a skatepark and you just saw a skilled skateboarder perform a heelflip. As a beginner, you really want to learn how to perform the trick, so you'll use this app to watch a video of a heelflip uploaded by another user.

Task 2

You've just found an awesome skate spot that no one else seems to know about. You want to share this spot with other skaters, so you'll use the app to add the current location.

Task 3

After lots of practice, you've finally managed to complete a kickflip, and your friend has recorded it on your iPhone. Now you want to check the trick off your list. Afterward, you want some feedback on your technique, so you'll use the app to share the video with a friend via email.

Screenshots of Major Improvements

Home Screen

Before: bland and contains unnecessary buttons

After: reorganized (with a working YouTube login) and more visually appealing

Trick List

Before: no progress bar, and a light green checkmark that classmates gave negative feedback about

After: progress bar based on number of trick completed, and a darker green checkmark

Add Spot

The Add Spot screen, with options to change name, picture, and location

To add a different location, the user can drag the skateboard pin wherever they like

User can add a picture by taking a new picture within the app or choosing from the Camera Roll

Video Tab

Before: Unimplemented Favs tab, which users said was unnecessary

After: Video tab, with all the useful video functions in one easily accessible place

Raw Data

Participant 1:

Task 1:

• 0:25 Enjoyed the tips that were given for the trick that he chose and mentioned that it's a neat feature
• 0:34 Saw the list user videos and said that's cool.
• 0:35 Noted that he liked the distance label so that he could know where he could perform the certain trick at a spot and if it was good for him.
• 0:48 Watched the youtube video play and enjoyed that feature.

Task 2:

• 0:08 Really liked the map feature of our app and enjoyed that it showed the current location.
• 0:30 After scrolling and zooming to find the street level view he mentioned that it would be better to have it already zoomed in at the current location.
• 1:12 Noted that the name label was kind of small when adding a spot. Also tried to exit by just clicking on another part of the screen instead of the done label.
• 1:53 Constantly clicked on current location to edit location which was still loading, he then said that we should have a loading icon so people know that it's loading.

Task 3:

• 0:34 Tried to add the video without putting in any information.
• 0:57 Mentioned that he liked the list of videos that were displayed.
• 1:32 After adding the video, noted that we should be able to upload our trick again in case we do a better trick later on.
• 2:24 Mentioned that he liked how the progress bar worked and the difficulty rating for tricks was an interesting feature.

Participant 2:

Task 1:

• 0:21 Selected Heelflip from the list
• 0:31 Mentioned how much he liked the YouTube integration
• 1:10 Said being able to watch other users' videos was "really super useful"
• 1:21 Finished task 1 and went back to the home screen

Task 2:

• 0:21 Hit Done before actually finishing typing
• 0:48 Took a photo from within the app
• 0:52 Accidentally hit Back instead of Done
• 1:01 Mentioned they liked being able to use a map to choose a location, but didn't like how far zoomed out it was by default
• 1:31 Task completed

Task 3:

• 0:21 Selected Kickflip from the trick checklist
• 0:32 Accidentally selected Add to Existing Spot
• 0:41 Finished checking off the trick but wasn't sure where to go to share the video
• 1:20 Mentioned that he was confused because User Videos didn't show him his video (not yet implemented)
• 1:43 After much searching, finally found his video listed under Kickflip
• 2:02 Sent the video link to a friend's email - task completed

Participant 3:

Task 1:

• 0:30 Asked "is it loading?" - the video took quite some time to load because of spotty WiFi
• 1:11 Wanted to exit the video but didn't know where the Done button was - shouldn't be an issue for iPhone users since we used the existing YouTube API
• 1:18 Fatal error - exited the app in order to quit the video

Task 2:

• 0:10 Tapped the Map view to try to add a spot rather than hit the Add Spot button
• 0:23 Mentioned that Add Spot is hard to see
• 0:45 Mentioned that the Add Spot screen does not look like a form - not immediately obvious information is needed from the user
• 0:53 Accidentally went back instead of hitting Done when editing location
• 1:01 Task completed

Task 3:

• 0:09 Went to Video tab rather than the Trick List, something neither of the other users did
• 0:17 Selected Kickflip from the trick checklist
• 1:19 Accidentally hit the Home tab while editing the trick's location
• 1:30 Finished checking off the trick
• 2:05 Sent the video link to a friend's email - task completed

Schedule

• Programming/Cleaning
  • Thur, individually work on Login, Map, Google API, Aesthetics.
  • Fri, day off (or do work not properly done on Thur)
  • Sat morning until lunch, consolidate, discuss, code
  • Sun morning, finalize edits and coding (Similar to last weekend, but in the computer lab this time)

• Interviewing/Write-up
  • Mon night right after lecture (EVERYONE attend), go interview real kids with iPhone
  • Tue throughout the day, flesh out write up
  • Wed, Finalize Wiki and unfinished stuff

---