PilotStudy-Group:MetalPonyZ

From CS 160 User Interfaces Sp10

Jump to: navigation, search

Contents

Introduction (5 points)

The Boy Scouts have made building model citizens a major focus of their program. To achieve good standing with his peers, the scout will have to be organized. The modern scout has several requirements he must fulfill in order to progress in the Scouting program. Currently, these requirements are spread across a collection of books for each individual badge that each scout must carry with him. The hassle of carrying around multiple books has caused many scouts to leave them at home. However, our scouting application, iScout, aims to prevent the inconvenience of the often forgotten, large and clumsy book, as well as provide other organizational tools to allow the scout to keep track his own progress, and also the progress of his fellow scouts. Specifically, our app seeks to improve the scout's awareness of the merit badge requirements for the set of badges he is currently working on by providing a means to look up a list of badges and their requirements directly from his iPod/iPhone and take note of his progress. Other tools include an event roster, a meal planner, and a fundraising group map/roster, all there to assist the scout in his most important organizational tasks.

The purposes of our experiment are as follows:

  • Determine whether our app is intuitive on first-time use, with no instruction
  • Determine what portions of our app are effective for users
  • Determine what portions of our app are confusing for users
  • Get user feedback on how to improve our app
  • Develop our own ideas on how to improve our app, based on user experience
  • Determine overall usability of our prototype

Implementation and Improvements(15 points)

At the time of evaluation, our iScout app consists of two main components: a badge tracker for Boy Scouts to track their own progress on badge advancement, and a fundraiser organizer that allows Boy Scouts to section off fundraising areas and assign drivers and members to fundraising groups. We also added a "Preferences" section to add new members or remove current members to the application's database.

Badge Tracker Improvements

  • Increased size of checkboxes for checking off requirements
  • Added loading indicator while waiting for individual badges to finish loading

Fundraiser Improvements

  • Fundraiser Troop map no longer requires Wizard of Oz (displays actual overlays set by user in group maps)
  • Group/Troop overlay colors
  • Default zooms in on Berkeley area

Preferences Improvements

  • New "Preferences" section
  • Implemented adding new users to the database
  • Implemented removing current users to the database

Method (10 points)

Participants

We chose to revisit Troop 24 located in Berkeley, CA from our Contextual Inquiry and Task Analysis. Our participants were scouts randomly selected by the Troop leader.

Participant 1

  • Boy Scout Rank: Star
  • Years in Scouts: 1.5
  • Age: 12
  • iPhone/iPod Touch ownership: Owns iPod Touch (Beginner)

Participant 2

  • Boy Scout Rank: Life
  • Years in Scouts: 7
  • Age: 17
  • iPhone/iPod Touch ownership: Owns iPod Touch (no applications) (Beginner)

Participant 3

  • Boy Scout Rank: Life
  • Years in Scouts: 5
  • Age: 16
  • iPhone/iPod Touch ownership: Does not own, some experience with friend's iPod touch (Beginner)

Apparatus

For this assignment, we used a second generation iPhone (iPhone 3G) to run our test. Our testing area was in the church that Troop 24 held their meetings. We set up a table in the back of the church placing the iPhone on the side of the test subject. To take notes, our notetaker used a pen and notebook, but afterward typed up the results to distribute to the rest of the group. One groupmate used a digital camera to film the user's hands interacting with the prototype.

Tasks

During each task, we looked for how easily the test subject accomplished each subtask. Specifically, we noted any hesitations/delays, slips, mistakes, and categorized them under unfamiliarity with iPhone UI conventions or misunderstandings created by our own design.

Easy: "Keep track of personal progress in multiple merit badges and look up individual merit badge requirements." Based on nearly unanimous demand for a portable index of merit badge requirements. To meet this demand, our team set out to present an easy-to-use, familiar design for the scouts to navigate through the numerous merit badges, implementing a requirement check-off system and a progress icon for each badge. For this task, we looked to see how easily the test subject could:

  1. Find the badge he is looking for
  2. Determine what, and how many, requirements he has completed, and how many he has left
  3. Set a date for completion of specific requirements
  4. Find all his completed badges

Easy: "Add a new person to the database." For other portions of our application (Food and Event Planner) we will also need to access the same people as in the fundraising portion. In our new preferences pane, we are able to add new members, whether they are troop members or parents. For this task, we looked to see how easily the test subject could:

  1. Find the Add Person menu
  2. Successfully add a new person
  3. Add the new person to an existing Group

Medium: "Keep track of friends' progress on merit badges." Advancement through merit badges is often done in tandem with other scouts. By allowing scouts to share one's badge work with another, they can better coordinate common badge-earning events together. For this task, we looked to see how easily the test subject could:

  1. Find the Friends view in the Badge Tracker
  2. Look up and verify all categories of badges for a specific Friend
  3. Look up how many requirements were completed for a specific badge

Hard: "Assign members of a troop to smaller fundraising groups to designated areas of a neighborhood." As a non-profit organization, scouting typically relies heavily on fundraisers to fund outdoor outings and other troop events. Several times a year, scouts will go into their surrounding neighborhood to sell goods, distribute food collection bags, and even pick up and clear christmas trees as a way to pay for camping. Our app provides a utility to help divide a scout troop into groups and draw on top of a neighborhood map to section off each group's coverage. The scout can also reference the driver's name and phone number for all groups. For this task, we looked for how easily the test subject could:

  1. Add a new Fundraising Group
  2. Edit Driver and Phone number of this new Group
  3. Find the Group map
  4. Draw an area on the Group map
  5. Correctly save the Group map
  6. Change the Group color
  7. Consult the Troop map.
  8. Delete Members from a Group
  9. Delete Groups

Procedure

For this assignment, we used a second generation iPhone (iPhone 3G) to run our test. Our testing area was in the church that Troop 24 held their meetings. We set up a table in the back and had our test subject and note taker, Linsey, on one side of the table and our narrator, Hugh, on the other side. Our note taker took notes using a notebook and pen, and after the experiment typed them up to distribute the results to the rest of the group. Raymond also recorded portions of the test using a digital camera.

The actual procedure occurred as follows:

  1. Ask the test subject sign the consent form, taking note of age, rank, years in the Scouts, and iPhone experience
  2. Hugh introduces our team, the application, and the purpose/workflow of the experiment to the test subject
  3. Hugh begins dictating tasks to the test subject
  4. As the test subject completes tasks on the app:
    1. Linsey records data about the subject's comments and interaction with our app
    2. Hugh pauses test subject's interaction to ask questions verifying the test subject's knowledge of the situation
    3. Raymond is filming the test subject's hands interacting with the iPhone
  5. After all tasks are complete, Hugh asks the test subject for his opinions on the application and Linsey notes the responses
  6. We thanked the test subject

Test Measures (5 points)

  • iPhone experience
    • Measured on a scale of None, Beginner, Expert
    • Experienced iPhone users will have a slight edge on basic usage of iPhone
    • Will be more familiar with iPhone UI standards
  • Age
    • Measured purely by age (Scouts are aged between 11-18)
    • Younger children are usually more impatient with instructions
    • Younger will more likely rush through the tasks without actually listening to instructions, thus making more mistakes
    • Older troop members are also more likely to be in organizational roles so they might find the fundraising portion more useful
  • Troop Rank
    • Each scout belongs to a certain rank
    • Depending on ranking, they might be more/less willing to use the badge tracker
    • Higher ranking scouts might find the fundraising portion more useful since they will be organizing
  • Number of mistakes made in badge tracker
    • Based on notes, we can count the number of mistakes made
    • Break it down between mistakes based on iPhone UI knowledge or based on our UI mistakes
    • We can prioritize fixes for the app based on this data
  • Number of mistakes made in fundraiser
    • Based on notes, we can count the number of mistakes made
    • Break it down between mistakes based on iPhone UI knowledge or based on our UI mistakes
    • We can prioritize fixes for the app based on this data

Results and Discussion (25 points)

Badge Tracker

Participant 1 Results

Task 1
Subtask Result
Find a specific badge Scout went Straight to Badge Tracker and then the American Business Badge, so navigation for the task was very intuitive
Determine what, and how many, requirements he has completed, and how many he has left Scout knew he has completed 5/6 requirements
Set a date for completion of specific requirements Scout was able to set requirement due date and check-off incomplete requirements with ease
Find all his completed badges Scout knew to use the navigation bar at the bottom of the screen to view all completed badges
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 0

 

Task 2
Subtask Result
Find the Friends view in the Badge Tracker Scout went to preferences first, then eventually navigated back to the main menu, badge tracker, and friends (Application Interface Error)
Look up and verify all categories of badges for a specific Friend Scout completed task smoothly after finding the "Edit" button
Look up how many requirements were completed for a specific badge Scout completed task smoothly after finding the "Edit" button
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 1

Participant 2 Results

Task 1
Subtask Result
Find a specific badge Scout went straight to the American Business Badge and was eager to check off incomplete tasks without being asked
Determine what, and how many, requirements he has completed, and how many he has left Since scout was at the bottom of the screen after checking things off, he didn't really get a chance to read off the completed requirements (Platform Unfamiliarity Error)
Set a date for completion of specific requirements Scout was able to find the first aid badge quickly and change the due date
Find all his completed badges Scout knew that he needed to go straight to the badge tracker and the 'Completed' tab to view completed badges
Number of Errors due to Platform Unfamiliarity: 1

Number of Errors due to Application Interface: 0

 

Task 2
Subtask Result
Find the Friends view in the Badge Tracker Correctly found friend's badges
Look up and verify all categories of badges for a specific Friend Correctly read off the specific friend's badge categories
Look up how many requirements were completed for a specific badge Scout correctly read off the number of completed requirement's for the friend's specific badge
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 0

Participant 3 Results

Task 1
Subtask Result
Find a specific badge Went straight to the American Business Badge
Determine what, and how many, requirements he has completed, and how many he has left for the specific badge Knew that the badge was not completed and that he only had 5/6 requirements fulfilled
Set a date for completion of specific requirements for the specific badge Added a new due date easily
Find all his completed badges Took a bit to find completed badges, said it was kind of confusing for him (Application Interface Error)
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 1

 

Task 2
Subtask Result
Find the Friends view in the Badge Tracker Swent to preferences first instead of the badge tracker (Application Interface Error)
Look up and verify all categories of badges for a specific Friend Knew which badges were completed upon entering Chuck N's badge view
Look up how many requirements were completed for a specific badge Easily read off the number of requirements finished for bugling badge
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 1

Discussion

From our interaction with the users, overall the users had little to no difficulty understanding the interface of the badge tracker. Determining the number of requirements they completed was very obvious to them, as well as how to check off requirements they completed. The users easily figured out the number of complete and incomplete badges they had with the navigation bar at the bottom.

Though our badge tracker is largely successful, we will have a few changes in the UI to iron out some of the small problems the users had. Because the users had a harder time finding the "Friends Badges" portion, we might switch the bottom navigation bar to simply "My Badges" and "My Friends" badges, and somehow incorporate the "Complete" and "Incomplete" portions within that view. We might do this by having a "Sort By" button which will bring up options for the user to sort his badges.

Although the users did not have any problems finding how to change their requirement's due dates, our current button for changing the due dates is unclear, so we will remedy this by having a new button that says "Edit Calendar".

 

Fundraiser

Participant 1 Results

Task 3
Subtask Result
Add a new Fundraising Group Didn't know to hit edit immediately when in the 'Groups' section, then took a while to realize that he needed to exit Edit Mode in Groups (Platform Unfamiliarity Error)
Edit Driver and Phone number of this new Group After editing the Driver's name hit 'Done', then took a while to realize he needed to go back into edit mode to edit the Driver's number (Platform Unfamiliarity Error)
Find the Group map Instead of clicking on the group map, exited the group and went to the Troop Map (Application UI Error)
Draw an area on the Group map Didn't know he needed to click 'New Shape' to start drawing at first (Application UI Error)
Correctly save the Group map Hit 'Done' on the navbar before saving the shape
Change the Group color Didn't immediately realize he needed to hit 'Done' after selecting a new color (Application UI Error)
Consult the Troop map. Found it easily due to his previous mix up with the Group Map
Delete Members from a Group Added/deleted members easily, and knew what the check and x meant
Delete Groups Correctly deleted a group
Number of Errors due to Platform Unfamiliarity: 2

Number of Errors due to Application Interface: 3

Participant 2 Results

Task 3
Subtask Result
Add a new Fundraising Group Didn't enter edit mode at first (Platform Unfamiliarity Error)
Edit Driver and Phone number of this new Group Hit 'Done' after editing the driver name, and didn't go back into edit mode immediately while trying to edit the number. (Platform Unfamiliarity Error)
Find the Group map Successfully found the group map
Draw an area on the Group map Took a while to draw an area since the phone was lagging
Correctly save the Group map Hit 'Done' before saving the new shape
Change the Group color Chose a new color and hit 'Done' immediately afterwards
Consult the Troop map. Successfully consulted the group map
Delete Members from a Group Added and removed members easily, and knew what the check and x meant
Delete Groups Was able to delete groups easily
Number of Errors due to Platform Unfamiliarity: 2

Number of Errors due to Application Interface: 0

Participant 3 Results

Task 3
Subtask Result
Add a new Fundraising Group Originally tried to edit an existing group instead of making a new one, and had to ask for help. Didn't realize he had to hit 'Done' after creating a new group. (Platform Unfamiliarity Error)
Edit Driver and Phone number of this new Group Didn't know he needed to enter Edit mode to edit the driver's information. (Platform Unfamiliarity Error)
Find the Group map Immediately found the group map
Draw an area on the Group map Tried drag-drawing instead of dropping pins at first. (Application UI Error)
Correctly save the Group map Correctly saved the shape and hit 'Done'
Change the Group color After selecting a new color, he didn't hit 'Done' immediately
Consult the Troop map. Easily found the Troop map
Delete Members from a Group Successfully added/removed members from a group and knew what the check and x meant
Delete Groups Successfully deleted a group
Number of Errors due to Platform Unfamiliarity: 2

Number of Errors due to Application Interface: 1

Discussion

The participants initially found the fundraising section more difficult to navigate, however, with a bit more interaction the user's were able to get a feel for the flow of the application. Many of our errors were derived from platform unfamiliarity (i.e. editing modes, saving states, etc.). Many of the users weren't sure of the functionality of the map portion of the fundraising section. Some solutions to this would be to add a help/tutorial section before entering the map drawing frame, or have a helpful message overlay over the map while drawing. There was also the one user who was trying to drag on the map instead of mapping pins, but once he realized there were pins, he said that he preferred pins to drag-drawing since the shapes created are cleaner and more accurate. Another problem with the map is that you can edit it at anytime, so we might need to change it to only be editable while in 'Edit' mode, then make it strictly viewable while not in edit mode.

We realize that there is room to add some more functionality that will actually clarify the purpose of the fundraiser's map portion. For first time users, we might make a pop-up modal instruction view that can be disabled after it's first appearance, or we might just add an info button into the map section explaining use. In the view where they select to edit the troop map, we should have more information to explain the purpose of editing the map. In either case, the users should be able to use the map drawing functionality much easier after getting acquainted with the interface. As for users hitting 'Done' before saving for the map (which currently doesn't save new regions), we can just add a check into the done function to save the current shape if one exists. In the group color menu, we will probably just change it so that clicking a color automatically exits the menu. Perhaps to make the application more friendly to first time users, we can change some of the edit buttons to "Add/Remove" buttons to make the interface more intuitive.

Though users did not complain about the order of how things were displayed in the fundraising group's detailed view, we feel that we should reorder the table cells because they are somewhat randomly arranged at this point.

In conclusion, the overall feel and structure of the fundraising section is really flawed and quite confusing, so we are probably going to need to go in and restructure the fundraising section.

 

Preferences

Participant 1

Task 4
Subtask Result
Find the Add Person menu Went straight to 'Edit Troop' in preferences
Successfully add a new person Added a new person to a group, but after going back to 'Edit Troop' never hit 'Done' (Platform Unfamiliarity Error)
Add the new person to an existing group Added the new person to a group in the 'Add New Member' menu
Number of Errors due to Platform Unfamiliarity: 1

Number of Errors due to Application Interface: 0

Participant 2

Task 4
Subtask Result
Find the Add Person menu Went straight into preferences and 'Edit Troop'
Successfully add a new person Knew he had to go into edit mode, but didn't initially realize that he had to click on the '+' instead of the entire cell (Application UI Error)
Add the new person to an existing group Hit 'Done' immediately after typing in the new person's name, so didn't have a chance to add them to a group (Application UI Error)
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 2

Participant 3

Task 4
Subtask Result
Find the Add Person menu Found preferences easily
Successfully add a new person Didn't realize he had to hit '+' instead of the entire cell to add a new member (Application UI Error)
Add the new person to an existing group Successfully added the new person to a group while still in 'Add New Member'
Number of Errors due to Platform Unfamiliarity: 0

Number of Errors due to Application Interface: 1

Discussion

The users found adding new members simple to do. They understood the general idea of how to add people, but they struggled because of some of problems in our application's interface.

We will need to make the entire cell selectable to add new members instead of just the '+' icon. We also need to remove adding users to groups immediately because the purpose of this page is to just add a person. Multiple portions of the application will be using this information - not just the fundraiser. Removing adding users to groups will eliminate the problem of the user accidentally not selecting a group when they intended to.

 

Overall Application

Participant 1

  • Application rating: 4/5
  • Comments:
    • Said he had difficulty with selecting an area for his group to cover in the group map section
    • He said that he liked the application a lot, would use it himself, and would give the application a 4/5 if he had to pay and a 5/5 if it were free

Participant 2

  • Application rating: 4/5
  • Comments:
    • Thought that drawing on the map was weird and that there should be an option to specify specific points instead of shapes
    • Suggested that we add some sort of information/help button onto our map
    • The user stated that if he had an iPhone, he would most likely not use it because he likes to organize things on paper. However, if he was required to use it, he would enjoy using it for it's simplicity.

Participant 3

  • Application rating: 4/5
  • Comments:
    • Really liked the badge tracker and found it intuitive
    • Liked the Fundraising Map overlay, and said he preferred the pins to actual drawing since the pins are more precise
    • Had a hard time deleting people from the troop
    • He liked the name 'iScout'

Discussion

Our application got a very good response overall. Our users were happy with the badge tracker except for some small problems that did not detract from the usability of the application. We had more problems with the fundraising portion because we did not provide enough information on how to use the fundraising portion. I believe adding a help section to our application will eliminate a lot of the problems we had there.

Appendices (5 points)



[add comment]
Personal tools