PilotStudy-Group:GroupM

From CS 160 User Interfaces Sp10

Jump to: navigation, search

Contents

Members

Introduction

Our iPhone application, iBC's, is an edutainment tool that allows children to creatively learn their ABC's and 123's. The user group is primarily children around the ages of 3 to 5 and their parents. The app solves the everyday problem parents face when they're busy trying to work, drive, or perform any number of tasks, while also entertaining their child. In this pilot study, we are testing a high fidelity prototype of our application, having implemented most of the core functionality of the program, including tracing, coloring, image viewing, and settings. The purpose of this iteration is to find out how kids and parents will interpret and interact with out app now that it has finally be ported to an actual iPhone. We wanted to test the understanding of different icons, menus, and how the parents felt about the different setting available on the application. During our tests, we tried to keep track of any errors that were made, as well as the underlying reason for those errors. The rationale of the experiment is that we will test four different tasks, two targeted toward the children and two targeted toward the parent, and see how the users accomplished these tasks. The kid tasks will focus more on the tracing and drawing aspect of the application, while the parent tasks will focus on viewing saved work and changing the settings of the application to better suit their needs.


Implementation and Improvements

Old Implementation New Implementation Changes Description
1. Stroke Overlay We added ordered stroke directions to the letter, number, and word tracing views; this was both suggested during the interactive prototype and planned (but not implemented) by our group beforehand.
No Image No Image 2. Audio We added audio to the letters, number, and words tracing views; when the user visits the view they will hear the pronunciation for the corresponding character in their instructions. In the interactive prototype this was done with Wizard of Oz. We have also added audio for saving and deleting pictures.
No Image No Image 3. Settings, Random Cards We implemented the random cards setting in our app so now it has the ability to show random letters, numbers, or words it the corresponding tracing views.
4. Settings Since the settings section of our app is going to be viewed mostly by parents, we decided to make it look more like settings page in a regular iPhone program, unlike the child-friendly interface used for the rest of our application.
No Image No Image 5. Changed Swiping to Buttons After the feedback from the interactive prototype, our group decided that instead of swiping to go to the next card in the tracing view, we can just use buttons. None of the users realized that they could swipe to change views, so we decided that simply using buttons might be a better idea.
No Image No Image 6. Implemented Undo In the latest version of our application, the undo last stroke in the tracing and canvas view has been implemented. Furthermore since users had problems understanding that the arrow icon meant undo, we decided to change the undo icon to an eraser so the user has a easier time understanding.
7. Added Done Button We added a done button as a task completeness trigger. After the child is done tracing the letter/number/word, they will tap the done check mark and our app will give them feedback on their tracing.
8. Added Corresponding Canvas Pictures Before our app simply used the apple canvas for all its canvas pictures, but now the user can access different connect-the-dot pictures corresponding to the correct letter.
9. More User Friendly Home Menu The Home menu used to be flat, and the buttons didn't afford pushing. Now we made the buttons more 3D looking, which means they afford pushing/tapping, which helps the user understand how to navigate the Home screen.
10. Added Icons to Home View From the feedback of the class and users, we realized that the meaning of some of the buttons on the Home menu are not obvious to the children using the application, since they cannot read yet. As a result, we decided to add icons next to the words in order to help users recognize where the buttons on the Home page lead to.
11. Changed Info Button to Question Mark During our tests, we realized that some users didn't realize the info button meant help; we changed them to question mark icons so our user group could recognize them better. When we used the generic iPhone info button, young kids didn't realize that it meant help, since they were not familiar with the analogy yet.
No Image No Image 12. Added Shake to Clear In this iteration of our application, we have implemented shake to clear in the tracing and canvas views.
No Image No Image 13. Single Tap to View Saved Images One of the feedbacks from our interactive prototype was that double tapping to view a saved image in the Album view was unnecessary and not typical iPhone behavior. As a result, we decided to change the double tap to a single tap.
14. Added Delete Button to Enlarged Saved Image From the interactive prototype feedback, we realized that there was no delete option when a user was viewing an enlarged version of the saved picture in Album view, so we added one.


Method

Participants

Our participants were children ages 3-7 and their parents, our target demographics; children did the letter tracing and drawing/coloring tasks, while parents went through the changing settings, and album viewing tasks. Although both groups used the album viewing, it became clear that it was more of a feature for the parents and much less of interest to the children. Due to the difficulty in finding users to test our app - finding people who would agree to let their children participate in a test of an application was difficult enough for our lo-fi prototype, and we had completely depleted the small pool of testers we had found - our participants were families visiting Berkeley during Cal Day, who we offered to demonstrate some of our work in the department to. To compensate for this, we aimed for broad coverage of the age group (for children) and varied experience using the iPhone (for both groups).

  • 1. ~6.5-year-old male, with some iPhone experience; parent noted that the child really likes to play with the iPhone
  • 2. Father of participant 1; ownes an iPhone
  • 3. ~3.5-year-old male, with no prior iPhone experience (but some use of a laptop)
  • 4. Mother of participant 3, with no prior iPhone experience, but does have experience using a computer/laptop
  • 5. ~5-year-old male, experienced with iPhone and familiar with its use
  • 6. Female guardian of participant 5, similarly experienced as her child

[We don't have that much information on all the users, since we found them during Cal Day.]

Apparatus

We used two iPhones, one for use of the app itself and a second running as a stopwatch with laps to mark error times and track rates; we used a table in the student lounge on the second floor of Cory Hall, recruiting participants near the doors to the building as they entered or were leaving. Notes were recorded on laptops by two members of the group, while another supervised the user and the last handled consent forms, timing for some participants, and similar incidental tasks.

Tasks

Three of the four tasks we chose to focus on are the same three we've focused on from the start - tracing a letter, drawing, coloring, and saving a picture, and viewing the album.. We believe these three tasks reflect the main functions of our application (tracing and coloring) and also cover some of the interesting features (for example deleting and saving a picture). The tasks are appropriately difficult and complex, ranging from simply viewing the album to coloring and saving a connect-the-dots picture. Furthermore the three tasks are cohesive and form a compelling story of why the user might want to use the application (children can trace and color letters and pictures in order to entertain themselves). The fourth task, editing settings, was chosen to demonstrate a more standard iPhone interface with the intent of providing familiar controls for the parent; while the album is also a zone primarily for the parent, unlike the settings panel it will likely be accessed by children as well, so the settings view is the best means of demonstrating the two design approaches required for our disjoint audience.


Picture Task Description
Easy - Changing Settings
  • This task is primarily for the parents. From the main screen the user can click on the gears icon, an iPhone standard image for settings. There, they will find several options. They can randomize the order of the letter-tracing cards, disable the audio instructions, and enable stroke-order arrows to help show the children how to write a letter.
Easy - Viewing and Accessing the Album
  • From the main screen the user can click on the "Album" button (with a clearer icon for children) to access the album page. From there the user can look at saved images and select one to view by simply tapping on it. Images can be deleted by dragging to the trash can icon, or from the full-sized view of the image.
  • This task is analogous to the user saving the child's coloring book or workbook pages; the child's work can be stored for later use and viewing.
Medium - Tracing a Letter
  • From the main menu, the user can select "abc", "ABC", or "123" to get to the letter/number tracing screen. Each button goes to fundamentally the same task, but with a different set of characters to trace. From the tracing screen the user can then select a color of their choice (the default is set to black) by tapping on a color pencil and then trace the letter. Navigation buttons with arrow icons allow moving between letters; this replaced swiping as noted earlier. If the stroke direction setting has been enabled, an overlay showing each stroke will appear as well. When finished, users can tap a checkmark (as explained in the instructions that play on loading and on selecting the audio button) to indicate they have finished, after which their tracing will be evaluated and appropriate feedback will be returned.
  • This is one of the main focuses of our application (the other being coloring and tracing a picture), where the child can trace letters in order to practice both recognizing and writing characters.
Hard - Connect the Dots, Coloring, and Saving
  • From the letter/number tracing screen, the user can access the canvas/drawing page by simply tapping on the picture near the bottom of the screen. Then a related connect-the-dots image will load and the user can trace and color the picture.
  • When the user is done coloring, they can click on the save button (the floppy disk) and save their lovely picture to the album. However clicking save multiple times while not making any changes to the canvas page will not result in multiple saves.
  • This function is the second main focus of our application. It helps the child associate letters to their corresponding picture, and keep the child entertained.

Procedure

Our group set up the experiment in the lounge on the second floor of Cory Hall on Cal Day. We had two people taking notes on their laptops, a greeter/facilitator, and a timer.

We waited in the lounge while people walked by outside in the hallways, and when we saw a family that fit our user group, the greeter would approach them and ask if they would help out in a pilot study for an iPhone development class. We then had all the users sign consent forms (the parent would sign for their child). The consent forms can be viewed in the Appendix below.

When the forms had all been signed, we sat the participants down and explained what the different tasks were. For the children we wanted them to test the tracing and canvas views (medium and hard task). We wanted parents to explore settings and be able to change some of the settings and see the affect it had on the application (easy task). We wanted both user groups to do the album viewing task (easy task).

After explaining their tasks, we gave a quick demo of the app (showing where the Home buttons lead, which reveals very little about how to perform any of the four tasks we were testing). We then told the users that we wanted them to "think out loud" while they were using our app. Then the timer started the stop watch and the greeter/facilitator stood by and answered questions when the user was stuck. The other two members of the group took notes and wrote down the error times.

When they were done with their tasks, we asked the users how they liked the app, and if was suitable for children. We also asked if there were any other comments they wanted to make about our app.

Afterward, we thanked the users for their time and help, and then jolted down some final notes as they left.

The script for the pilot study can be viewed in the Appendix.


Test Measures

The goal of presenting a new interface to a user is to promote recognizable features and functions in order to limit the amount of time searching for the correct method to perform a task. In order to provide such an interface, we selected testing measures focused on time-till-completion and errors committed in order to determine the efficiency and effectiveness of our user interface.

Task Test Measure/Why
1. Easy - Viewing and Accessing the Album

Viewing and accessing the album is considered the easiest task since it requires only one action from the home screen. Therefore, our measures for this task recorded the amount of time it took the user to figure out how to access the album as well as the number of errors committed trying to access the album. Participants observed for this test were the parents of the child users.


Test Measures

  • Time to Access the Album
    • This measure focused on how long the user scanned the screen before selecting the leaf labeled "Album." The main goal of this test was to measure the affordance of the home menu as well as the accuracy of our icon selection.
  • Time to Delete a Saved Drawing
    • This measure focused on the users understanding of the album view and the recognition of the available functions within the view: viewing and deleting a saved drawing. Currently, viewing a picture is completed by single-tapping a thumbnail and deleting requires the user to drag the desired image to the trash or tapping the trashcan icon from within the picture viewing mode.
  • Errors Committed
    • This measure calculated the amount of wrong selections performed by the user when asked to view the album or when asked to delete a picture. Similar to tracking the elapsed time, the goal of counting committed errors is to help identify which functions may need more visual prompting or may need better adoption of current user expectations for viewing and deleting images.
2. Easy - Changing Settings

Accessing the settings menu is considered an easy task since it requires only one action from the main menu: touch the sprockets in the upper right-hand corner of the screen. Changing the settings is also quite simple since all the options are configured as on/off switches. Therefore, our measures for this task focused on the user's understanding of the home menu as well as the functions available in the settings menu.


Test Measures

  • Time to Access the Settings Menu
    • This measure calculated the amount of time it took the user to scan the home screen and find the icon representing the settings menu. The goal of this measure was to help determine the selection quality and placement of the settings icon as well as the overall layout of the home menu.
  • Errors Committed
    • This measure focused on the amount of errors committed trying to figure out which button corresponded to the settings menu. By counting the errors, we were able to determine the quality of our icon selection and placement as well as the level of ease for the setting's switch manipulation.
3. Medium - Tracing a Letter

Tracing a letter/number is considered a medium difficulty task since it requires more user interaction. First, a user must select one of three leaves representing a tracing activity, then, must carefully trace within the letter/number template by following the correct stroke directions presented in the view. Therefore, the testing measures chosen for this task focus on the user's ability to select the correct task option and then perform the desired task.


Test Measures

  • Time to Access a Tracing Activity
    • This measurement calculated the amount of time it took a user to correctly identify a leaf-button from the home menu that corresponded to a tracing activity. The goal of this measure was to help determine the appropriateness of the activity icons in relation to the user's age.
  • Errors Committed
    • This measurement focused on the amount of errors committed by the user after the instruction to trace a letter was given. Similar to tracking the time, this measurement helps determine the appropriateness of the icons as well as the simplicity of the interface layout for first-time users.
4. Hard - Connect the Dots, Coloring, and Saving

Connect-the-dots, coloring, and saving is considered a hard task since it requires the most amount of user recognition of images representing tasks and interactions with the user interface. In order to reach the coloring canvas, a user must first enter a letter tracing activity by selecting the correct leaf button in the home menu, then, he/she must single-tap the image directly below the letter. Once at the coloring canvas, a connect-the-dots drawing can be colored using different paintbrush colors and tip sizes. Lastly, to save the colored picture, the user must identify the disk icon in the tool bar at the top of the screen as the saving function and then press it. Acknowledging the following alert completes the task and saves the picture to the album. Because this task is a multi-step task, test measurements were used through each step to gauge the ability of the user to recognize icon images and correctly respond to the interface prompts.


Test Measures

  • Time to Access a Tracing Activity
    • This measurement calculated the amount of time it took a user to correctly identify a leaf-button from the home menu that corresponded to a tracing activity. The goal of this measure was to help determine the appropriateness of the activity icons in relation to the user's age.
  • Time to Access Coloring Canvas
    • This measurement calculated the amount of time it took the user to access the coloring canvas from the letter tracing view. The goal of this measure was to determine the level of affordance provided by the image in helping the user recognize that it was selectable.
  • Time to Save the Drawing
    • This measurement calculated the amount of time it took the user to create a relationship between the saving function and the disk icon at the top of the screen. This measure helps determine the appropriateness of the icon selection as well as the user's ability to respond to an audible prompt.
  • Errors Committed
    • This measurement focused on the amount of errors committed through the entire task completion. We understand that this is a multi-step task, but by calculating the total amount of errors we are able to determine the accessibility of the coloring activity as well as the level of difficulty performing such tasks like saving may be for our target user group.


Results and Discussion

  • Since we had 6 users, we decided to summarize the results by tasks.

Results

Easy - Viewing and Accessing the Album

User Time to Access Album Time to Delete Saved Drawing Errors Committed
Parent 1 2 sec 4 sec 0
Child 1 13 sec N/A 3
Parent 2 2 sec 5 sec 1
Child 2 15 sec N/A 6
Parent 3 2 sec 3 sec 0
Child 3 6 sec N/A 3

Discussion

  • One of the things we noted in our pilot study is that none of the parents realized they could drag a thumbnail picture to the trash can to delete. Instead most of them went to the enlarged thumbnail view to delete. We are planning on fixing this problem in our application by making the Album view resemble the iPhone photo album so there will be less confusion. We also got feedback from Parent 3 that there should be a "Do you want to delete?" prompt in the enlarged thumbnail view when the user presses the trash can icon.
  • From our pilot study we realized that parents will be the primary user of the Album page. When the children accessed the Album they were not interested in what they saw and quickly wanted to leave the page, opting to play with letter tracing and canvas coloring. As a result of this, our group is planning on making the Album page more parent friendly.
  • All the parents had no problems getting to the Album page. They knew that tapping on a picture would lead to an enlarged thumbnail, and they also had no issues with deleting a picture. However like noted above, none of them realized that you could drag a thumbnail picture to the trash icon to delete.
  • Parent 3 also commented that she would like more visual cues to show which picture had been deleted.
  • When the children accessed the Album view, they had very little interest in it. Child 2 viewed an enlarged thumbnail picture and tried to draw on it like the canvas view. None of the children decided they wanted to delete any pictures from the Album.

Easy - Changing Settings

User Time to Access Settings Errors Committed
Parent 1 18 sec 2
Parent 2 22 sec 4
Parent 3 15 sec 1

Discussion

  • Before our group had played with the idea of making the settings page password protected so only the parents are able to access it. However feedback from the users in the pilot study revealed that a password was not necessary for the settings page. All three parents thought that it would be too much of a hassle, and didn't really mind if their child could change the settings of the app. However from the Interactive Prototype we have changed the overall design of the settings page to reflect that the parents would be the main users of the page.
  • All three parents had a hard time locating the settings icon on the Home menu. Parent 1, who owned an iPhone had difficulty recognizing the gears icon as the link to settings. He thought the info button was settings. Parent 2, who was unfamiliar with the iPhone had to access a few other pages first from the Home page before she found the settings page. Parent 3, who had extensive iPhone experience did not realize that the gear icon led to settings either. She only figured out that the icon led to setting since nothing else on the Home menu could possibly lead there.
  • All three parents found it pretty easy to change the different settings on the page. However when asked what they though each settings does, none of them had a clear idea. Parent 1 decided to answer the question by changing a few settings and then playing with the app to see what had changed, and from this experience he then had a clear idea of what all the settings did. Parent 2 was confused for awhile, but after thinking about it, she realized what the random cards and stroke overlay settings did. Parent 3 commented that it might be useful to add a description of what each setting does when the user taps on the different settings. Our group decided that adding a description of each setting was an excellent idea and would really help in clearing up the confusion all three parents experienced.

Medium - Tracing a Letter

User Time to Complete Tracing Activity Errors Committed
Child 1 1 min 47 sec 12
Child 2 2 min 24 sec 14
Child 3 1 min 15 sec 7

Discussion

  • All three children really liked the tracing task. They were engaged by the application, and it seemed like they liked the tracing more than the canvas drawing.
  • None of the children tried to use the Undo function. On of the reasons our group believes this happened is because the arrow undo icon is unclear. As a result, we have changed the arrow icon to an eraser icon so children will better recognize the icon.
  • Child 1 didn't really understand the tracing concept when he first entered the tracing page, but after his dad showed him what to do he really wanted to do the tracing by himself. He went to the next letter correctly by pressing the forward button near the bottom of the page, and was able to trace the next letter all by himself. He seemed to really enjoy the tracing aspect of the application.
  • Child 2 was very young and near the bottom of our user group range. He also did not have any experience using the iPhone OS so the tracing task was pretty difficult for him. He didn't understand the task, but after some help from the greeter and his mother he was able to explore the application and find the tracing view. After his second attempt to trace the letter, he knew how to follow the stroke overlay. The child didn't realize that the arrow buttons near the bottom of the page would switch the tracing view for him, but after his mother pressed the arrow to demonstrate, he understood the function. Furthermore, Child 2 had issues picking a different color to trace in, because the color pencil icons were too small and close together.
  • Child 3 had the most experience using iPhones. His mother commented that he loved playing with her iPhone. The child didn't have a problem accessing the tracing page, but he did have problems trying to pick different colors to trace in. Again the fat finger problem occurred, the color pencil icons were too small and closely spaced that it was hard for Child 3 to correctly choose the color he desired. Furthermore, as he was tracing the letter, his hand kept on accidentally touching the green check mark that signifies the user is finished tracing. To solve this problem our group is going to move the green check mark to the other side of the tracing view, to keep children from accidentally pressing the button when they are using the app.

Hard - Connect the Dots, Coloring, and Saving

User Time to Access Tracing Activity Time to Access Coloring Canvas Time to Save Drawing Errors Committed
Child 1 12 sec 1 min 38 sec 24 sec 11
Child 2 16 sec 1 min 49 sec Did not finish this task 13
Child 3 5 sec 1 min 29 sec 11 sec 7

Discussion

  • One of the major problems in our app is that pressing the apple picture to go to the canvas view is not obvious. Our group is planning on placing a prompt on the tracing page so users will know to press the apple picture to enter the canvas view.
  • The children also had problems tapping on buttons on this view (fat finger problem). The menu buttons were too small for them.
  • Child 1 didn't know how to save the picture he drew on the canvas. However after his dad showed him how to do it the first time, he became very good at saving his pictures.
  • Since Child 2 was so young, he had a hard time understanding what to do in the canvas view. He tried to draw across the entire screen and decided to not save his picture.
  • Child 3 wasn't interested in connecting the dots on the canvas view, instead he just tried to color on the screen. Furthermore, when asked how he might clear the canvas screen, he thought that pressing the line widths buttons would clear the view. Our group thought by increasing the visual cues/feedback when the user pressed the line widths button, that might solve this problem of not understanding what line widths did.

Discussion Of Overall Application

  • Conclusions: Conducting the pilot study was very useful in understanding the problems that still remain in our app. One of the major problems we discovered in the pilot study was the fat finger problem. In our lo-fi prototype we never experienced this problem, however during this experiment, many of the users kept having a hard time pressing the top menu buttons. Additionally, we found another major problem, which is that pressing the apple/other picture on the tracing page to go to the canvas view is not obvious to the user. We also had some confusion with the icons we were using, such as the setting and undo icon. Furthermore, the album page can use more work to make it parent friendly.
  • Despite all these areas that need improvement, all of our pilot study users really liked the app and thought it was a great idea. Furthermore, Parent 2 even used the app as a break time for herself. She had two other children with her and while her oldest son was playing with the app, she took a break from having to entertain all the kids at the same time, which is the very problem our app tackles (helping busy parents while also teaching the child).

Changes Already Made From Pilot Feedback

  • Since we got such great feedback from our users, we decided to start implementing some of the changes:
    • Make the icons in the menu bar larger. Many users had the fat finger problem where they were not able to tap the desired button correctly.
    • Space out the color pencil palette more, some children had problems tapping on just one color pencil at a time (fat finger problem).
    • Move the Done check mark in the tracing view to the opposite side. During the pilot study children would accidentally press the green check mark while they were tracing letters/numbers. By moving it to the other side, we can prevent this error from happening
    • Change the Undo icon to something more recognizable (we changed from an arrow to an eraser icon).

Future Changes

  • There are also many changes that we noted from the pilot study that has not been implemented yet:
    • Make the Album view more parent friendly, since they will be the main users for the page.
    • Get rid of the drag to trash in the Album view and make the Album function more like the iPhone photo album.
    • Have the app prompt the user to delete when deleting from the enlarged thumbnail view.
    • In settings, have a short description of what each setting does.
    • The audio that plays for each letter is still too long, need to shorten.
    • Many parents had a hard time recognizing that the gears icon was suppose to lead to settings. The way they figured it out was that since it wasn't the info button, then settings had to be the other button. We can make it easier for the parent to recognize settings by changing the icon (for example switching to the hammer and wrench icon).
    • The canvas view was not obvious for the child to find. They need prompting to realize that by tapping on the apple or corresponding picture, it would lead them to the drawing task. We can make how to access the canvas view more obvious to the user by putting some kind of prompt on the screen next to the picture.
    • Left handed, right handed settings option. For example if the child is left handed, then we will need to move the Done check mark in the tracing view to the other side of the screen to keep the child from accidentally hitting it when they are tracing a letter/number.

Appendices

Materials

Raw data

Consent forms

References



[add comment]
Personal tools