Pilot Usability Study:GroupYourLifeMaxPreston

From Cs160-sp08

Jump to: navigation, search

Introduction

Our System

Our application is a mobile calendar and daily planner. Our system is designed to make it easier for busy people like college students to schedule and view their daily activities while they are on the go. Our program facilitates three general kinds of tasks: fixed tasks, recurring tasks, and flexible tasks. Fixed tasks occur once at a set time for a set duration, like a party or a group meeting. Recurring tasks are like fixed tasks, except that they repeat on certain days of the week. For example, going to our CS160 lecture would be a recurring task. Flexible tasks have a deadline and a set duration, but there is no designated time to complete them. For example, a homework assignment would be a flexible task. Our application examines the user's current schedule to find a large enough empty time slot to schedule a flex task.

The purpose of the experiment

We have already completed a working prototype for our application, and we have since made multiple improvements to it. The purpose of this experiment is to find a person outside the class as an initial subject to test our systems without any prior bias. The target subject is any college student, as that is the group our product is designed for. By observing the actions of the subject and gaining feedback from them, I hope to use this data to find bugs and design flaws in our application, as well as other ideas for improvement. Having an independent test subject is important because we will be able to find problems that we had not necessarily noticed or thought were issues due to our excessive experience designing and constructing our application.

Implementation and Improvements

We have implemented a significant amount of functionality to our project since presenting our interactive prototype:

Total redesign of the calendar screen
- The previous implementation of the calendar screen had many flaws. The greatest flaw was that you had to use the menu to navigate to different weeks and times, which was extremely slow and irritating. With the new version, using the arrow keys allows you to smoothly travel to different weeks and times with the arrow keys alone by navigating off the edge of the screen. This is much faster, more intuitive, and easier to use.
- Another issue was that time slots that had activities in them were displayed as buttons with an unattractive-looking blue rectangle in the middle. There was no information about the task when you clicked on it, and it was impossible to tell consecutive tasks apart without navigating to them and viewing the task details. In the new version of the calendar, the first few characters of the task's title are viewable in their corresponding time slots on the calendar screen. This makes it much easier to determine which task is which. It also shows when multiple tasks are scheduled in the same time slot.
Addition of the list view
- The list view is a screen that can be used to view tasks for a specific day. Rather than being displayed as a calendar, it is simply an easily navigable list of the tasks in the day, which eliminates the need for navigation across unfilled time segments. This just gives another option to the user depending on their needs.
Implementation of flex task allocation algorithm
- Flex tasks were previously not being allocated at all due to database and logic issues. The algorithm now allocates flex tasks, although it is not yet very intelligent.
Major rehaul of the add fixed, flex, and recurring task screens
- Due to concerns with the usability of these three screens, I redesigned all three of them to be more easily navigable and understandable. I repositioned most of the elements, increased the size of the text boxes, modified text size, and implemented time and date pickers, as well as text boxes (rather than spinners) to save time for the user. I also implemented displays that say the currently selected date and time. Since text boxes and time pickers are used, this also allows the user to set events with durations and at times that aren't necessarily multiples of 30 minutes. This gives more options to the user without complicating things.
The ability to add tasks at a certain time by clicking on an empty time slot on the calendar
- When you click on an empty spot on the calendar, it instantly transports you to the fixed task screen with the date and time information already filled in. This saves time for the user.
Misc
- Other features such as task editing and an alarm system were worked on extensively, although they are not yet complete.

Method

Participant

Since I don't have a laptop, and since desktops are incredibly non-portable, and since it would be extremely infeasible to ask a total stranger to come all the way to my house to participate in a "pilot usability study," I was forced to ask one of my housemates to participate in our study. The subject is a fourth year History major with low technical experience and no prior knowledge about this project. In regards to expertise, he described himself as only really knowing how to use word and the internet, and being able to do things like change his screensaver. He has a basic cell phone (no palm pilot, blackberry, iphone, etc). Since he is my friend and there is a possibility of experimental error due to this, I was extremely blunt in telling him that this was for a study, rather than to show off the project, and that he should thus be entirely honest in what he says so that we can use his feedback for improvements. From my observations, he was unbiased and said what was on his mind, rather than saying things to please me.

Apparatus

The experiment was conducted in my room, at my computer, with two chairs in front of it and the emulator open on my computer's desktop. There was no outside interference- no one else entered the room, no cellphones went off, and nothing else of that sort occurred. The available inputs were a standard keyboard and mouse in front of the monitor on my desk. The room was brightly lit and there were no visibility issues. Due to a lack of recording equipment (I don't have access to a tape recorder or video camera), I simply took detailed notes with a pencil on a piece of blank printer paper. I left my watch on my desk so that I could note the time different events occurred.

Tasks

Our group chose the three tasks, although each of us could choose the details however we wanted to:

Easy-difficulty Task: Navigate to and delete a task on the calendar screen
- Before the subject had even entered my room, I had already set up the specified task on the current screen of the calendar. The goal of this task was to test whether or not basic calendar navigation is intuitive to a new user. I checked to see whether the subject had trouble finding the task, navigating to it, selecting it, or deleting it. This is important since we recently rehauled the entire calendar interface.

Medium-difficulty Task: Add a flexible task
- I told the subject that he has a homework assignment due on Friday at 3:00pm which would take one hour to complete, and that he should add this flex task. Testing flex tasks is really important for several reasons. Primarily, the allocation algorithm has only been recently implemented, and it has not been sufficiently tested or tweaked yet. Thus, we need to determine how easy it is for new users, as well as any other issues relating to flex tasks in general, since it is very likely that problems could come up. Also, since I revamped the "add flexible task" screen, it would be useful to see how intuitive it is for a first-time user. Things to observe were whether the subject had difficulties opening up the flex task screen, whether he had trouble or made mistakes filling out the task info, and whether he could find it after scheduling it.

Hard-difficulty Task: Add a recurring fixed task
- I told the subject to schedule his History class (since he's a History major), which meets on Mondays, Wednesdays, and Fridays at 3pm for an hour, and the last day of the class is May 20th. Reasons for testing recurring fixed tasks is because adding them is more complex than adding normal tasks, and because I revamped both the "add fixed task" and "make task recurring?" screens. It would be useful to see whether they are less confusing than before, as well as any difficulties a new user might have entering values and adding the task. Things I took note of were whether the subject had difficulties opening up the fixed task screen, whether he had trouble filling out the task info on both screens, and whether he had trouble finding it after scheduling it.

Procedure

1. I convinced the subject to participate in the experiment and told him to meet me at my room when he is ready.
2. I set up the emulator on my desktop with the fixed task for the first task.
3. I had him read and sign off on the consent form.
4. At this point, I started taking notes on what was going on, including the time at the beginning of each task. I explained the context of the experiment, the functions and features of our application, what the emulator was, what all the buttons were for and how to operate it.
5. I very quickly showed him basic navigation of the interface by moving around the calendar, opening the menu, and briefly showing the different task screens, without explaining how they work.
6. I then gave him permission to use the mouse and keyboard, and told him the directions for the easy-difficulty task.
7. I took notes and observed him complete the task, only helping him out if he had a question (such as forgetting the task title), or if he became completely stuck and couldn't figure out how to proceed.
8. I followed the same procedure for the medium-difficulty and hard-difficulty task.
9. I then asked the subject if he had any comments for the application or ideas for improving it. I asked him to elaborate on his responses as necessary, and asked if he had any other issues, specifically naming various aspects of our application.
10. After gaining as much information and feedback as possible, I thanked the subject and ended the experiment.

Test Measures

I took note of definite errors that the subject made while completing each task. These are important to review to see if an aspect of the interface is misleading or confusing, so that we can make changes which would reduce the odds of an average user making this kind of error in the future. The severity of this kind of error can vary based on how much it is the user's fault and how much it's the application's fault.
I took note of errors that occurred which could not be attributed to the subject's behavior. For example, if something nonsensical happens which baffles me as well as the test subject, then that would be this kind of error. This kind of error actually occurred when the subject had no idea where the flex task was after adding it. This kind of error would be a critical flaw in our application that mandates fixing before our project can be considered complete.
I took note of feedback and comments the subject made during and after the study. Feedback from an outside participant like this is extremely useful since they their lack of experience with the interface can make them notice things that we might not otherwise consider.
I measured the amount of time it took to complete each task. This was to determine if a task takes an extremely long time to complete, in which case our interface would need to be changed to either be more clear or to reduce the number of actions necessary for completing a given task.

Results

Errors/Problems

Easy-difficulty Task: Navigating to and deleting a task on the calendar screen
- 1. The subject didn't realize that he had to hold the button down for a moment to click on a task. He tried to click on it quickly several times without anything happening before I had to intervene and tell him that he had to hold it down for a short amount of time. Once he learned this, he never made this error again.

Medium-difficulty Task: Adding a flexible task
- 2. The subject didn't realize that he had to go to the menu to add the task. Instead he navigated to the deadline time slot and and pressed enter, which went to the fixed task screen. He then started filling it out without instantly realizing that he was on the wrong screen. I had to tell him that he was actually adding a fixed task and that he had to use the menu to add a flex task.
- 3. After canceling out of the fixed task screen, he did the same thing again by mistake before canceling again and pressing the menu button.
- 4. When the flex task was added, neither of us had any idea where it was. He then navigated to the deadline and was confused as to why nothing was there. After navigating the calendar randomly, he eventually found the task scheduled for 5:30 am.

Hard-difficulty Task: Adding a recurring fixed task
- 5. On the recurring tasks screen, he didn't realize that he had to press the "recur until" button to set the end date.

Time taken

Easy-difficulty task: ~2 minutes
Medium-difficulty task: ~7 minutes
Hard-difficulty task: ~2 minutes

Note that tasks may have taken slightly longer than normal when he stopped to give me feedback during the different tasks. Additionally, since I had to stop talking and jot down notes during each task, they may have taken a little longer than they would have otherwise.

Feedback

(During the medium-difficulty task)
- The subject was a little confused about what a flex task was, as he thought that it was supposed to be allocated on on the deadline rather than needing to be done earlier. He said that the task should have been scheduled at a more reasonable time than 5:30 am. Also, since nothing appears on where the task's deadline is, the deadline isn't really clear. However, he also mentioned that navigation is really easy and it's really straightforward to use.

(After completing the hard-difficulty task)
- He said that overall the application is pretty straightforward and he likes it. He currently prefers to schedule things with paper, but if he didn't have a daily planner, he would use our application as his organizer. He said that the biggest thing that he thought needed to be changed with our application is that flex tasks should show the deadline on the calendar. He said that this was clearly a bug. Another thing that he said was really important is that there should be a grid on the calendar so that it looks more like a normal calendar, and it would be more organized that way. Lastly, it would be really nice if there was more than one color for different days, and different colors for different tasks and days, so that things would stand out more. Right now it's just like reading, and it would be really helpful if there were visual cues to separate the different days, tasks, and hours.

Discussion

This study was actually quite valuable. I think that nearly all the errors in the experiment could be avoided by improving our interface. Likewise, I believe that all of the feedback for improvement that was given is both valid and important.

In regards to things we could change for the "real" experiment, I would definitely want to use a video camera to record both what the participants are both saying and doing. If we used one, we could replay the study over and over to make sure we picked up on every detail of what happened, as well as for getting far more accurate time measurements. Also, I would want to conduct the study with completely neutral participants in a completely neutral location. Although I think that my subject was unbiased, it is best to reduce factors that could create experimental error as much as possible if we want scientific results, especially if we are working with multiple subjects at different times.

In any case, there were several major issues that came up which definitely need to be resolved. The biggest problem is flex tasks. This is immediately apparent, as the most errors occurred with this task, and it took about 3 times as long for the subject to complete this task as it took for him to complete the other tasks. The subject even felt as though they were buggy. Flex tasks are unintuitive, there are difficulties with creating them, and they don't give enough feedback or choices to the user. To solve the lack of visibility, the task's deadline should be displayed on the calendar screen, the task should be allocated more intelligently, and there could be a display upon creation that says when the task has been allocated, allowing the user to change the time if necessary. Also, when you click on an empty calendar square, instead of instantly going to the fixed tasks screen, a popup should appear that allows you to choose between creating a fixed task and a flex task.

Another thing the participant found to be very important was the aesthetics of the calendar. Since it's just a grid right now, it's true that it can take some time to interpret where things are on the screen. Adding grid lines would make it easier to find the date and time of a given section and to differentiate between spaces in the calendar. Also, by coloring grids based on the task type, it would be more clear which tasks are which and what they're for. We could also give different tones to the calendar based on the time of day, month, or day number, so that it would be easier to get a general feel of where you are in the day or month. Another idea is to change the color of the grid lines based on these factors. The colors could also be set based on the distance from the current time, so that it will be easier to see where you are in comparison to the current time.

One smaller issue is that you have to hold down the button for a little bit to click on a time slot in the calendar. We believe that this is a bug in android, so we can't do anything about it, but we can still look into it.

However, despite these problems, I think the overall results were very positive since the subject liked our application and was able to figure it out fairly quickly despite having low technical experience.