PilotStudy-Group:Team42
From CS160 User Interfaces Fa06
Contents |
Introduction
We are evaluating a system that combines a digital pen, paper-based daily planner, and web interface. The digital pen is used to schedule events in the daily planner, from which events are uploaded directly to a computer and then visualized on the web. Users may then use an Internet browser to access any events and tasks s/he has recorded in the planner. The user may also use the website to take advantage of other features, such as the sending of automated email reminders.
The purpose of the experiment is to gather feedback from testers on a digital and physical version of our prototype. Our previous evaluation round involved the use of a low-fi prototype. Based on comments from the testers in that round, we made changes to our interface and implemented them in this iteration. We intend to gather further feedback from testers on the usability and effectiveness of our new interface.
Implementation and Improvements
Since we submitted our interactive prototype, we have continued to improve and further implement aspects of our prototype. Most of our changes were made to the server interaction between the Swing component and the web interface, as we had to resort to many Wizard of Oz techniques there for our last iteration. For the specific functionality of our current prototype, we have:
- Added pen support for multiple weeks
- Previously we were able to navigate between weeks on the web interface, but for some reason, the data collected by the Anoto pen remained the same for every week. In this iteration, we fixed the problem so that now every week online reflects its respective week in the paper planner.
- Fixed alignment issues in the paper interface
- Previously our printouts for the daily planner were .8" too high, and therefore partially cut off. In this iteration, we inverted our printout so that nothing was cut off because of the printer margins. We also made adjustments to the dimensions of data collection so that the paper and web interfaces matched up more closely.
- Created daily views in the web interface
- Previously we only had weekly views, which meant that the text uploaded by the Anoto pen was very small and difficult to read on a typical computer screen. In this iteration, we created individual pages for each day of the week, so that when a user clicks on a day inside the weekly view, s/he will be taken to an enlarged view of that day.
- Added support for multi-hour events
- Previously we were only able to hard code the web visualization of multi-hour events by manually defining style classes for colored strips by the hourly time slots. In this iteration, we worked on taking the "hours boxed" data collected by the Anoto pen and translating it to create the appropriate style classes. We also built an array of colors that the system could loop through so that consecutive multi-hour events will not blend together.
- Added partial support for setting individual reminders
- Previously we had only implemented the Customize Reminders tab, which allows users to change the default reminder settings for all reminders checked off in the paper interface. In this iteration, we decided to use AJAX to add support for changing individual reminder settings on the daily views of the web interface. Each event now has a short description next to it - "No reminder set" or "Reminder set to X units in advance". Clicking each short description will replace it with a small, instantly savable form, where the user can decide if and how far in advance they want a reminder to be sent for that event alone.
Method
Participants
We recruited participants by asking our friends to refer us to people they knew who typically used paper-based daily planners. In this way, we were able to get objective comments from participants in our target user group, as well as study how easy it was for them to make the transition between a typical daily planner and our digital daily planner, which combines the former with a web interface. All of our participants were students at UC Berkeley.
KS is an undergraduate senior in the mechanical engineering department. He calls himself a "paper and pen kind of guy" and does not use PDAs or digital scheduling applications.
JZ is a graduate student at the Goldman School for Public Policy. He is 25 years old. He schedules his time primarily with a paper-based planner.
TL is a 5th year undergraduate Statistics major. He is 22 years old, and primarily organizes his time with a paper planner due to its portability and better interface compared to something like a PDA.
Apparatus
Our original plan was to set up the R3 toolkit and a server on a laptop so that our experiment could be portable and allow us to travel to locations that were convenient for our participants. However, due to technical difficulties, we carried out all of our tests in a computer lab instead. We positioned each participant in front of a desktop computer, placed the paper planner on the desk in front, and sat ourselves to the sides of the participant. Each participant was free to use the digital pen and/or computer mouse at any time during the pilot study.
Tasks
Scheduling an event
The user is given the task of scheduling an event in the near future. To do so, the user must write down the task in the daily planner using the Anoto digital pen. The user will then be asked to find the event on the web interface. We will look at the level of difficulty of the user's transition between the paper and web interfaces, and note the ease of use with which the user is able to navigate between weeks.
Indicating a multi-hour event
The user is given the task of scheduling a multi-hour event. The user will be given an event to specify in their daily planner and will be told that it lasts more than one hour. The user should write the event in the planner, draw a box around the corresponding hours, and then find the event on the web interface. We will look for the user's understanding of the "boxing" concept and check that s/he is able to understand the multi-hour representation on the web interface.
Getting a reminder about an event
The user will be told to set up an appointment and arrange to receive an email reminder for that appointment. The user should check the box that triggers the sending of a reminder, and then enter his or her email address under the Customize Reminders tab of the web interface. We will want to observe whether or not the user also clicks on daily views and attempts to customize individual reminder settings there.
Procedure
Upon meeting a tester, we introduced ourselves, explaining that we were working with testers as part of a course in user interface design. We explained that we decided to design a system involving a daily planner, computer, website, and digital pen. We noted that the purpose of the testing session was to collect as much input as possible from the tester about the usability and effectiveness of our design.
We then outlined in broad terms how the daily planner, Anoto digital pen, computer and website interact with one another. We asked the tester to sign the consent form, and asked them to be vocal in expressing their reactions to the interface.
Three observers took notes and observed the tester. The tester was seated in a chair in front of the computer with the website. The planner and digital pen were placed in front of him or her. The facilitator sat to one side in order to ask and answer questions.
The facilitator began the testing session by running through the demo script (see Appendix below.) The facilitator picked up the pen, introduced the daily planner and gave an example of how the website could be used. He also pointed out an instruction sheet that explained how to access some of the application’s features.
We then handed a task description to the tester (see Appendix.) The tasks include setting up a single-hour event, setting up a multi-hour event, and arranging to be reminded by email about an event. We would tell the tester to try his or her best to complete the task written on the paper. While the tester was working on the task, we would occasionally ask the tester open-ended questions to explore his or her thought process. The three observers would time the task, count the number of errors made, and take note of both positive and negative critical incidents. The observers took notes by hand while tracking the time on watches and/or laptops.
We repeated the above process until all three tasks were completed. At the end of the process, we asked some follow-up questions regarding their thoughts on various aspects of the site, planner or tasks, and then concluded our testing session by thanking the tester for his or her time.
Test Measures
- What critical incidents occurred and where did they occur?
- We wanted to identify events that our participants were either confused by or pleased by. Pinpointing negative events helped us to notice flaws in our design, while making note of positive events helped us to understand what we had done right. In both cases, this test measure served to expand our understanding of what comes naturally to our target user group and what does not.
- How long it did users to complete each task?
- While this is not the best measure to take for think-aloud sessions, we felt that for the most part, shorter completion times meant that our participants understood what was being asked of them and could easily see how to execute the task. Because our tasks covered basically all the functions of our interface, if participants were able to complete them efficiently, it was a good indication that our design was working.
- How many errors did users make in completing each task?
- By errors, we mean such things as making the wrong clicks, taking unnecessary steps, or having to backtrack. We wanted to keep track of where our participants made errors, what they were thinking as they made them, and how they reacted when the system failed to respond in the way they had expected. This error log helped us to identify problem areas in our design.
Results
Task 1 (Easy) Scheduling an event
When first getting acclimated to the system, almost all the users complained that the images of their writing on the webpage were too small to read. Some participants also hoped for OCR, so that they could read the text clearly. They were slightly dismayed to only see their own handwriting. A side note on the pen itself, JZ thought the pen was too big, but still useable. The system still had some minor navigational issues.
| Task 1 | |
|---|---|
| Participant | Task duration |
| 1 | 0:06 |
| 2 | 0:01 |
| 3 | 0:01 |
| Mean | 0:02 |
| Standard Devation | 0:02:21 |
Task 2 (Medium) Indicating a multi-hour event
This was not as intuitive, because most users are used to boxing the event. In the case of JZ, he drew an arrow in the description area. We referred him to the instructions again and he noted that this is not something he would usually do. Exacerbating the problem were site errors that caused broken links when navigating between pages. When the output showed up (the line), all the users did not like the line. They preferred having a background color or boxing the even instead. The line is not intuitive. Also on the testing computer, users noticed that the line itself was slightly off, depending on how precisely they drew their line.
| Task 2 | |
|---|---|
| Participant | Task duration |
| 1 | 0:02 |
| 2 | 0:03 |
| 3 | 0:01 |
| Mean | 0:02 |
| Standard Devation | 0:00:49 |
Task 3 (Hard) Getting a reminder about an event
This was by far the most difficult task. Almost all the users did not even try to click on a day or event in the week view, thus getting to the proper reminder page. All the users went to the Customize Reminders section first. There they set the options for their email and when they would like to receive the email. One user commented that having the roll-over stand out more may make it more intuitive to click on. He also noted that maybe having a mouse-over text box appear would be even better. Almost all the users wished for more options to set times such as "the night before".
Setting the reminder on the planner itself is also not entirely clear. After finding and checking the reminder box, many users were confused as to what to do next. Many suggested just having a pop-up box appear for the reminders check would be the most effective way to go about entering the time for a reminder.
| Task 3 | |
|---|---|
| Participant | Task duration |
| 1 | 0:07 |
| 2 | 0:04 |
| 3 | 0:03 |
| Mean | 0:04 |
| Standard Devation | 0:01:42 |
Errors
Task 1
Participant 1: 1 Errors {rewriting task for alignment}
Participant 2: 1 Error {cannot read small writing of task}
Participant 3: 0 Errors
Task 2
Participant 1: 2 Errors {redraws box to show up better, broken link}
Participant 2: 1 Errors {draws arrow instead of a line}
Participant 3: 0 Errors
Task 3
Participant 1: 2 Errors {reminder button location, broken link}
Participant 2: 3 Errors {broken link, reminder location, reminder options}
Participant 3: 1 Error {broken link}
Discussion
What might you change for the "real" experiment?
More rigorous testing of the system in advance: We ran into some technical problems during our first testing session. In particular, we ran into problems using the pen with a laptop, and had to retreat to the computer lab to finish the session. We learned that it is very important to rigorously test any software or hardware that will play a role in user testing. A system that only sporadically works is not good enough, and we will definitely avoid using such a system in the future.
More collective practice with the system in advance: We all worked on different areas of the prototype, so we had each developed different degrees of familiarity with various aspects of the interfaces. However, it is important for the whole team and the facilitator in particular to have a complete understanding of how all these aspects come together. Due to insufficient practice, the testing session did not go as smoothly as we would have liked. The pilot study was useful in pointing out the need for a well-rehearsed script and test, and although we managed to run through a small number of tests, it is something we need to address before attempting to test a larger user population.
More tasks for all use cases: Because of user feedback in our low-fi tests, we decided to change our tasks. However, after conducting the pilot study we realized that while the new tasks covered most of our digital planner's functions, they did not cover them all. For example, "To Do" lists play a prominent role in our interface, but we did not have a task that focused on them. In future experiments, it would be better for us to create more comprehensive tasks that would cover all possible uses for our application - especially because this pilot study was also carried out on a small number of users. Having more tasks would allow us to get more extensive feedback on the full range of features that our application offers, and it would also be more relevant to a larger user population.
More accurate time-tracking: We relied on watches and laptop clocks when collecting process data for these tests, but because we were so intent on capturing our observations, we only made note of HH:MM increments instead of HH:MM:SS. However, the nature of the tasks means that not all of the tasks take more than a minute, and in that case, it would be easier to differentiate between them if we had a method of time-stamping our notes. Our methods worked for a pilot study, but for a "real" experiment, it would probably be more useful to have more accurate process data for more accurate analysis.
What might you change in your interface from these results alone?
Better integration of weekly and daily views: A major addition to this iteration of our prototype was the implementation of daily views, largely because our testers pointed out weaknesses in our weekly view - text was too small to read and individual reminder settings could not be seen or set. Each of the days in our weekly view can now be clicked, which will bring the user to a daily view that addresses both problems. None of our testers, however, picked up on how to access the daily view, even though the mouse changes to a hand cursor upon mouseover. As a result, in future iterations of our website, we want to make these links to the daily view much more prominent, perhaps by changing colors more dramatically or providing more feedback in the form of "click me" text on mouseover.
Greater prominence of individual reminder settings: While none of our testers clicked through to the daily view on their own, when we prompted them to do so, they still did not grasp the interactive nature of the individual reminder settings available on the daily view. We believe this is because AJAX forms of that nature are not as widespread as we thought, and users are not as familiar with the typical feedback (yellow highlight that fades in and out) of AJAXian elements. In the next iteration, we want to make it clear that the individual reminder settings are both clickable and changeable.
Modification of multi-hour display: Our testers did not find our color strips to be a clear indication of multi-hour events, although one did say that he could get used to it. We proposed an alternate display by changing the background color of the hour blocks over the length of an event, and the testers seemed much more receptive to that idea. It also makes the transition between the paper and web interfaces smoother, since boxing hours in the former should really "box" (by shading) the same hours in the latter. In the next iteration, we will replace the color strips with colored backgrounds.
More functionality of to do items: Although we did not specifically test the "To Do" view in a task, two of our testers did try to use this part of the interface, and were frustrated by the lack of features there. Our current interface uses Wizard of Oz techniques for the Search field and Date Range Selection within the to do items. One tester disliked having to set a date range, and just wanted to see all of his items. The other thought that using the Search field would produce matching items that could be individually clicked on (effectively the individual reminder settings we offer within our daily view). There is clearly a disconnect between the way we present this page of the web interface, so in the future, we will need to study a larger user population in order to identify what features should and should not be there.
Workload Breakdown
| Dexter | Eric | Mike | Patti | |
|---|---|---|---|---|
| Prototype Improvements | ||||
| Add pen support for multiple weeks | 0% | 0% | 100% | 0% |
| Adjust alignment issues with pixel conversion | 0% | 0% | 100% | 0% |
| Create daily views in web interface | 20% | 0% | 0% | 80% |
| Create function to send email reminders | 100% | 0% | 0% | 0% |
| Add support for multi-hour events in web interface | 80% | 0% | 0% | 20% |
| Added partial support for individual reminder setting | 60% | 0% | 0% | 40% |
| Prototype Testing | ||||
| Recruit testers | 0% | 33% | 33% | 33% |
| Prepare testing script | 0% | 100% | 0% | 0% |
| Facilitate the tests | 10% | 70% | 10% | 10% |
| Collect process data during tests | 33% | 0% | 33% | 33% |
| Pilot Study Writeup | ||||
| Introduction | 0% | 100% | 0% | 0% |
| Implementation and improvements | 0% | 0% | 0% | 100% |
| Method | 0% | 80% | 0% | 20% |
| Test measures and results | 75% | 0% | 0% | 25% |
| Discussion | 20% | 40% | 20% | 20% |
Appendices
Consent form
CONSENT FORM
PURPOSE OF THIS STUDY
The purpose of this study is to investigate and improve upon the interface of the Digital Daily Planner, which utilizes the Anoto Pen in order to digitize organizational processes. The study is being conducted as part of a final project for UC Berkeley Course CS 160, and your input will be greatly appreciated for the future development of this product.
PROCEDURES
You will be simulating the normal use of the Digital Daily Planner by performing the following tasks: scheduling events (single- and multi-hour), browsing an online interface, and setting email reminders. Your actions, voice and thoughts relating to the Digital Daily Planner may be recorded using audio tape, hand-written notes or other methods.
CONFIDENTIALITY
Your identity in this study will be treated as confidential. The results of this study may be shared with others, but we will not give your name or include any identifiable references to you.
TERMINATION OF STUDY
You are free to withdraw from this study at any time. There will be no penalty or loss of benefits to which you are otherwise entitled if you choose to withdraw.
AUTHORIZATION
I have read and understood this consent form. I freely volunteered to participate in this study and agreed to the procedures described above. I understand that I will receive a copy of this form for future reference, and I further understand that nothing in this consent form is intended to violate any applicable federal, state, or local laws.
Participant Name: __________________________________
Participant Signature: _______________________________
Date: _____________________________________________
Demo scripts
Introductory Script
Below is the script we used to introduce the system and one feature.
We are developing an application for a daily planner that works with a digital pen. With this digital pen, you can write in a daily planner made with specially designed paper. The pen remembers what you wrote and transfers that to the computer. The computer then uploads the data to a server, so you can retrieve a copy of your daily planner from any Internet web browser. You can also do things with the data once it’s on the Internet.
We will be giving you a paper planner, a digital pen, and a laptop computer. We will ask you to perform tasks with these tools. Those tasks will involve writing in the planner and viewing the results in the computer. We will be observing how you work with these tools, in order to learn how to improve our interfaces. Please talk openly about your thought process, what is interesting to you and what is frustrating to you. Also feel free to ask questions at any time.
To show you how the application works, we will show you an example. I will first show you how events in the daily planner are also stored on the web site. Then I will use the pen to cross out an event in the daily planner. You will see that this action is also reflected on the web site.
[Opens the planner] This is a page from the daily planner. It looks like any normal daily planner.
[Flips to the page with a to do item] Here we have a to do item.
[Points to the desktop] Here is our desktop screen. I'll click on the Planner icon to go to the Planner web site.
[Computer shows the corresponding online page] Here I can see the same item on this page, which copies the writing on the daily planner.
[Shows the planner page again] Now, let's imagine that I have finished this item. So, I cross out the appointment to indicate that.
[Cross out the appointment in the physical daily planner.]
[Puts pen in holder] Now, I put the pen in the holder, so that what I wrote is downloaded into the computer and put online.
[Computer shows the corresponding online page, with the item crossed out] Now, when I look at the page online, the item has been crossed out. Whatever I do in the planner is recorded and any changes are reflected on the website.
Script for Tasks
Task 1: Scheduling an event
Imagine that you have a doctor’s appointment that you want to keep track of. Please record in the daily planner that you have a doctor’s appointment at 2 pm on November 17th. Then access the web site and find your appointment.
Task 2: Indicating a multi-hour event
Imagine that you are a businessperson and you have a multi-hour meeting in the future. Please record in the daily planner that you have a business meeting from 9 am to 1 pm on November 18th. Then access the web site to see a copy of what you just recorded.
Task 3: Getting a reminder about an event
Imagine that you want to get a reminder for a dental appointment that you have on November 19th at 10 am. First, please provide your email to the system. Feel free to explore the system or the daily planner to figure out how to do this. Then, record the dental appointment in your daily planner and do what you need to do to ensure that you will be notified by email about this appointment. We have set things up so that you receive this notification email immediately, rather then at the time that you set. Please check your email using the icon on the desktop to see the notification message.
Critical Incidents Log
Task 1
| Time | Event Type | Event Description | Rating | Duration (h:mm) |
|---|---|---|---|---|
| Participant 1 | ||||
| 6:21 PM | Before start | Doesn't understand the limited Sat/Sun schedule | 2 | 0:01 |
| 6:22 PM | Task start | 0:02 | ||
| 6:24 PM | Boxes event | 4 | 0:01 | |
| 6:25 PM | Rewrite event | Task is not aligned right, and rewrites. | 4 | 0:02 |
| 6:27 PM | Rewrite event | Writes more slowly | 4 | 0:00 |
| 6:27 PM | Task end | 0:00 | ||
| 6:27 PM | Suggestion | Not clear alignments; add more reminder options | 3 | 0:00 |
| 0:06 | ||||
| Participant 2 | ||||
| 6:43 PM | Task start | Writing slowly now | 2 | 0:01 |
| 6:44 PM | Writes task | 0:00 | ||
| 6:44 PM | Task end | 0:00 | ||
| 6:44 PM | Suggestion | Can't read what he wrote | 2 | 0:00 |
| 0:01 | ||||
| Participant 3 | ||||
| 7:09 PM | Task start | 0:01 | ||
| 7:10 PM | Write | Quickly writes in task | plus | 0:00 |
| 7:10 PM | Navigate | Quickly navigates interface | plus | 0:00 |
| 7:10 PM | Task end | 0:00 | ||
| 0:01 |
Task 2
| Time | Event Type | Event Description | Rating | Duration (h:mm) |
| Participant 1 | ||||
| 6:35:00 PM | Task start | 0:00 | ||
| 6:35:00 PM | Alignment | Hour bars are off | 2 | 0:02 |
| 6:37:00 PM | Task end | 0:00 | ||
| 6:38:00 PM | Suggestion | Wishes hour blocks were more clear | 2 | 0:00 |
| 0:02 | ||||
| Participant 2 | ||||
| 6:44:00 PM | Task start | 0:01 | ||
| 6:45:00 PM | Draws | Draws an arrow in the descriptioin area | 3 | 0:01 |
| 6:46:00 PM | Instruction | Rereads the instructions given beforehand | 0:01 | |
| 6:47:00 PM | Redrew box | Redraws box in hours area | 3 | 0:00 |
| 6:47:00 PM | Task end | 0:00 | ||
| 6:47:00 PM | Suggestion | Box the entire event with a color box, not a line | 3 | 0:00 |
| 0:03 | ||||
| Participant 3 | ||||
| 7:10:00 PM | Task start | 0:00 | ||
| 7:10:00 PM | Draws box | Immediately draws box in right place | plus | 0:01 |
| 7:11:00 PM | Task end | 0:00 | ||
| 0:01 |
Task 3
| Time | Event Type | Event Description | Rating | Duration (h:mm) |
| Participant 1 | ||||
| 6:27 PM | Task start | 0:02 | ||
| 6:29 PM | Bad page | Customize reminder tab wrong | 3 | 0:03 |
| 6:32 PM | Rewrite event | Unclear about reminder options | 3 | 0:02 |
| 6:34 PM | Rewrite event | Website error | 3 | 0:00 |
| 6:34 PM | Task end | 0:00 | ||
| 0:07 | ||||
| Participant 2 | ||||
| 6:48 PM | Task start | Writing slowly now | 2 | 0:02 |
| 6:50 PM | Set reminder | Entered in data | 0:02 | |
| 6:52 PM | Renavigates | Broken links | 0:00 | |
| 6:53 PM | Task end | 0:00 | ||
| 6:53 PM | Complaint | Can't read event | 2 | 0:00 |
| 0:04 | ||||
| Participant 3 | ||||
| 7:11 PM | Task start | 0:00 | ||
| 7:11 PM | Writes task | plus | 0:00 | |
| 7:11 PM | Reads task | plus | 0:00 | |
| 7:11 PM | Customizes reminder | Does it easily | plus | 0:03 |
| 7:14 PM | Task end | 0:00 | ||
| 0:03 |
