PilotStudy-Group:ThePenIsMightier

From CS160 User Interfaces Fa06

Contents 1 Introduction 1.1 An Introduction to Blogpad 1.2 Purpose 2 Implementation and Improvements 3 Method 3.1 Participants 3.2 Apparatus 3.3 Tasks 3.4 Procedure 3.5 Testing Protocol 4 Test Measures 5 Results 6 Discussion 7 Workload breakdown 8 Appendices 8.1 Script 8.2 Consent Forms 8.3 Demographics information form 8.4 Raw Data

Introduction

An Introduction to Blogpad

Our interface for the Anoto Digital Pen allows bloggers and journal-keepers to create an electronic record of their writings, no matter where they are, while using the easy and familiar interface of pen on paper. Our application lets bloggers and journal-keepers create posts as soon as they see something inspiring, without having to wait for a large chunk of free time or access to an internet-connected computer, and without carrying around a fragile, unwieldy laptop or tablet PC.

Purpose

The purpose for this user study is for us to get a better idea of how actual users would react to the interface design of our application, and correcting any usability problems before the application attempts to achieve more widespread use. Rather than looking for alternatives to the overall design of the application, like we did for the lo-fi prototype study and the interactive prototype study, we are looking for input on more specific details of our application's interface, such as the color and placement of the buttons. As well, we want to discover which modules, if any, are more difficult for users, so that we can improve the

Implementation and Improvements

We have implemented an entirely new dialog for the multi-page posting task. For reference, this task is to post one writing as multiple blog posts, for the use case of writing several days of entries before docking the pen. The dialog's purpose is to allow the user to select part of his writing as one post, and the rest of his writing as another post.

Previously, we had been struggling between other solutions, each with its own benefits and drawbacks. Among our previous options were a click-and-drag dialog, changing the mouse cursor to include directions, pop-up speech balloons, and modal buttons. After doing some research about partial-selection dialogs, we came up with a new hybrid. The user's writing appears as a collection of previews of pages, with each page having a small checkbox in the corner. This easily allows the user to view his writing, know which are selected or not selected, and change his selection, providing key features of direct manipulation. Clicking anywhere on the user's page toggles whether that page is selected for posting or not, and by default, the entire piece of writing is selected as one post.

We clearly saw the benefits of the new dialog during our user testing. Whereas in our previous trials, the multi-post selection reliably caused the user confusion, users flew through the dialog in this round of trials. This complex part of our "difficult" task became a one of the quickest dialogs the users used in the entire application. Users could easily see all of their writing and its current state of being selected or unselected, as well as have a visible and easy-to-use tool to toggle their selection.

We also added the fuctionality of docking the pen and transferring the document(s) to the BlogPad software, and allowed for the exporting of a blog entry. Other improvements included being able to select the directory in which blog information is stored and being able to select the directory and filename of the jpg file we wish to export to.

Method

Participants

Participant A

Our first interviewee was a twenty-two year old male studying Electrical Engineering and Computer Science in his third year at Cal. In a questionnaire he rated his experience with blogging as a three out of five, and his familiarity with using a computer at five out of five. Participant A also noted that he does not keep a journal. Participant A was selected in part as a user who represented expert computer users, but an expert user that was not involved in the blogging community. This allowed for greater study of the merits of the interface from the user's standpoint on the computer, rather than on the pen and paper.

Participant B

Our second interviewee was a twenty year old male studying Mechanical Engineering in his third year at Berkeley. In a questionnaire he rated his blogging experience as one out of five and his computer experience at a three out of five. Participant B noted that he did not keep a journal of any kind. Participant B was chosen as a representative of users who may use computers but also enjoy the affordances of pen and paper. As a Mechanical Engineering student, he brought both parts of the interface together and tested both the merits of the interface and it's utility to easily disseminate drawings which he quickly sketched for us.

Participant C

Our third interviewee was a twenty-one year old male studying Computer Science in this fourth year at Berkeley. In his questionnaire he rated his experience with blogging at a four out of five (mentioning as a caveat that this was a level of his familiarity about how it works, not in his actual utility of the medium) and his computer experience as a five out of five. This user was also not a journal keeper. Participant C was chosen based upon his enthusiasm for the interface of pen-and-paper and computer when described to him by one of our investigators. We thought that his opinion was useful in that he seemed like a user that, if given the proper motivation, may begin to use such an interface (the Anoto pen and our BlogPad software) in the future.

Apparatus

For the pilot study, we chose to be in the CSUA lounge on 3rd floor, Soda Hall. The CSUA lounge is a spacious and quite room, which is a great enviornment for conducting user studies. The machine that was used by our users was placed on a long, rectangle table, where the user sat on the long edge. The greeter was stationed to the right of the user, and the note-takers were placed to the right of the user and behind the user. The machine used by the user is a laptop running Microsoft Windows. The Anoto Pen system was connected to the laptop via USB port.

We placed our observers at four corners (back-left, back-right, forward left, and forward right) around the user, in order to record more complete observations. This allowed us to view the user's mouse actions as well as facial expressions, and send our observations to our central logging station as described in the procedure.

Tasks

We felt that the tasks from the previous interviews were representative of the average user's actions, and did a good job of demonstrating the essential features of BlogPad. Thus, we used the same tasks as example tasks for the pilot study, as they will be used to show that the changes we have made since our previous user studies are effective in fixing the interface's flaws.

Task 1

Save writing to local computer as image.

We believe this task is a perfect example of a representative easy task. This is because we observed the users in the quick iterations of the lo-fi paper prototype. This is a short, easy task, but not too easy because new users would still need to internalize a new interface. Even when an environment is familiar and intuitive, users tend to act slower and read more carefully when presented with a new application.

Task 2

Save writing and upload writing to a blog as a single posting.

This task is a representative example of a medium task, as this is the task that we anticipate most users will use often. The real inspiration behind BlogPad is to allow its users to post his blog entry directly from his hand written journal. In this task, we ask the user to post everything that he has written to his online blog. This task is slightly harder than the first task, but users should not have trouble completing it, as suggested by the lo-fi prototype user studies. However, it will be interesting to see if the new placement of our buttons for the 'posting interface' would cause any trouble to user.

Task 3

Save writing and upload writing to a blog as multiple separate posts.

This task uses most of the more complicated functionality we have available. Perhaps before the iterations with real users, it could be called “too hard” or confusing, but the input we received both directly from the users and from observations of the users allowed us to make some small changes to make this task more comfortably doable. This is the most difficult task of the three tasks - we ask the user to post what he has written as multiple blog entries. The hardest step is when the user has to use the 'posting interface' to select the beginning and end of the entries, and one that is still an ongoing study within our group. Our temporary solution is to take advantage of the fact that most text on computers is read left-to-right. The user will be naturally drawn toward the left side of the screen first, seeing the modal buttons and using them to mark the start and end for one of multiple blog postings. This step is discussed in more detail in the prototype overview.

Procedure

After informally greeting the participant, we first asked the participant to complete the consent and demographic forms which we have previously prepared. After the forms were completed, we introduced our group and group members to the participant. We informed the participant that over the duration of this experiment, we will be observing him and occasionally ask questions, mainly questions about his thought process during the task. We encouraged him to 'think aloud' during the experiment. We let the participant know that we are creating an application which will take blogging back to the pen-and-paper interface as well as making it such that blogging no long requires instant access to a internet-ready computer. We also gave the participant a quick overview of the three tasks that he will be performing for us.

Before the participant actually performed the task, our interviewer performed a quick demo of how to open and close the BlogPad application. After the demo, the interviewer asked the participant to perform the first task: to write a blog entry which takes up at least two pages, and save that as an image file. While the participant was working through the task, the interviewer asked questions to find out what he was thinking. After the participant completed the first task, we asked him to start on the second task: to post the previously saved entry to the default blog account as a single entry. Again, during this task, the interviewer continued to ask questions to find out what he is thinking about. After the second task was completed, the interviewer asked the participant to perform the final task: to post the previously saved blog entry to his blog account as multiple entries. Like the previous two tasks, the interviewer asked questions to keep the participant to think aloud. After the task was completed, the interviewer asked if he had any questions or suggestions. We close to interview by thanking the participant once more.

During the interview, there were also three note-takers who were observing the participant and taking notes of any special gestures, such as when and where the participant was confused, moments where participant was impressed by the application by saying something such as “cool”, or “great”, and comments made by participants during and after the experiment. Another responsibility that the note takers had was to take note of how long it took the participant to perform the tasks and how many clicks it took. To make this task simpler, we used the Trillian instant messenger chat program as our aid. Because Trillian, by default, keeps track of timestamps for ever message sent, we send each note as a message, including ‘s’ for start, and ‘c’ for click. Using the log from Trillian, we can not only find out how long it took each participant to complete each task, how many clicks it took, but also when the click took time.

Testing Protocol

We will be using aim + logging for quick and easy note-taking. Upon the user doing some sort of note-worthy activity, we will:

• IMMEDIATELY send a single character (then press enter)
• type up a note and send the note.

We will then use timestamps to identify user actions.

This gives us an "exact +/- 1second" message (timestamp of the single letter) to measure times by, then a description of the user action that comes at our leisure. For example, when the user clicks the "next" button at 12:00:00, I will attempt to send "c" to our logbot by 12:00:01, then when I get the chance, I'll send a message saying "user clicked 'next' button". the second message can show up whenever, and we still have the exact time the user clicked, and we know what he clicked.

To make sorting out the messages later (AND time between messages), we are going to assign different letters for different "channels" of user communication.

• click = C, then explanation
• verbal = V, then explanation
• facial = F, then explanation
• keyboard = K, then explanation + which key(s)
• other = O, then explanation (like hovered over something expecting a tooltip, etc, or user yawned, or user threw chair, etc.)

'Explanation' can be anything from "user clicked for no reason" to "user attempted to click on picture, which should do nothing" to "user clicked on save button instead of post button, and failed the task miserably"

Test Measures

We measured many variables, including:

• Total time of completing each task - This is our main measure. We want to know how long each of our tasks take, so that we can attempt to improve on any exceedingly long steps. As well, we can see which dialogs, if any, are the cause of extending the tasks.
• Time between clicks, and purpose of each click - This is a secondary measure we looked at, to see where users were potentially being led astray, or to find more confusing parts of the dialog. If a user takes a long time on one step, we would probably want to simplify the step. If users click somewhere with no feedback, we would want to provide feedback, or better directions toward their actual goal. If users make an error, we want to clarify how to complete their task.
• Number of times user expressed verbal confusion - This is a secondary measure that allows us to see which dialogs were less straightforward.
• Number of times user went back - Clicking on the "back" button implies that the user is entirely lost, unsure of his previous actions, or confused about the next step. We want to provide enough recognizable feedback that the user will be sure of his actions, and we want to provide enough clarity in the dialogs to allow the user to confidently progress through his task.
• Number of misclicks - This measure allows us to get an idea of
• Did the user attempt to hover the mouse anywhere? - This measure allows us to see if there are any areas where the user desires more information.
• Time spent on each window - This measure allows us to check for bottlenecks in our process. If any are found, we want to speed up the slowest step by either splitting it or clarifying it.

Results

Task 1:

During our first interview with participant A, some connection issue occurred so it slightly delayed the progress. Fortunately that was the only delay that happened during the interviews.

For participant A, he had a hard time figuring out which button to click in order to save the hand written document as image file. After slight exploration, he was able to complete the task. After participant A’s experiment, we made some minor label changes according to his feedback. The changes were effective, as neither participant B or C had experienced the same confusion.

For participant A, it took 547 seconds to complete the task, using 19 clicks.

For participant B, it took 183 seconds to complete the task, using 9 clicks.

For participant C, it took 172 seconds to complete the task, using 6 clicks.

It took participant B 76 seconds to complete the task, using 9 clicks, and participant C used 116 second to complete the task, using 6 clicks.

Task 2:

For this task, no one participant had a partiruclarly hard time with completing the task. In fact, participant C was able to complete the task using half the time the other participants did, even when used the number of clicks as participant B.

For participant A, it took 76 seconds to complete the task, using 11 clicks.

For participant B, it took 71 seconds to complete the task, using 7 clicks.

For participant C, it took 34 seconds to complete the task, using 7 clicks.

The average time that it took the participants to complete this task is 60.33 seconds, and the standard deviation is 22.94. The average clicks used the participants is 8.33, with standard deviation 2.3.

Task 3:

This is suppose to be the hardest task of the three. However, the result here suggested that the difficulty between this task and task 2 is not much of a difference.

For participant A, it took 116 seconds to complete the task, using 12 clicks.

For participant B, it took 55 seconds to complete the task, using 13 clicks.

For participant C, it took 44 seconds to complete the task, using 17 clicks.

The average time that it took the participants to complete this task is 71.667 seconds, and the standard deviation is 38.786. The average clicks used the participants is 14, with standard deviation 2.04.

Discussion

We discovered that our new design for the multi-post dialog was much more effective than any of our previous options. In other trial runs, it was the longest and most confusing steps, with users clicking un-manipulatable areas, pausing for long periods of time, or getting lost entirely. Instead, this step became one of our fastest steps, with only positive verbal feedback, and no long pauses.

We also learned that some users are much more exploratory than we had previously thought. User 1 and User 3 very quickly flew through all available buttons and tabs, clicking on buttons and switching tabs multiple times. They used an "exhaust all possibilities" approach to learning the interface. This information reaffirms the importance of being able to easily move back and forth from within tasks or wizards, as well as the ability to entirely cancel a task. This also very clearly showed us the limitations of short-term memory and the size of the users' locus of attention. Although these were smart engineering students at UC Berkeley, they still rapidly switched tabs and visibly "rescanned" the page with their visual processor, showing that they did not remember what was on the tab they had seen 5 seconds ago.

These testers told us during debriefing questioning that they had indeed not read every option; in fact, one of them had not read the text for the four available buttons in a popup dialog because one button was already highlighted as default. This shows us the usability our preselected defaults lends to the users; however, it also shows us that for important or less-reversible decisions, we should perhaps not provide a default for fear of a user rapidly clicking through and unwittingly harming themselves. In fact, our third tester accidentatly closed the dialog for our hard task, to make multiple blog posts from one piece of writing, instead of progressing to complete his second post. As he clicked the "Finish" button, he uttered an expletive, and said that he accidentally closed the dialog even though he had seen the button he knew he should have clicked. If we are to have any irreversible tasks at all, confirmation or purported lack of a default option is important.

Our second user was quite the opposite from the first and third. He progressed through tasks very slowly, and read most of the instructions in their entirety, exploring the application minimally to complete his tasks. We ensure that we provide for users similar to him by providing clear instructions that are as short as possible.

Before the final presentation, the module to interact with the LiveJournal API will be fixed, so that users can see their actual writings immediately be posted online, as opposed to our samples. We met with enormously positive feedback, and only detected two moments of confusion, which were immediately fixed before our second interview.

From this experiment alone, we already found a few changes to implement; in fact, Tom implemented a few hot-fixes between interviews. This included a phrasing inclarity that our first tester expressed confusion at, and the result was easily seen in our subsequent testers - no confusion. Also, part of our application required loading a browser window, which took more time than we expected. Tom immediately switched the mouse cursor to an hourglass, after we saw that the first user was confused while waiting for feedback to his actions. In this manner, we had a sort of delayed direct manipulation with our testers, which was very useful in letting us know what changes worked, and how well they worked.

Workload breakdown

Tony Lai: Implementation: 0%, Report: 40%(Introduction, most of method, except for 'participant', Results, Scripts, Forms)

Charles Lee: Implementation: 10%, Report: 60% (Heavily edited: Introduction, Script, Method. Authored: Experiment Protocol, Implementation and Improvements, Discussion. Also was Experiment Facilitator, and attempted to implement LiveJournal API (not-so-successful).

Tom McClure: Implementation: 50% (image selection dialog, pen dump import and dialog, persistence layer, image export, image publishing, cosmetic improvements to interface layout).

Vahe Oughourlian: Implementation (40%) of windowing system, cleanup of windowing system, minor bug fixing, wiki editing.

Appendices

Script

Introduction

Hi, thanks for participating in our user study. We will be observing you while you perform three tasks with our application. Please "think aloud" as you work, so we can follow your train of thought.

Mission Statement: We are creating an interface for the Anoto Digital Pen, with the intention of allowing bloggers and journal-keepers to create an electronic record of their writings, no matter where they are, while using the easy and familiar interface of pen on paper. We want to allow bloggers and journal-keepers to create posts as soon as they see something inspiring, without needing to wait for their next free chunk of time at an internet-ready computer, and without carrying around an unwieldy computer.

Background: The three tasks are going to be to save a piece of writing to your computer, to post a piece of writing as a blog post, and to post a piece of writing as multiple blog posts, in the case that you have written several days worth of blog posts and are now uploading all of them at once. We will remind you again of what these tasks are later.

Initial Demo

Here is the BlogPad icon (on desktop). If we double click the icon, the main window of BlogPad will open, and here it is. At this screen, we can view our library of blog posts, and read or sort them. If we click the 'X' button at the top right, the program will close, as such.

Further Instructions

(Task 1) Now, let's try your first task, to save your writing to your computer. Please do talk your way through your thoughts as you do so. Here's the pen.

(Task 2) Good job, let's try the next task, which is to post your writing to your blog.

(Task 3) Great, now let's do the final task: upload the writing in the pen as multiple, separate blog entries, say, two, for example. Note that you will not have to setup your blog login/password information again.

(Follow-up) Thanks for helping us out. Are there any parts that you found confusing or unclear, any parts you thought were absolutely great, or perhaps any features you think would be useful?

(Closing) Alright, thanks again for your time.

Consent Forms

Demographics information form

Raw Data

---