PilotStudy-Group:4Corners

From CS160 User Interfaces Fa06

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participants 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 5.1 Demo 5.2 Task 1: Creating and Modifying a Blog Entry 5.3 Task 2: Changing Default Appearance 5.4 Task 3: Embedding An Image 5.5 Overall Comments 6 Discussion 6.1 What We Learned From The Pilot Run 6.2 What We Might Change For The "Real" Experiment 6.3 What We Might Change In The Interface From These Results Alone 7 Workload Breakdown 8 Appendices 8.1 Charts 8.2 Cue Cards 8.3 Raw Observation Data with Notation of Critical Incidents 8.4 Consent Form 8.5 Demo Script

Introduction

The concept behind our Anoto-enchanced blogging application is to target casual personal bloggers who wish to share information about their lives and experiences with friends. We seek to address the fact that people are inherently accustomed to writing on paper (like old fashioned diaries for example) and are prone to forgetting their ideas if they are not jotted down on the spot. Many people have reduced their blogging frequency over time and for various reasons are not able to blog as often as they would like to. The main problem is essentially one of available time, as bloggers often need to take a chunk of time out of their busy schedules to sit down at a computer terminal to draft or publish an entry. Convenience and portability are the main issues to be resolved, and our solution seeks to address these facets by providing an easily accessible interface using the Anoto system for jotting down ideas anytime, anywhere. The inherent portability of digital pens and paper allow everything to be written down when inspiration strikes, then get uploaded to a computer once the blogger gets home, after which he can make use of editing functionalities and configuration options.

The purpose of the pilot study experiment is to test our existing interactive prototype, particularly with respect to the convenience, user-friendliness, efficiency and features of our interface design. We tested our most current prototype with three users, gathering as much critical data as possible from each test. This data was collected for analysis on aspects of our interface that could be improved, and also to judge the quality of the changes we have implemented since our lo-fi prototype. It was important to get actual feedback from target users instead of relying on our own biased pilot tests, both because a) we had invested plenty of time into our current code and needed impartial opinions on which features did and did not work the way we envisioned them, and because b) we needed to make sure that somebody who had never used our system before could quickly figure out what to do and how to do it. Finally, this gave us plenty of information on which aspects of our interface stood to benefit the most from improvement, as well as some clues on how to implement these improvements.

Implementation and Improvements

• Changed the vertical scroll bar to be inside the My Blog, My Drafts and editing tabs. In our original prototype, the vertical scroll bar was independent of the tabs, and could be used to view more than one entry at a time in My Blog and My Drafts, but served no other real purpose. This created problems when editing because clicking on edit for the last entry meant you had to scroll back up to see the editing panel. It was also a problem because the main tabs themselves scrolled out of view when you moved the scroll bar thumb down, and this meant you had to scroll back up to access different tabs. We solved all these problems by making the scroll bar part of the My Drafts and My Blog panels themselves. In other words the horizontal scroll bar does not exist, and the vertical scroll bar only appears inside the tabbed pane when needed, and does not affect the position of the tabs themselves when used.

• Removed the horizontal scroll bar. This scroll bar was a product of inefficiently set panel sizes and was never actually used, so we took it out. It also suffered from the same problems as the vertical scroll bar, but these are no longer an issue.

Here you can see both the abovementioned changes. Notice that the scrollbar is not at the top but the four main tabs remain visible.

• Added delete buttons to entries in My Drafts. This was not a requested feature but something we realized might nevertheless be important to our users after we submitted our initial prototype. Simply, we inserted buttons under all entries in My Drafts to allow for the deletion of entries, so that users could discard unfinished drafts if they lost interest in, or decided against, eventually publishing them. These entries would subsequently disappear, to be replaced by the next most recent entries.

Notice the newly added delete button, described above.

• Added icons to the four main tabs. We added little unique icons to the My Blog, My Drafts, My Friends and My Settings tabs. These icons allow for easier visualization and distinction between the separate tabs, so that the user might more quickly grow accustomed to their look and select tabs based on location and icon rather than reading them each time. Additionally, they make our interface slightly less plain and more pleasing to the eye, which reduces user frustration.

Notice the different icons on each tab.

• Fixed miscellaneous layout issues. These were mostly positioning of buttons and popups. There was one dialog popup in our interactive prototype that appeared in the bottom right of the interface, slightly obstructed by the window's edge, so that it had to be dragged back up to click on its OK button and dismiss it. We fixed this, among other minor layout problems.

• User can no longer click on other tabs while editing. This was perhaps the worst bug in our initial prototype. While editing a draft, a user could click on other tabs before saving or canceling. This would sometimes cause the program to act in unexpected ways. To rectify this, we enforced tab positions while editing and removed the option of clicking on other tabs. To make this clear to the user, the icons on other tabs are greyed out while the user is editing an entry in My Drafts or My Blog.

Notice that the other icons are greyed out. Clicking on those tabs has no effect, because the user is currently editing an entry.

Method

Participants

• Participant 1: The first particpant is a 4th year UC Berkeley student majoring in Electrical Engineering and Computer Sciences. He uses a Xanga blog (xanga.com), but only updates it once a month on average. He often thinks of great blog entry ideas when he is not near a computer, and many times these ideas are forgotten before he gets a chance to put them into an entry on his blog. Also, his busy schedule does not allow him to be at home for much of the day, so he is interested in carrying some sort of device that allows him to record his thoughts wherever he goes.

• Participant 2: The second participant is a 4th year UC Berkeley student. She is majoring in Computer Science and is graduating in December. She was selected because she is a very regular blogger. She posts blog entries to her personal blog almost every week. She is very interested in posting hand-drawn images to her blog which is not possible with her current blogging service(bloggers.com). She is also interested in posting blog entries in Chinese.

• Participant 3: The third participant is a 4th year UC Berkeley student double majoring in Anthropology and Japanese with a minor in English. He has a premium Xanga membership (www.xanga.com) and blogs frequently. He updates his Xanga regularly with public as well as private entries ranging from about 1-5 posts per week. He often doodles and sketches and would like a way of jotting down his creative thoughts when inspiration strikes on the go. Doodling and writing Japanese kanji characters are currently not supported by Xanga; this is the main reason participant 3 is interested in our digital pen and paper blogging interface.

Apparatus

All the three tests were carried out in Soda Hall. The first two tests were done in 330 Soda and the third in one of the rooms on the second floor of Soda. Both these sites are fully equipped with tables and chairs that were used to set up the tests.

A personal laptop running on Windows XP was used for the purposes of conducting the test. The interactive prototype of 4Corners Blog was run on Eclipse IDE. The cradle of the digital pen was hooked up to the laptop and streaming mode was turned off. The digital pen was placed on the table along with the paper interface or "4Corners Blog Pad". We also arranged for a separate mouse just in case the users were not comfortable using a laptop.

Traditional pen/pencil and paper were used by the observers to take notes. A stopwatch was used to time each of the tasks performed by the users.

The participants were seated on a chair across the computer. The facilitator sat next to the participant. The three observers took different positions in order to have an unobstructed view of both the participants and the interface on the table. The atmosphere was calm and the climate was comfortable.

Tasks

We wrote the goal of each of the task on index cards and presented them to the participants before beginning each task (see Appendix for these cards). The following are the three tasks that we conducted for the Pilot usability study:

• Easy Task - Create and Modify a blog entry
  

The goal of this task was to create a blog entry on the paper UI of "4Corners Blog" application and later modify the same entry on the provided graphical user interface. The following steps were expected of the user in order to accomplish the goal of this task:
1. write a blog entry on the paper UI and check save as draft box.
2. dock the pen in the cradle.
3. fire up 4Corners application; a window will appear displaying the drafts tab which will have the most recent entry on the top of the page with date stamp.
4. click on EDIT button
5. A screen will appear with the entry for preview puposes and a bunck of other buttons, such as "Background Color" button, "ink Color" button and "Stroke Width" button. 6. The user can either click on one of the buttons and modify the blog entry or right click on the blog entry
7. In case of right click, a drop down will appear with a list of options
9. click PUBLISH or SAVE

• Moderate Task - Change the default appearance of the entire blog

The goal of this task was to change the default appearance of the entire blog such as Background color, ink color and stroke width. Participants were provided with the front page of 4Corners Blog application. The following steps were expected of the user in order to accomplish the goal of this task:
1. Click on the SETTINGS tab; a screen will appear with four options: Manage Account, Change Default Appearances, Security Settings and Upload pictures
2. select the "Change Defualt Appearances" option; a screen will appear with the following options: Background Color, Ink Color and Stroke Width. This screen will also have a preview box where the most recent published blog entry will be displayed. The purpose of this preview box is to provide the user with an instant feedback on the options being selected. The blog entry in the preview box will change based on the option selected by the user.
3. select one of the three options; as mentioned earlier the appearance of the blog entry in the preview box will change.
4. click on SAVE; the user will be taken to the "YOUR BLOG" page. The appearance of this page has been updated in accordance with the selections made my the user.

• Difficult Task - Insert an image into a draft entry

The goal of this task was to create a blog entry leaving some enpty space on the upper half of the blog space and later add an image to this blog entry using 4Corners application. The additional instruction for inserting an image was stated in the paper ui. The following steps were expected of the user in order to accomplish the goal of this task:
1. write a blog entry using the digital pen leaving some space on the top of the blog entry.
2. dock the pen.
3. fire up 4Corners application; a window will appear displaying the drafts tab which will have the most recent entry on the top of the page with date stamp; there is a "Insert Picture" button at the bottom of the blog entry.
4. click on the button; a pop-up to select image will appear.
6. select an image file from the machine
7. click on the OK tab to save the selection; the selected image will appear on the blog entry.
8. select the PUBLISH button.

Procedure

For the purpose of conducting the tests effectively and efficiently, we assigned specific tasks to each of the team members. Tabassum was the facilitator; Ramy, Maksim and Chen were the observers. All of us acted as greeters. We decided on keeping three observers in order to make sure that we get an exhaustive description of the tests and the behavior of the users. We decided against shuffling the roles and each one of the team members gained expertise in the task assigned to him/her. This was done to avoid confusion and produce efficient execution of the tests. Two of the three tests were scheduled on Friday and one was scheduled on Sunday. On the day of the tests, we practiced the steps of the test with a team member playing participant.
The appointments were made based on the availability and convenience of the user. Appointments were scheduled with the participants such that there was no time conflict between the appointments of the three participants. This was done to give uninterrupted and undivided attention to each of the participants, to protect the identity of the participants, and to avoid information leak.
Participants were cordially greeted by all the team members and were provided with a comfortable chair to sit on. We introduced ourselves and had light conversation with each of the paricipants to make them feel comfortable before beginning the tests. The facilitator explained the proceedings of the test to the users. She presented the consent form to the participant and requested them to read it carefully and understand the text on the consent form. The participant was asked to place his/her signature on the space provided if he/she agrees to participate in the usability test.

Next the facilitator explained the context of the test in the order mentioned below:
1. what is digital pen and paper technology and how does it work?
2. why and how are we involved in making an applicaiton for this technology?
3. what is the main idea behind our project?

The participants were asked to act naturally and were assured that they were not the ones being tested and none of the answers given by the participants would be considered incorrect. Furthermore, the facilitator made it clear that this was a usability test of our system. The main idea behind conducting such tests is to measure how well people can use our application for the intended purpose .

A DEMO was performed which showed each of the major user interfaces of 4Corners blogging application to the participants. Participants were encouraged to think aloud and state what comes to their mind when they look at each of the UIs. Furthermore, the facilitator asked questions whenever the participants seemed a little confused or stopped following the "thinking aloud" protocol.

The goal of each of the three tasks was written on separate index cards in bold text (see Appendix). Scenarios were created wherein the participant would perform the task naturally. At the beginning of each task the participants were presented with the corresponding index card and the facilitator described the corresponding scenario. This information was written on the paper interface of the application. Participants were intermittently asked to think aloud. No reactions, facial or audible, were made by the team members while the tests were being performed.

At the end of the tests, the facilitator asked the participants regarding the choices that they made during the tests. The participants were also requested to give their overall opinion about the system and to suggest modifications to make the system more user friendly.

We showed our gratitude to the participants and thanked them for their time.

Test Measures

For each task in each session, we used several benchmarks to help us identify positive points of our design and possible areas of improvement to our design:

• Time on the task - We measured the time it took for each participant to complete each of the three tasks. Prior to the user testing, we thought of our expectations for times to complete each task by a casual blogger. The recorded times from the three sessions provided data that we could compare to our expectations and identify which task caused the user to be less productive than we expected. We also would be able to identify points where we could simplify the entry process to reduce the time of completion

• Number of unexpected behaviors - For each task in all three sessions, we noted the number of unexpected behaviors (or errors) that the participants made. Prior to the user testing, we thought of a sequence of actions for each task that seemed the most logical for us to complete the particular task. An unexpected behavior occurred whenever a participant tried to use a feature/option that did not directly help him/her achieve the specified task, or whenever they performed an action/sequence of actions that differed from our expectations. Whenever an unexpected behavior occurred, we noted it in our observations, and added it to the participant's count for the particular task. Unexpected behaviors gave us a measure to try to explain why certain participants behaved a certain way with their knowledge and values taking into account. By noting and analyzing the number of unexpected behaviors, we would be better prepared to prioritize the implementation of improvements to our interface.

• The "Big Picture" - Lastly, we sought to collect as much information about each participant's actions as possible. The exact sequences of actions that each participant took to complete each task were recorded. All questions or requests for clarification from the participant were also written down. Any recommendations or comments about how the participant envisioned the interface to function were recorded. Finally, any observations regarding mistakes or misconceptions made by the user were recorded.

Results

Demo

All three of our test subjects didn't have much trouble moving through our demo as they correctly identified the overall interface including the functionality of the four tabs, the date and timestamp of the entry, the blog entry itself, and the clickable links.

Subject 1 browsed through the tabs and encountered certain areas with "yet to be implemented" pop-up dialog boxes such as the links to friend's most recent entries under the "My Friends Blogs" tab, this is to be expected. However, the test subject was still able to identify the intended functionality correctly by stating that it would give the reverse chronological order of entries of the respective friend's name that was clicked on. The subject identified the functionality of all four tabs correctly and incidentally performed task 2 while browsing through the GUI free form.

Subject 2 identified the "My Blog" tab to allow for the changing of text, colors, and noticed that the mouse pointer changes from the arrow to a hand on a hovered link. The subject identified that the "My Drafts" tab contains private entries that are not visible to the public unless the user changed the permission or security settings. He correctly identified that the "My Friends Blogs" tab contains a list of links to friends and should be sorted by the most recent date. Lastly, subject 2 identified all the features under the "My Settings" tab: manage account allows for the changing of password and email address, change default appearance is useful for uniform settings throughout the entire blog, security settings allow the user to manage privacy in entries as well as blocking certain people.

Subject 3 was surprised that not many posts existed on the "My Blog" page with special interest that there weren't any traditional computer typed posts and all were handwritten. The subject pointed out that the number of comments is ambiguous and a possible fix would be a numbering system enclosed in parenthesis. She correctly identified the "My Drafts" tab to contain things written but not yet published to the main blog, the "My Friends Blogs" tab is easy to identify which links are clickable by the underlines and mouse cursor changes, the "My Settings" tab allows users to change username and password under the manage account button as well as to change the default appearance of the blog. Lastly, subject 3 commented that the buttons of our GUI look flat in general but that is likely to be a limitation of the R3 paper toolkit and after we mentioned this to her, she gracefully stated that she could accept them the way they are.

Demo

	Participant 1	Participant 2	Participant 3
# of Unexpected Behaviors	1	1	2

Statistics of Dependent Variables

• Mean # of Unexpected Behaviors = 1.333 unexpected behaviors per participant
• Standard Deviation # of Unexpected Behaviors = 0.4714045207910318 unexpected behaviors per participant

Task 1: Creating and Modifying a Blog Entry

Our index card instructions were clear enough for the users to follow for this "easy" task and thus problems were few and far between. Despite clear instructions, our users still hesitated before making their first choices on the blog interface and were unsure of their selections. We are reasoning that this is because it is the first task and its reasonable for the user to feel a tad bit unsteady when making their first decisions and will get better once more accustomed to the interface. Hence task 1 should naturally take the longest time to completion.

Subject 1 wrote an entry with a simple picture sketch before ticking the respective box on the paper UI to save this blog entry as a draft. Then our test subject proceeded to dock the pen in the cradle, loaded the pen data to the computer, and then saw that his writing appeared as the most recent entry under the "My Drafts" tab. Proceeding to do the edit part of this task, subject 1 clicked edit for the new draft entry in "My Drafts" but was confused about how to modify the entry. He didn't believe that he could erase strokes so he took the physical pen and paper and added "add this line" to his original entry before going through the whole data transfer phase again. The result was obviously a new saved draft and the subject believes thats how he would approach this task. Also, the subject commented that he would like the ability to erase or add rather than just changing colors.

Subject 2 was delighted to be tested and quickly generated a lengthy blog entry before publishing it. He then docked the pen, loaded the pen data, saw that his writing appeared as the most entry under the "My Blog" tab because he ticked the corresponding box. He modified that entry by clicking edit before changing the background color to gray and text to gold before saving the changes.

Subject 3 commented that the paper UI looks simple and intuitive with clear instructions. She drew a sketch with some writing by it and then saved this as a draft by ticking the appropriate box on the paper UI. She docked the pen, loaded the pen data, and saw that her writing appeared as the most recent entry under the "My Drafts" tab. The test subject modified the entry by clicking edit and followed that up by changing the background color to white and ink color to gold before saving the changes.

Task 1

	Participant 1	Participant 2	Participant 3
Task Completion Time	5 min	6 min	6 min
# of Unexpected Behaviors	2	0	0

Statistics of Dependent Variables

• Mean Task Completion Time = 5.666 min
• Standard Deviation of Task Completion Time = 0.47140452079103334 min
• Mean # of Unexpected Behaviors = 0.666 unexpected behaviors per participant
• Standard Deviation # of Unexpected Behaviors = 0.9428090415820634 unexpected behaviors per participant

Task 2: Changing Default Appearance

This "medium" task turned out to be simple and quick and thus was the task performed the quickest by all test subjects on average.

Subject 1 went into the "My Settings" tab and clicked on the default appearances button. He changed the background color to blue and the ink color to white and then correctly identified that changes applied to the entire blog as desired. The subject also decided to change the colors of a single entry afterwards to satisfy personal curiosity and this went flawless as well.

Subject 2 breezed through this task by following the default appearance link and then didn't even need to make any changes since he was satisfied with the gray and gold color scheme that he had applied to his first blog entry (our interface creates a preview based on the first blog entry's appearance). He simply had to apply the color scheme to the entire blog by clicking the button to save changes.

Subject 3 also had little trouble as she went under the "My Settings" tab and changed the background color to orange and the ink color to light blue. She commented that the colors were applied to all the posts as it changed the default appearance of the entire blog as intended.

Task 2

	Participant 1	Participant 2	Participant 3
Task Completion Time	0.5 min	0.33 min	1.0 min
# of Unexpected Behaviors	0	0	0

Statistics of Dependent Variables

• Mean Task Completion Time = 0.61 min
• Standard Deviation of Task Completion Time = 0.2843706501498821 min
• Mean # of Unexpected Behaviors = 0.0 unexpected behaviors per participant
• Standard Deviation # of Unexpected Behaviors = 0.0 unexpected behaviors per participant

Task 3: Embedding An Image

This "hard" task ended up being successful and not so "hard" after all. All three test participants had little trouble in understanding what was intended to be done since we brushed up the instructions and cleaned out the flaws since our low-fidelity prototype. This task resulted in similar, albeit slightly slower, task completion time to task 2; it was definitely much faster than the "easy" task 1.

Subject 1 inserted an image from the "My Drafts" tab by following the directions and clicking on the insert picture button. He chose "cat.jpg" from our predefined folder containing various images before publishing the entry successfully. The test subject commented that he would like to control the positioning of the picture in the blog entry rather than having it be centered at the top by default.

Subject 2 opted to start from scratch with the paper UI for this task. He drew a mountain at the bottom of the page with some writing and correctly followed the directions to leave adequate space at the top of the page. The test subject saved the entry to drafts and went under the "My Drafts" tab to insert the "cat.jpg" image. This subject also commented that he would like to be able to move the image around.

Subject 3 chose to embed an image into an existing image rather than start from scratch with the paper UI. The test subject chose "dog.jpg" but wasn't done and decided that she wanted to change to "cat.jpg" instead. In doing so, she commented that pictures can't be deleted if misselected or more than one desired picture is selected. She published the entry and the first image she embedded appeared under "My Blog" but the second image was stuck at the "My Drafts" tab.

Task 3

	Participant 1	Participant 2	Participant 3
Task Completion Time	1 min	0.5 min	1 min
# of Unexpected Behaviors	1	1	1

Statistics of Dependent Variables

• Mean Task Completion Time = 0.8333 min
• Standard Deviation of Task Completion Time = 0.23570226039551573 min
• Mean # of Unexpected Behaviors = 1.0 unexpected behaviors per participant
• Standard Deviation # of Unexpected Behaviors = 0.0 unexpected behaviors per participant

Overall Comments

Subject 1 wondered how he would insert an image directly from the paper UI in the future by following directions and leaving adequate space at the top of the page. The subject thought that the paper UI was clear enough except for directions on how to specify the location of the picture. Perhaps a bummer is that subject 1 didn't know about the right click functionality available and thus suggested to have buttons easily visible rather than have the right click menu or to have the cursor change from the pointer arrow to a hand or some other obvious symbol to denote a difference. He thought that its a lot of work to use the interface overall and would rather resort to typing everything up on the computer if possible as it may decrease his blogging efficiency. However, the subject does approve of taking Anoto and paper on the go being efficient and portable and the abilities of drawing and sketching as opposed to basic typing. The subject likes the right click but noticed that for now the insert text feature text area expands infinitely. Lastly, subject 1 commented that our GUI is consistent with our low-fidelity prototype as all the features on paper were all apparent.

Subject 2 commented that the overall GUI looked like MS Paint and would probably rather just type than to hassle with the Anoto as it would be faster. However, the subject commented that it could serve as a useful doodling pad or to jot down creative thoughts on the go and could be very useful if all the digital paper was available in a spiral notebook for portability. The test subject suggested to make the right-click functionality more obvious by using different cursors as a distinctive mechanism. He suggested we make "insert text" a button by itself rather than have it part of the drop-down menu as it appeared to be hidden. Subject 2 would have appreciated instructions on the paper UI telling the user to tick a box before transferring data to the computer.

Subject 3 mentioned that the mouse hover text was useful but she would like to see compact buttons to allow the user more space to see the blog entry on the screen rather than having to scroll down. The test subject was impressed with the preview box for changing colors as she stated that many current blogging interfaces out there (blogger, xanga, livejournal, etc) don't even have a real time module of similar nature. Overall subject 3 was impressed with our interactive interface and look forward to seeing our final product.

Overall Comments

	Participant 1	Participant 2	Participant 3
# of Unexpected Behaviors	3	3	3

Statistics of Dependent Variables

• Mean # of Unexpected Behaviors = 3.0 unexpected behaviors per participant
• Standard Deviation # of Unexpected Behaviors = 0.0 unexpected behaviors per participant

Discussion

What We Learned From The Pilot Run

From the three pilot run sessions, we learned that even that although most of our interface's functionality was relatively easy to pick up, the labeling in several places could be improved. The most common complaints centered on the participants not being able to find certain functions due to poor or non-existant labeling pointing to these functions. In addition, each participant's comments reflected on their tastes and priorities, as one stressed the dragging and resizing of pictures while another stressed the overall color scheme. Based on the participant's responses, we learned the following about our prototype:

• The labeling must be improved: This improvement must be made both on the paper UI and the application interface. The following is a list of places where participants cited a lack of labeling as a hindrance in finding the functionality that they need:
  • The Entry Editing pane: There is currently nothing on this pane that gives a clue as to the existance of a right-click editing menu. One participant didn't know about this menu until being told of it at the end of the session, while another only knew about this right-click editing menu because they participated in the Lo-Fidelity Prototype testing session as well. All participants stated that at the very least labeling should exist to let the user know of the right-click menu's existance.
  • The paper UI also suffered from poor instructions according to participants. It was not obvious from the directions on the paper UI how to go about inserting an image into an entry (when in the publishing process would there be a chance to insert a digital image?). In addition, participants were not made aware that if they try to write on the paper UI entries after saving that entry as draft/publishing the entry then the new strokes would become part of a new entry and not be added to the previous entry.
• Our application UI prototype had areas with wasted space. The entry editing pane has buttons that are laid out such that they are on top of one another. This causes the entry preview to not fully be visible without having to scroll down. In addition, the application's tabbed pane is smaller than the application's window frame, thus creating an unused border.
• Our idea for the functionality to set the default colors of the blog entries differened from the ideas of the participants. Currently, if the default color scheme (the background and/or ink color) is changed, then all entries in the "My Drafts" and "My Blog" tabs adopt this color scheme. Participants pointed out that they might want to keep specific color schemes of certain entries without having that entry adopt the default color scheme.
• Participants viewed object positioning and as important. Due to a lack of time and a pressing schedule, we didn't implement the functionalities of resizing objects (text boxes, pictures, ink strokes), positioning objets, and cutting/copying/pasting of objects for this prototype. Participants stated that they would like this sort of feature.
• Hassles of blog writing with Anoto outweighed the benefits of the Anoto system. Although participants felt that the process of writing blog entries with Anoto pens was useful for special characters and drawing, they still preferred typing text over writing with pen for most cases. Participants felt that although the process of jotting Anoto entries took too long and the current prototype was too restricting on what they could edit in their entries (as of now the only editing options are background/ink color change, insert picture, and insert text box).
• The prototype didn't provide enough feedback. One participant noted that it would be preferable for the entry editing panes to provide dialog boxes asking for confirmation to save and/or discard changes made.

What We Might Change For The "Real" Experiment

Had we performed a study with a much larger testing group, we would make the following changes to our experiment:

• Based on the general difficulty the participants experienced in finding the editing-right click menu, a label or cursor change to let the user know of the menu's existance would need to be added.
• Better directions would be provided on the paper UI to indicate to the user that a one of the boxes must be checked to transfer the entry to the computer, and also the nature of adding text and images in the entry publishing process.
• More specific instructions on what to edit in the entry will be made in task #1. Each participant in the pilot study chose what to edit in their entries themselves. In order to better compare the participants' number of errors and task completion time, it is better to define what features of entries the participants would need to edit.

What We Might Change In The Interface From These Results Alone

If the results of our experiment hold for a larger user population, we would make the following changes to our prototype:

• Eliminate wasted space
  • Line up the editing buttons in the entry editing pane
  • Increase the size of the tabbed pane to fill up any unused space in the application frame
• Remedy the effects on the draft and published blog entries caused by changing the default color scheme. Only entries that have the previous default color scheme plus all entries to be inputted into the application in the future will adopt the new color scheme. All entries with user-defined color schemes will retain these color schemes.
• Change the cursor when it is moved over the entry preview area of the entry editing pane
• Allow for the insertion of images in the "My Blog" tab
• Add an "Insert Text" button to the entry editing pane. This option is only available under the right-click editing menu in the present design.
• Allow the user to change to other tabs while editing and entry and provide prompts to save or discard changes made in the editing pane when the user wishes to leave to another tab. Currently, there is no feedback of this sort and the user must either save or cancel the editing changes made before being allowed to move to other tabs.
• Add instructions on the paper UI to make it clear that either the "Publish Entry" or "Save as Draft" boxes must be checked off to enter the entry into the 4Corners blogging application
• Add instructions on the paper UI to make it clear that the user will be able to insert typed text and/or images in the 4Corners blogging application after they send the entry to the computer.
• Implement as much of object cut/copy/paste, object resizing, and object positioning as possible. We define an object as ink strokes, inserted images, and inserted text. Thus far this has been a programming obstacle and it may not be feasible to implement all of these features of entry manipulation, but it should be attempted to implement as much of them as possible since participants desired these features.

Workload Breakdown

• Ramy (25%):

Study Observer: Ramy jotted down notes during the test, marking down the user's comments, thoughts and any moments of disappointment or confusion as well as recording any critical incidents along with their time. Ramy also kept a close watch over his stopwatch to note times of incidents and occurrences down to the exact second.

Pilot Usability Study report write-up: Ramy was responsible for the Introduction, Implementation and Improvements, and Consent Form sections of the write-up, as well as proofreading the entire write-up for accuracy near the end.

Prototype improvement coding: Ramy was responsible for changing the scroll bars to be inside the My Blog and My Drafts tabs instead of outside the tabbed panes. He also contributed to debugging and fixing minor issues with the application.

• Tabassum (25%):

Study Speaker/Facilitator: Tabassum walked the user through the demo (see Appendix), made them comfortable throughout and repeatedly prompted them to think aloud to generate more data for the observers. Aside from the greeting phase of the test, she was also the only person to talk to the users, in order to maximize impartiality. Whenever users had any questions or comments, Tabassum was the one to address them and made sure everything was absolutely clear.

Pilot Usability Study report write-up: Tabassum was responsible for the Method and Demo Script and Cue Cards sections of the write-up.

Prototype improvement coding: Tabassum was responsible for eliminating the possibility of clicking on other tabs while editing an entry in the My Blog or My Drafts tabs. This fixed one of the biggest bugs in the code. She also contributed to debugging and fixing minor issues with the application.

• Chen (25%):

Study Observer: Chen jotted down notes during the test, marking down the user's comments, thoughts and any moments of disappointment or confusion as well as recording any critical incidents along with their time. Chen also kept a close watch over his stopwatch to note times of incidents and occurrences down to the exact second.

Results Analyzer: Chen was responsible for taking all the observations and statistically graphing and analyzing them. To this effect, he also made sure to take the most extensive notes while observing the tests in order to compute the means and variances of the dependent variables (time it took users to complete a task, number of unexpected behaviors).

Pilot Usability Study report write-up: Chen was responsible for the Results and Charts sections of the write-up, as well as assigning Severity Ratings to the Raw Data Observations.

• Maksim (25%):

Study Observer: Maksim jotted down notes during the test, marking down the user's comments, thoughts and any moments of disappointment or confusion as well as recording any critical incidents along with their time. Maksim also kept a close watch over his stopwatch to note times of incidents and occurrences down to the exact second.

Pilot Usability Study report write-up: Maksim was responsible for the Test Measures and Discussion sections of the write-up.

Prototype improvement coding: Maksim was responsible for adding in the option to delete unwanted draft entries from the My Drafts tab, which we all thought would be a useful feature to add. He also contributed to debugging and fixing minor issues with the application.

Appendices

Charts

Cue Cards

Raw Observation Data with Notation of Critical Incidents

Ramy Test 1 Log:

11:08 Start, Demo

11:09 No problems with edit/comment

11:10 Understands tabs, demo

11:11 Checked friends tab after changing color, but appearance unchanged - problem?

Task 1

11:12:20 Writes entry

11:13:23 Stops

11:14:15 Checks entry, ink slightly garbled

11:15:10 Confusion about modifying, edits in paper UI instead of GUI, adds a line but this adds another entry (3. Minor usability problem)

Task 2

11:18:10 start

11:18:35 end - no problems, no confusion between edit/appearance

Task 3

11:20:40 start

11:21:05 end, chose cat picture, comment - should be able to move the picture (2. Cosmetic problem)

11:21:30 successfully publishes picture

After thoughts

11:22:35 Not obvious how to insert image from PUI

11:25:50 Edit Panel needs to be cleaned up, should make right-clicking obvious

11:28:00 Prefers typing but likes the interface

11:29:10 Finds "insert text", types "wow"

11:30:15 carriage returns out of blog field range

11:31:13 Thoughts on missing features: images

11:31:30 Done.

---

Maksim's Test Logs

Participant #1

General:

• In "My Blog" tab, identified correctly the Edit button functionality but thought that Comments button was for placing comments yourself (vs viewing comments for a particular entry)
• In "My Settings" tab, not sure if subject understood functionality of the "Upload User Picture" option

Task #1:

• Total Task Time: 5 minutes, 30 seconds
• Wrote short entry with a few simple hand-drawn pictures (sun, stick figure)
  • Time taken to write entry and check off "Save as Draft" box: 1 min, 3 seconds
• Clicked Edit for the new draft entry in "My Drafts"
• Participant asked how he could erase something in the entry while in the application (3. Minor usability problem)
• Participant then took the pen again and tried to add more writings to the entry that was just transferred, but when transfered to the computer these new writings went into a new entry (3. Minor usability problem)
• Noted that being able to erase parts of entries plus add more strokes to an already-entered entry would be especially nice to have

Task #2:

• Total Task Time: 39 seconds
• Went into My Settings->Default Appearances
• Chose Blue background, White ink
• Seemed to understand how changing appearance of one entry was different from changing global appearances of the blog

Task #3:

• Total Task Time: 1 minutes, 1 second
• Clicked on the add image button and selected cat.jpg
• Noted that he would have liked to resize the image and/or move it to a different spot in the entry (2. Cosmetic problem)
• Clicked Publish

Misc Comments:

• Might have been confused about how to go about inserting a picture when presented with the paper UI
• Wasn't aware that a right-click editing menu even existed on the Edit Entry screens. Directions need to be added to make users aware of this
• Wanted to have an Edit menu above the entry, not just a right-click edit menu
• Thought that having the cursor change when it is moved over the entry that is being edited is a good idea

Participant #3

General:

• Noticed that there were only three posts in both "My Drafts" and "My Blog" tabs
• Noticed that there were no "traditional" posts in either "My Drafts" or "My Blog"
• Pointed out that the number of comments for each post is not displayed

Task #1:

• Total task time: 6 minutes
• Said that the paper UI looks intuitive to use
• The dot pattern was a little distracting
• Wrote an entry with some hand writing, checked off "Save as Draft"
• Changed background color to white, also changed text color to dark color
• Pointed out that the presence of the right-click editing menu is not obvious - the participant knew about it only because they also participated in the lo-fi prototype testing (3. Minor usability problem)
  • To remedy this, it was suggested to add directions or having the cursor change in the editing entry pane to signify the presence of this right-click menu

Task #2:

• Total task time: 55 seconds
• Changed background color to orange
• Noted that the color scheme for all entries changed, but what if she wanted to keep the color scheme that they specified for the entry published in task #1 for that entry? (2. Cosmetic problem)
  • Need to remedy this

Task #3:

• Total task time: 59 seconds
• Added a picture of a dog

Misc Comments:

• Hovertext directions over the tabs seemed very useful
• Paper UI doesn't say how much room to leave on top for image insertion
• Edit buttons in the Edit panes are aligned in such a way that wastes space
  • Should line up the buttons
• The application frame is bigger than the tabbed pane, so space wasted
  • Should either make tabbed pane bigger or application frame
• Told that the user should be able to switch to other tabs while editing an entry, and the application should prompt with a dialog if the user wants to save or discard changes
• Noted that the real-time preview of the color scheme in the change background/ink windows is a very good part of the program
• Noted that the default color scheme settings preview starts with the color shcme of the latest published entry, which is not necessarily useful
  • Participant cited an example that they would make one entry with a dark color scheme to signify anger but wouldn't use this color scheme for other entries

---

Chen's Test Log: Test Participant 1 - Friday 11/10/2006, 11:10 AM

Demo:

• In browsing through the tabs, subject 1 encountered certain areas "yet to be implemented" such as the links to friend's recent entries under the "My Friends Blogs" tab, however he was able to identify the intended functionality correctly of giving the reverse chronological order of entries of the respective name clicked on
• The test subject actually performed task 2 while browsing through the GUI free form
• Identified the functionality of all 4 tabs correctly

Task 1:

• Total Task Time: 5 minutes (11:15 - 11:20 AM)
• Wrote an entry with a simple picture sketch - then saved this as a draft
• Docked the pen, loaded the pen data, saw that his writing appeared as the most recent entry under the "My Drafts" tab
• Clicked Edit for the new draft entry in "My Drafts"
• Confused about "modify" in the task 1 definition, subject doesn't believe he can erase strokes so he took the physical pen and paper and wrote down "add this line" before going through the whole data transfer phase again, obviously this saved as a new draft and he believes this is how he would do it. (ERROR 11:17 AM) (3. Minor usability problem)
• Subject would like the ability to erase/add rather than just changing colors (3. Minor usability problem)

Task 2:

• Total Task Time: < 1 minute (11:20 AM)
• Went into My Settings --> Default Appearances
• Changed the background to blue and the ink color to white - identified that the change applies to the whole blog correctly
• Subject also changed the colors of one entry correctly by hitting the edit button

Task 3:

• Total Task Time: 1 minute (11:21 AM)
• Inserted an image from "My Drafts" tab by following directions, chose "cat.jpg" from a predefined folder containing various images, published the entry successfully
• Subject would like to control the positioning of the picture in the blog entry rather than having it centered at the top by default (2. Cosmetic problem)

Overall Comments:

• For the future, subject wondered how he would insert an image directly from the paper UI by following the directions and leaving adequate space at the top of the page
• Thought that the paper UI was clear enough except for directions on how to specify the location of the picture - yet to be implemented (2. Cosmetic problem)
• Didn't know about the right click functionality for task 1 (4. Major usability problem: important to fix)
  • Suggested to have buttons easily visible rather than have the right click menu, otherwise have text reading "right click for more options"
  • Suggested to have the cursor change from the pointer arrow to a hand or other symbol to show difference
• Thought that its a lot of work to use the interface overall and would rather resort to typing everything on the computer if possible, said it maybe decrease his blogging efficiency, however subject does approve of taking anoto and paper on the go to be efficient and portable and the ability to draw and sketch compared to basic typing
• Subject likes the right click --> insert text feature but the text area keeps expanding infinitely (2. Cosmetic problem)
• Commented that our GUI is consistent with our low-fidelity prototyping on paper as the features are all there

Session concluded at 11:30 AM

Chen's Test Log: Test Participant 2 - Friday 11/10/2006, 2:30 PM

Demo:

• Subject identified that the "My Blog" tab allows for the changing on text, colors, editing colors of each entry individually, change of mouse pointer from arrow to hand on hover
• Subject identified that the "My Drafts" tab contains private entries that are not visible to the public
• Subject identified that the "My Friends Blogs" tab contains a list of links to friends and should be sorted by most recent entries
• Subject identified that the "My Settings" tab contains customizable settings
  • Manage account allows for the changing of password and email address
  • Change Default Appearance is useful for uniform settings throughout the entire blog
  • Security Settings would be to manage privacy in entries as well as blocking people
  • Thought the terms "security" and "privacy" to be misleading, are they interchangeable?
  • Why not insert picture through "publish"? (GOOD QUESTION, yet to be implemented) (3. Minor usability problem)

Task 1:

• Total Task Time: 6 minutes (2:42 - 2:48 PM)
• Wrote a quotation on paper --> published
• Docked the pen, loaded the pen data, saw that his writing appeared as the most recent entry under the "My Blog" tab
• Modified the entry by clicking edit --> changed the background color to gray and text to gold before saving the changes

Task 2:

• Total Task Time: 20 seconds (2:48 PM)
• Went into My Settings->Default Appearances, didn't need to change anything as he just had to apply the gray and gold color scheme to the entire blog

Task 3:

• Total Task Time: < 1 minute (2:49 PM)
• Subject opted to start from scratch with the paper UI --> drew a mountain at the bottom of the page with some writing (correctly followed the directions of leaving adequate space at the top of the page) --> saved to drafts
• Subject realized the new entry went to the "My Drafts" tab because he ticked the respective box, then successfully embedded the "cat.jpg" image
• Subject also commented that he would like to be able to move the image around (2. Cosmetic problem)

Overall Comments:

• Subject said that the overall GUI looked like MS Paint and would rather just type than to hassle with the anoto as it would be faster
  • However, subject commented that it could serve as a useful doodling pad or to jot down creative thoughts on the go
  • Could be very useful if all the digital paper was available in a spiral notebook for portability
• Subject suggested to make the right-click functionality obvious --> would have tried it out if he had a real mouse rather than the touchpad on a laptop (4. Major usability problem: important to fix)
  • Could try using different cursors as a distinctive mechanism
• Suggested to make "insert text" a button rather than part of the drop-down menu (2. Cosmetic problem)
• Subject assumed something like right-clicking was yet to be implemented because many other features were that way
• Feels like post-it notes with the default black ink on yellow background look
• Provide instructions on the paper UI to tick a box (3. Minor usability problem)

Session concluded at 2:55 PM

Chen's Test Log: Test Participant 3 - Sunday 11/12/2006, 3:47 PM

Demo:

• Subject was surprised that not many posts existed on the "My Blog" page with special interest that there were no "traditional" typed posts
• Number of comments is ambiguous --> numbering system in ( ) yet to be implemented (3. Minor usability problem)
• Subject identified that the "My Drafts" tab contains things written but not yet published to blog
• Subject identified that the "My Friends Blogs" tab is easy to identify which links are clickable by the underlines and mouse cursor changes
• Subject identified that the "My Settings" tab allows users to change username and password under the manage account button as well as to change the default appearance of the blog
• Subject commented that buttons look flat in general (limitation of the R3 paper toolkit) (2. Cosmetic problem)

Task 1:

• Total Task Time: 6 minutes (3:52 - 3:58 PM)
• Subject commented that the paper UI looks simple and intuitive with clear instructions
• Subject drew a sketch with some writing --> saved as a draft by ticking the appropriate box
• Docked the pen, loaded the pen data, saw that her writing appeared as the most recent entry under the "My Drafts" tab
• Modified the entry by clicking edit --> changed the background color to white and ink to gold before saving the changes

Task 2:

• Total Task Time: 1 minute (3:59 PM)
• Went into My Settings->Default Appearances, changed the background color to orange and the ink color to light blue
• Commented that the colors were applied to the all the posts as it changed the default appearance of the entire blog

Task 3:

• Total Task Time: 1 minute (4:00 PM)
• Subject chose to embed an image into an existing entry rather than start from scratch with the paper UI, subject chose "dog.jpg"
• Commented that pictures can't be deleted if misselected or more than one desired picture is selected (3. Minor usability problem)
• Published the entry --> first image went to "My Blog" but the second image chosen stayed at "My Drafts" (ERROR 4:00 PM)

Overall Comments:

• Subject said that the mouse hover text was useful
• Subject would like to see compact buttons to allow the user more space to see the blog entry on the screen, possible fix would be to line up the buttons in a row rather than column to take up less space (2. Cosmetic problem)
• Finite state machine --> go to a tab quickly and need a pop-up dialog box prompting user to save or cancel changes (2. Cosmetic problem)
• Subject was impressed with the preview box for changing colors --> allowed many options to change the colors
• Subject said a possible improvement would be to make the color of the preview to be the color of the most entries by default under the "My Blog" tab (2. Cosmetic problem)

Session concluded at 4:15 PM

---

Consent Form

            4Corners CS160 User Interface Project Consent to Participate in Research

                                      *Maksim Lirov
                                      *Ramy Ghabrial
                                      *Tabassum Khan
                                      *Chen Chang

This session is for the purpose of testing team 4Corners’ interactive prototype to gain accurate assessment and user feedback of our blogging system user interface. There is no physical risk, as you will be dealing with a graphical computer interface and making various selections. All data collected from today’s session will remain strictly confidential and be used for the purposes of our project only. Your identity will be protected and secure within our group. Should any discomfort or issues arise at any time and for any reason during this session, you have the option of backing out of the testing. We thank you in advance for your time and commitment towards this test.

Please sign here if you agree to the above and consent to be tested: ______________________

Demo Script

Greeters: Welcome to 4Corners Pilot Usability test. We appreciate your time.
Facilitator: We are students of cs160, User Interface Design and Development, class. We are building an application using the digital pen and paper technology. I will begin by explaining you what exactly is digital pen and paper technology and how does it work. The concept is that of using pen and paper as an interface to the digital world. A faint pattern of dots is printed on the paper which provides unique information about what gets written on them. The digital pen is embedded with electronics capable of capturing and storing time-stamped content. The digital copy of the content can be transferred to your computer by connecting the pen to it.
My team, 4Corners, is building a software application using this digital pen and paper technology. The main idea behind the project is to provide a portable and convenient interface to regular bloggers.

Facilitator: Today we have gathered here to conduct our pilot usability test. If you consent to participate, you will be asked to complete three different tasks using the application on this machine(points to a laptop placed on a desk. Before asking for your consent, I would like to mention that this is not your test. We are the ones being tested here. None of the things you say or do will be considered incorrect. Here is the consent form. Please read carefully and place your signature on the space provided if you agree to participate in this usability test.

• If the user agrees to the usability test and signs the consent form
  

Facilitator: Thank you for your consent. We appreciate it.

Facilitor: The application has two parts: the Paper Interface and the Graphical User Interface. This is our Paper UI (presententing the Paper UI of 4Corners Blog). This is a blogging pad which you can carry around anywhere along with the digital pen and write your blog entries on the go. Later when you have access to your machine, you can dock the pen in this cradle and your blog entires will get transferred to your machine.

Facilitator: (firing up 4Corners Blog Application on the machine) This is the second part of our application, 4Corners Blog graphical user interface.

Facilitator: Before beginning the test, I would like you to get familiar with the application. The blog entry will be written on 4Corners paper ui using the digital pen. The digital pen will be docked in a cradle that comes with the pen and the content that it captured i.e. your blog entry will be transferred to 4Corners application running on your system. 4Corners will allow you to modify, save, publish and share this blog entry. For the purpose of demonstrations, I will present to you the four main user interfaces of our application one at a time. This is not a test; however, we would appreciate it you can think aloud and say what comes to your mind as you look at each interface. For example, what do you think each of the component of the UI should do, what do you think should happen if you click on something, etc.

Facilitator: Do you have any questions you would like to ask before beginning the usability test?

Facilitator: As I mentioned earlier, you will be asked to accomplish three tasks using the 4Corners blogging application. The three tasks are written on individual index cards. I will present one task at a time. I cannot give you detailed instructions to perform the tasks but if at any point of time you have questions or you are confused, please say so because that will show issues with our prototype. My team members will be taking some notes based on how you go about doing the three tasks and will also record the time that it will take you to complete them.

I would encourage you to think aloud if you feel comfortable doing so. Remember, nothing you do or say will be considered incorrect. We are being tested here not you. Also, remember that you always have an option to withdraw from the test.

The test begins..

Please note that the facilitator repeatedly encouraged the participants to think aloud and asked questions in order to understand their thoughts. All our participants felt comfortable with this request and participated without any objections. At the end of the tests, rest of the team members also participated in asking questions and all our participants were glad to answer them.

---