PilotStudy-Group:GGiD

From CS160 User Interfaces Fa06

Introduction

The System

NewsToGo is a multi-component system, which allows individuals to connect their own personally written ideas with a corresponding publication and share them with others. Intregrated into Firefox, a popular web browser, NewsToGo gives users the ability to choose articles as they encounter them on the internet and to later organize these selections for printing or deletion. The printing feature produces a formatted paper version of the article, extending the pen affordance of editing into digital realm. Using a “palette” of options, the user can perform numerous actions on the text, such as highlighting and commenting. In addition, many affordances of the computer are imported, such as email sending and data archiving. The Data archiving feature, known as FastFacts, provides users access to selections marked with tags at a later time through a browser on their computer.

Purpose and Rationale

NewsToGo’s purpose is to empower individuals in publication-driven professions, such as public relations agencies, to be productive away from the computer and office. Specifically, NewsToGo brings new functionality to paper printouts, a common medium for reading articles when not at the desk. The solution merges common processing tasks associated with articles, such as sharing and annotation. The integration of these features into the paper allows the user to fully complete a task such as article reading, while eliminating numerous unnecessary tasks, such as transcription of notes and article formatting for distribution among coworkers. Thus, NewsToGo enables the user to accomplish all tasks associated with an article on a single platform without any redundant steps.

Pilot Study

Our purpose in conducting the pilot study is to provide concrete feedback on our implementation, rather than just the overall workflow. Key aspects not previously available, such as a precise look and feel, can now be properly tested. Also, specific measures can be obtained at this time, including the amount of time to complete a task and the number of errors a user incurs while using the program. Ultimately, we hope to uncover any issues that were not apparent through our low fidelity prototyping.

Implementation and Improvements

Firefox Extension Changelog

The most recent version of our Firefox Extension is here:

Image:Newstogo pilot.zip

Unzip to view the code, and change the file's extension to .xpi to install. In addition to the changes below, it now has been tested and works on Windows XP SP2 with Firefox 2.

• Larger and less cramped layout for the main window: about twice as wide, with all the buttons being on one row now.
• Continued work on calling a local Java method from the extension: the support for this now exists for this in our extension; we're able to call methods from non-package based classes that can depend on jar files. We're having problems with packaged files, which we unfortunately need to work.Some problems include the fact that the extension comes bundled with R3, which is very large, and that R3 cannot be made into part of a jar file, which would make our lives much simpler.
• Made the FastFacts more Wizard of Oz friendly and less hardcoded: the filtering function and remove functions actually work now; an entry was added to the context menu add FastFacts (again as a Wizard of Oz mechanism).
• Fastfacts list more readable: each element in the list now displays what kind of FastFact it is before the FastFact itself.
• NewsToGo button name in toolbar: has been changed to NewsToGo Center make its function more understandable.
• Message in dialog box for adding articles changed: it now includes the title of article, and the message itself has been changed to make it more understandable.
• Consistency in labels: Any lingering reference to SparklerNotes has been changed to FastFacts.
• Help Section: has been made more comprehensive.
• Changed most icons to Windows or Outlook icons and have attempted to match icons for functions in the extension with a similar function they would have played in Outlook, etc.
• Cleaned up code so function names better reflect what they do and gave xul/javascript proper indenting

Paper UI

In the Interactive Prototype, we had a complete Paper User Interface with all the regions and layout completed. However, this prototype proved to be incredibly difficult to actually print. We had managed to print a couple of the cover pages, but no member of our group was successful in printing even one copy of one of the article pages. Although we tried printers from a wide variety of locations and many techniques for converting the PDF files, the sheer size and complexity of the articles with the Anoto pattern overlay proved to be too much for any printer or any system we tried. We really needed a workable copy of an article for this portion of the project, so we attempted to change the paper interface to suit the printers better. In the end, shortening the article length and only using half of the page space per article page resulted in a PDF that was printable, although it still took about a half hour per page to print. The major elements of our Paper UI remain intact, although the space for comments is smaller.

We will probably keep the current format of our Paper UI through the end of this project, as it is the only feasible way we can produce prototypes for use in testing.

Batch Processing

The most significant portion of our Interactive Prototype that was lacking was the post-pen-docking portion. Our goal was to have the proper e-mail formatted and sent upon dock, as well as the associated FastFacts that the user recorded being processed and stored in a location where the Firefox extension could access it. As time passed and after discussion with the staff, we realized that this goal was too lofty and decided to scale down for this pilot test.

As a result, we focused our efforts at trying to process the XML data that would come in after docking, and rendering a .JPG image of that data. Because the PaperToolkit (R3) does not currently provide a high-level way of dealing with this data, we had to find a third party XML parser and customize it to meet our needs. So far, we have been able to extract X and Y coordinates from the XML data that the pen produces, and then render a drawing using those coordinates on screen as well as output the drawing into a .JPG file. We can do this per page that the user writes on, but only for the Comments regions (as we output .JPG files in the Comments region’s exact size). In addition, the coordinates given by the pen are relative to the per-page Anoto pattern used in the PDFs, so it is impossible to discern which region the pen writes on at any given time - we simply take in all of the data and crop out the region which we believe is the Comments region. This has proven only half-successful, as we oftentimes get .JPG files that crop out parts of comments. The on-screen Swing UI used for this portion is able to show all pen strokes, however.

Here is an updated version of our prototype's code portion. README within has also been updated.

Image:PilotStudy-Group-GGiD-UpdatedPrototype.zip

In later development, we plan to take the generated .JPG files and attach them to e-mail. If formatted correctly, we will get the desired effect of having handwritten comments right alongside article content, mirroring the Paper UI and thus the paper copy of the article. Difficulties now include discerning what parts of the XML can clue us in to which region the pen is reporting from, so we can begin detecting “clicks” of the pen on the tools in our article markup ‘toolbox’.

Method

Participants

Participant A is a 26 year old account manager, in charge of coordinating account executives and coordinators working for her clients. Her major in engineering and time in the technology industry allows us to understand how a technology adept individual would view and use our program, highlighting any conflicts or missing features that would benefit the professional user. Her managerial role would make her a key email recipient of work generated through our application, helping us get valuable feedback from the receivers end.

Participant B is a 29 year old account manager, who currently runs the major accounts of Adobe and Informatica. Like Participant A, her managerial role provides crucial feedback on the usability of the sharing features. In addition, she does not hold the same computer expertise as the other participants, allowing us to evaluate our system from a beginner’s standpoint. Lastly, Participant B commutes to and from work on MUNI, thus making the work affordances of our solution a perfect fit.

Participant C is a 24 year old account executive, previously an account coordinator when we interviewed her for the lo-fi prototyping experiment. Her job entails a high volume of publication reading and sharing amongst her coworkers. She also requires recent relevant information to be ready at any time for her clients, therefore making her a perfect benefactor of the fast fact capability. In addition, she commutes from San Francisco to Emeryville, and therefore could utilize such tool to be productive during this unutilized time away from the office.

Apparatus

Our interview setup consisted of a laptop (Lenovo Thinkpad) to facilitate testing of part 1. There was a wired mouse attached for a more desktop-like experience. In particular, the computer was running the Firefox web browser in an environment mimicking the Windows graphical interface, since the plug-in was having issues with Windows. Also, we printed out pre-selected articles, because printing would take too long for the duration of the interview. The actual Anoto pen was also present with the docking station hooked up to the computer, to simulate the actual docking action involved with the later stages. Additionally, we used a MacBook Pro to capture the interview on video for future review. We interviewed our test group in their office to maintain the work environment where most of the computer interaction would take place.

Tasks

These benchmark tasks are a slightly revised version of the three tasks we used for the lo-fidelity prototype and the interactive prototype. They are adapted for the pilot test and the functional elements of our project that were available at the time of the test. Essentially, the easy, medium, and difficult task are the same - but we changed how they are organized and described in an effort to make things more clear and accurate. The three main tasks outlined below generalize the specific types of tasks we had users do. Specific tasks are subsections of the general task.

Easy Task: Queuing Articles, Managing the Queue, and Printing Articles

The User, using the Firefox web browser, has come across a news article that he/she would like to read, but does not have the time to do so. He/she would like to save it for later, because it deserves a further review. It might even be useful to some of her colleagues in another team. To save this article, he/she needs to use buttons (as added by a Firefox extension) to add the article on the screen to a 'queue' of articles for later.

Queuing the articles

The user browses news websites, looking for news relevant to their current work. For example, they may currently be working with EBay, and would like to find news about EBay to pass on to publishers writing a story about EBay. The user clicks around on websites such as CNN.com, Time.com, and Google News. When a suitable article is found, the user clicks an “Add to Queue” button on the Firefox toolbar, and confirms the addition.

Managing the Queue

The user may have added too many articles to their queue for them to feasibly read all on paper, or added duplicate articles to their queue. Clicking another button on their Firefox toolbar would bring them to an interface showing their queue, along with ways to manage it. The user uses the interface to remove the articles he/she doesn’t want.

Printing Articles

In the final stage of this task, the user is ready to go and needs the printout of the articles he/she selected. Using the same interface (that showed the queue), the user selects the articles he/she wants on paper, and hits a button to print them out. The user is now ready to take the paper versions with him/her.

Medium Task: Annotate a Paper Article

The User has now obtained a print-out of some articles that he/she wanted to read over on paper. The paper is Anoto-pen-enabled, so the User knows to use his/her Anoto pen to mark the special regions. The User sees a portion of the article that is particularly interesting, and wants to highlight it for reference later. Alternatively, he/she sees a particular fact about a company that could be useful later, and wants to save it as part of a running list of "FastFacts" that he/she is building. In addition, the user would like to make comments or take notes on the article in his/her own handwriting. Using the paper interface and its provided toolbox in the upper right-hand corner, he/she marks up the article to accomplish these objectives.

Hard Task: Send the Article, and manage FastFacts

The user must use the paper interface’s “Cover page” to send the associated article to colleagues. The cover page is full of detailed instructions and regions in which to write, and it is mostly up to the user to understand the instructions provided and use the interface to send the article. Trusting that the actions requested will happen, the user can dock the pen and move on to managing the FastFacts back at the Firefox extension interface. The user would like to see their FastFacts, filter them by category, and perhaps print them out onto a FastFacts sheet for easy access. [No Anoto functionality on the printout.]

Procedure

Before we had any test users come into our testing environment, we made sure to set up all of the tools we needed to test with. We prepared a specific location for a test user to sit at, first in front of the laptop used for the Firefox extension pilot test and later in front of the Paper UI and the Anoto pen dock (which was, conveniently, plugged into another laptop running our batch processor). This ensured, we hoped, that the test experience would remain consistent between tests and that all users would face the same interface in the same conditions.

When the test user came in, we would welcome her, have her sit in the pre-designated seat, and begin to go through the script we had prepared. We introduced ourselves, explained where we were from and took the steps needed to set up the experiment. We made sure to promise confidentiality, encourage the user to speak freely and move at their own pace, as well as asked for permission to conduct the tests as well as measure variables as the test was conducted (timing using a stopwatch, for example). When the user had consented, we would begin.

Our prototype interface was relatively straightforward, so we didn’t want to give too much of a demonstration in order to test the intuitiveness of our UI. We would explain the overall purpose of our project, and tell the tester the ‘phases’ of the test, but ultimately would just task him/her with our three tasks one at a time and observe how the tester would perform.

For example, in the first task, we told the user to use the Firefox web browser (with our extension installed) to seek out a news article he/she would like to read off-screen, and use the added buttons in the Firefox toolbar to queue up the article for printing. Because there were only two buttons that our extension added to the Firefox toolbar, it was not difficult for most users to figure out how to add articles to the queue. Likewise, they usually could find the queue and print the articles without our guidance or any previewing by us. Our roles were to keep the user on track to completing the given tasks, and make sure that they completed them thoroughly (i.e. actually doing some management of the articles in the queue instead of bypassing that and going straight to printing).

After printing, we would shift the user over so that they would be looking at a pre-printed paper copy of an article, and move the laptop into the hands of one of us so that they could prepare the Wizard-of-Oz steps for the last task. In the second task, the users were less prepared, and we did have to walkthrough the paper UI for them. A brief description of the layout of the pages was in order, since the Cover page was a feature unique to our project. Also, the ‘tap-able’ button tools needed to be explained, but we did not go so far as to tell them how exactly to annotate the paper using those tools. Similarly, on the Cover page, the concept of the ‘check-box’ needed to be explained as well. [Text instructions were printed on the Cover page, and we encouraged the users to read those before they began as well.] Tasking the users with annotating the article was simple enough, but more errors were made here than anywhere else.

Moving on to the third task, we had the user dock the Anoto pen. Here, the XML file would be generated on the second laptop, and we would run our batch processor and show the user the generated .JPG file, explaining that it would have been placed in an e-mail alongside a copy of the article. During this time, another member of our group would take the paper UI that the user wrote on, and transcribe any FastFacts made back into the Firefox extension. Thus, when the User went back to the Firefox extension after the batch processing demonstration, they would see their FastFacts in the extension and could manage them, as we tasked them to do. Again, the interface of the Firefox extension was simple enough so that the users usually could figure it out without us having to demonstrate it for them.

Upon conclusion of the three tasks, we asked for overall impressions, feedback, and comments on our system. We thanked the user for her time, and took time organizing the data gathered and resetting the testing area.

Test Measures

The first test measure was the time needed to complete a particular task. Due to the variability in times from each participant, dependent on factors such as how many comments they wrote, this measure focused purely on static tasks. For example, for the Firefox extension, this test measure would account for the time necessary to add an item to the queue. This data is important to our study, since we would like to compare this time against procedures from previous solutions, such as simply printing out the document. If the tasks of our solution end up taking significantly longer than its predecessors counterparts, users may not be as inclined to use the new feature.

Our second test measure is the mistakes made during each task. The integration of our solution into common applications like Firefox make the users inclined to assume the system will work through certain actions. Also, with the use of a pen, many of the participants may believe that the same affordances map over. However, due to the limitations on processing the returned data, not all actions are valid, such as writing in the printed text regions and certain forms of underlining. Seeing these mistakes will help us identify them.

The number of questions asked is also a crucial measure for our project. Portions of NewsToGo were designed to be immediately usable without any training, and other tasks (such as using the Anoto pen) are sufficiently documented to allow users to quickly learn how to accomplish the necessary tasks. This measure will highlight the failure of both these intended facilities.

Lastly, we measure the number of unsupported features the user attempted to utilize during the interview. This measure is distinguished from mistakes, since the user is mentally believes such tasks can be accomplished under the system rather than such actions are appropriate to complete a given task. The complications between merging pen and computer affordances lead to many features being left out. However, due to the vision of the project, the user may truly believe some functionality is available.

Results

Results of the tests (1 page)

 Time: 55 sec 
 Mistakes: 0
 Questions: 1
 Unsupported Features: 1

 Time: 207 sec
 Mistakes: 1
 Questions: 0
 Unsupported Features: 2

 Time: 81 sec
 Mistakes: 0
 Questions: 0
 Unsupported Features: 1

 Time: 114 sec
 Mistakes: 0
 Questions: 0
 Unsupported Features: 1

 65 sec

 Time: 173 sec
 Mistakes: 1
 Questions: 2
 Unsupported Features: 1

 Time: 197 sec
 Mistakes: 1
 Questions: 0
 Unsupported Features: 1

 Time: 184 sec
 Miskakes: 0
 Questions: 0
 Unsupported Features: 1

 Time: 185 sec
 Mistakes: 1
 Questions: 1
 Unsupported Features: 1

 10 sec

 Time: 25 sec
 Mistakes: 0
 Questions: 0
 Unsupported Features: 0

 Times: 132 sec
 Mistakes: 2
 Questions: 2
 Unsupported Features: 3

 Time: 70 sec
 Mistakes: 1
 Questions: 1
 Unsupported Features: 1

 Time: 76 sec
 Mistakes: 1
 Questions: 1
 Unsupported Features: 1

 44 sec

Discussion

Based on our quantitative results, it seems users are able to go through the first task relatively quickly. Of course this task time really depends on how many articles the user wants to add; we just had them add one article. People seemed to have a lot of misconceptions about our paper interface based on the number of unsupported features they tried to use. We may need to make the instructions more comprehensive on this area. The third task was the most "Wizard of Oz"-esque, so there were a lot of questions and suggestions, but few errors. As detailed below, we would ideally revamp this task based on user input. Although the second task took the longest for our users, this is expected as they actually had to read an annotate a news article. The time taken here really is a function of the size of the news article (users were each given articles of approximately the same size), and smaller articles of course would have shrunk the time. Additionally, this second task is not necessarily a bottleneck to the third task, as most users will be doing the second task "on the go" and will not be able to start the third until they are back at a computer. If the pilot study were to hold true for a larger group, which it may for the data discussed just now due to the consistency of the data mentioned between users, then we definitely know what we need to work on.

Given how much feedback we got from our Pilot Study, and how many ideas we have, I will disseminate the information in a kind of outline, discussing each element. Some of the ideas are more practical or realistic than others, and thus I will assign each one of three categories: Definitely fix (D) - this item is either very high priority or may be a lower priority but simple to fix. May fix (M) – It isn't critical we get this fixed, but it would be good feature. Implementing it would take some work, and whether or not it gets implemented would depend on time constraints. Wishlist (W) – these are good ideas we heard but either would replace existing functionality or would take a lot of work to implement. Given this, it is unlikely they will be implemented, but they are things we would ideally include in the final version. As it turns out, in a quick update, all of the (D)-class ideas were implemented since originally writing this. Below are the remaining ideas and their estimated feasibility.

Ideas for Task 1

• Scrollbar for Help Box (M). The help section has gotten really long; however, we're having issues with getting a scrollbar in the window that works right. We hope to have this working by the end of the project.
• Checking for printer friendly pages (M). Users in the pilot study did not try to add the printer friendly page of an article to tho queue without prompting. All users mentioned the awareness of the existence of such versions of articles later though. It might be nice to have a check (perhaps check the url for any set of strings, such as print/printer/etc), and if the page does not pass the check, have a dialog box asking if they want to add a possibly non printer friendly page. Of course the window would have a checkbox for not displaying the message again, but would have served its purpose of letting the user know we are intending for them to add printer friendly pages.
• Buttons that appear pressed when they are clicked (M). Currently buttons in the extension have no visual feedback if they are pressed; unless a confirmation window comes up, they will not know whether or not it was pressed. This may be a limitation of XUL though.
• Dynamic population of articles in the queue (M). If a user leaves the view queue window open and then adds another page, it will not be visibly reflected on the queue until they close the queue window and reopen this. This is fairly important, but the Firefox extension already is testing our very limited knowledge of XUL and Javascript, so it may not be fully implemented.
• Keyboard navigation for selecting articles in the queue (M). For a lot of applications users can select neighboring items by holding shift and hitting the appropriate arrow button. We noticed a user trying to do this, but our interface does not support this. We will see if XUL supports this.
• Ability to preview articles in the queue (W). If a user has a lot of similar articles they may not remember which is which, or the pages may even have the same title. A preview of sorts may be necessary. Perhaps they could click on the article's URL to have it open the article, or a small pictorial preview could be displayed for a selected article, or (perhaps the best idea) have the tooltip for each article be the first 10 or so words of the article. Implementing any of these ideas would be challenging given time and knowledge constraints of the knowledge.

Ideas for Task 2

• ”Action Items” (M): We would like a user to be able to check a box or boxes on the bottom of the screen that will generate a subject title for the email. The check boxes would correspond to priority of the article or action that needs to be taken, such as sending a response about the article to a client, or simply how soon the article needs to be read. Keywords corresponding to these priorities and actions would appear in the subject of the email along with the company (or companies) the article corresponded to (users said the actual title of the article was largely irrelevant). Obviously all of this requires a significant amount of work, and we have more integral parts of the project that currently don't work but this would depend on. It is, however, something we would like to at least partially implement.
• OCR and subcomponents (W): OCR itself seems unlikely to be implemented in our prototype. If we create a UI framework for it though, it seem likely we will need letter-boxes – that is, each letter of the email address is written in a box for OCR to recognize. Our users in testing would write somewhat randomly and haphazardly on the page, which probably would make OCR more difficult than it has to be.

Ideas for Task 3

• Rethinking FastFacts (M/W): The list interface on the FastFacts tab really was not designed to be practical with fewer than a hundred or so articles (the filtering feature helps make this number higher than it would be with a plain list). However, users could easily have hundreds, perhaps thousands, of FastFacts they accumulate over time. Currently it would be very difficult to implement an iTunes like search for contents of FastFacts, but we have added headers in the list for each group of items to make it easier for users to browse facts since interviewing users for the pilot study. What we really want to do is have only the FastFacts headers (such as Firefox or Adobe) show up, and they can double click these headers and it will open up a new window with all the FastFacts for that company/product/etc. This would be ideal as it mimics the interface they have on their company's server that stores FastFacts. However, this immediately brings up the question of why we have FastFacts stored on the extension to begin with, if users are used to looking them up on a server. We may just end up having the FastFacts from their most recent annotations available, which they can upload to a server through the Firefox extension, and then remove them from local storage on Firefox.

Workload breakdown

Project Work

This grid shows the relative amount of work each member of group GGiD has been doing on project work. We try to divide the workload evenly, and it was easiest to do so with each person generally having his own 'module' or parts of modules to control.

This Writeup

Rob: Implementation and Improvements, Discussion, Appendix:Raw Data
Vijay: Results
Bryce: Introduction, Method, Test Measures, Appendix:Demo Script/Instructions
Simon: Implementation and Improvements, Method, Appendix:Consent Form

Appendices

Demo Script / Instructions

Introduction & Consent

Thank you for taking time out of your schedule to participate in our pilot study. Your input is helpful for the improvement of our project’s usability. We will be having you complete numerous tasks. Do not feel hesitant to proceed if you are uncertain; the feedback gained during this interview is a reflection of our system’s performance, not yours.

In addition to your feedback during the study, we need to also report back information about the demographics of our participants. All this information will be kept anonymous; however we would like to make sure you are aware of this disclosure. In addition, if you do not mind, we would like to record this experiment on a video camera so that we may further analyze the data collected.

Here is a copy of a consent form we would like you to read over and sign:

Image:GGID Consent Form.pdf

Background

Our project is to allow the productivity of work away from the computer; specifically, allowing you to share your ideas and extractions from articles with your other team members. The solution relies on three main components: The web browser, the paper interface, and the email client. Our time today will be focused on accomplishing a task dealing with each of these aspects.

A brief overview: the system works by you selecting articles within the web browser using our extension. Then the pages are printed out for you to edit. These annotations are then sent out to the people through your email client.

Tasks

Now we will begin with the tasks. Feel free to think aloud while you complete these tasks.

Task 1: First, add the article that we have loaded on the screen to print in NewsToGo and then proceed to print it. Before you print it, try deleting items that you have presumably added in the past from the queue.

Task 2: With the paper interface, go through and add comments, highlight, and select FastFacts. When you are done, fill out the information on the front page for sharing this data with others and categorizing for yourself.

Task 3: Now that you have synced the pen, your task now is to view your fast facts notes. In particular, you want to make sure you can filter through them and also select one you want to view in more detail.

Raw data

User 1

• Statistics
  • Task 1: Overall time: 55 seconds, Mistakes: 0, Questions: 1, Unsupported Features: 1
  • Task 2: Overall time: 2 minutes 47 seconds, Mistakes: 1, Questions: 0, Unsupported Features: 2
  • Task 3: Overall time: 1 minute 21 seconds, Mistakes: 0, Questions: 0, Unsupported Features: 1
• Task 1
  • Tended to add non-printer's version;sometimes these versions did not have a toolbar in the window and user was not aware that it is possible to right click to add an article when toolbar is not available
  • "If I have a whole lot of articles than how do I know what article is what" - Asked because title didn't show up after first article (fixed now).
  • Doesn't think people are necessarily aware of shift-selecting.
  • Clicked on single articles at first, seemed to be expecting it to work without having to shift click.
  • Given above, kind of interesting she didn't click on Help button. She did say she saw it though..
  • She added an article while the view queue window was still open, and then was confused when it didn't show up in the box. This is because the extension does not support dynamic list population right now.
• Task 2
  • Inclination was to underline and then hit tooltip
  • Some confusion with Add/Confirm dialog because she had to squint at it a while. The message in the dialog has been tweaked a little (see changelog) since then.
  • "What is the difference between FastFacts and Highlighting" - seems like she skimmed through the instructions on the first page.
  • Thought it would be good if comments showed up next to the image of the highlighted article page in the email, rather than below them. Kind of makes sense, because otherwise there's a gulf of execution between how the user thinks their comments will appear later and how they actually will, which may lead to some user errors (ie they write a comment right next to and regarding something they underlined thinking people will know what they're talking about, but when the comments show up below the page in the email others reading it will have no idea).
  • Emailed to aliases rather than actual email addresses
  • She wrote the email addresses/aliases somewhat randomly on the page - OCR if it were theoretically implemented would be easier with letter boxes.
• Task 3
  • User said she would have liked to have seen an interface that looked more like the following:

Firefox
 -Fastfact 1 about firefox blah blah..
 -Fastfact 2 about firefox blah blah..
Adobe
 -Fastfact 1 about adobe blah blah..
 -Fastfact 1 about adobe blah blah..

User 2

• Statistics
  • Task 1: Overall time: 2 minutes 53 seconds, Mistakes: 1, Questions: 2, Unsupported Features: 1
  • Task 2: Overall time: 3 minutes 17 seconds, Mistakes: 1, Questions: 0, Unsupported Features: 2
  • Task 3: Overall time: 3 minutes 4 seconds, Mistakes: 0, Questions: 0, Unsupported Features: 1
• Task 1
  • Like User 1, User 2 tended to Add the normal article instead of the printer friendly version, the opposite behaivior of what they do at work normally and the opposite behaivior of what is needed for our project.
  • At first did not know if she should click Add To Queue or NewsToGo to add an article.
  • "I don't really know what the NewsToGo button would do". Attempted to fix since the pilot study by changing the label to NewsToGo Center.
  • "Do I need to highlight the text of the article before clicking add or do I just add it?".
  • User tried to use Shift+DownArrow to select multiple articles.
• Task 2
  • Error over how to highlight and FastFact the same line.
  • Immediately associated the categories box with FastFacts
  • User talked a lot about how she would like to see "Action Items" implemented, which basically adds to the email what the readers should do in response. It is unlikely we will be able to implement this in time.
  • Also mentioned something about doing actions regarding the reporter of the article (adding them to a database) which is kind of related to the above about Action Items.
• Task 3
  • "Seems like it would be hard to sort through a laundry list of FastFacts"
  • Suggested FastFact would follow an Outlook-like interface; the list would only contain FastFacts by category (such as company or product) which could be double clicked. This would cause a window to pop up with all the FastFacts for that category.

User 3

• Statistics
  • Task 1: Overall Time: 25 seconds, Mistakes: 0, Questions: 0, Unsupported Features: 0
  • Task 2: Overall Time: 2 minutes 12 seconds, Mistakes: 2, Questions: 2, Unsupported Features: 3
  • Task 3: Overall Time: 1 minute 10 seconds, Mistakes: 1, Questions: 1, Unsupported Features: 1
• Task 1
  • Slight hesitation after queueing up article (in between the two main steps of this task), although she had previously been informed about the queue in the previous interview. Brings into question how intuitive a separate queue is for users acquainted with the normal unqueued printing facilities.
  • In the two steps themselves (adding/selecting from queue), She had no questions and completed each task without mistake
• Task 2
  • "Am I supposed to mark it with the pen?" There appears to be a disconnect between the pen's role in the paper functionality. Instructions with more obvious correlations between the pen and action may help
  • She started underlining text as a first instinct without selecting Highlight from the palette first. We should possibly explore setting a default palette for the text area, such as the more commonly used feature (in this case it would be highlighting).
  • She tried to write notes only to herself in the text of the page. The fact we do not have a toggle to turn off palettes makes it currently impossible to distinguish non-inclusive data.
  • Tried to use highlight feature by underlining; however, we had concluded that capturing this type of data would be hard through this action. Also, points out that the affordances of normal pens carry over and we need to understand how much preconceptions affect the user's comprehension of the actions on hand
  • She bullet pointed her comments as if to separate them, brings up uncertainty of what she felt bulleting would do
• Task 3
  • "Are fast facts on the front page the same as the facts labeled on the on the article pages" – the naming of the different fields need to be clarified. Fast Facts on the front page does not imply the container/label action that will take place on enclosing fast facts.
  • "If I were to double click it would it expand to show it to me in the normal program?". We did not intend for such type of action
  • Did not immediately notice the filter option, only thought she could double click an article.
  • After seeing filter, didn't immediately realize she had to click the filter button afterwards

Interface Changes

Firefox Extension Changes

New Add Confirmation Window	New FastFacts Window	New Queue Window	Old queue Window
New Help Window	Old Help Window