From CS160 User Interfaces Fa06

Contents 1 Report Section 1.1 Group Members 1.2 Problem and Solution Overview 1.3 Tasks 1.3.1 Easy Task: Adding an Article 1.3.2 Medium Task: Highlighting (markup) a paper article 1.3.3 Hard Task: Sending the paper article to a co-worker with a message 1.4 Revised Interface Design 1.4.1 Changes from Low-Fi testing and Rationale Behind the Changes 1.4.2 Storyboard 1.4.3 Other Issues 1.5 Prototype Overview 1.5.1 Overview of UI 1.5.2 What was left out / Why 1.5.3 Wizard of Oz Techniques Needed 1.6 Prototype Content 1.6.1 Screenshots 1.6.2 Firefox Extension Code 1.6.3 Paper UI Generator Code 1.6.4 Paper UI PDFs 2 Presentation Section 2.1 Presentation Slides

Report Section

Group Members

Bryce: Implemented HTML to PDF to JPEG conversion for transferring web pages into R3-friendly image files, Tasks, Overview of UI

Robert: Implemented a Firefox Extension in XUL/Javascript to allow users to add URLs to a queue, then view and process that queue, Prototype Content, Revised Interface Design

Simon: Implemented paper UI using R3 to make a layout for the image files, Problem and Solution Overview, Tasks, Prototype Overview, Prototype Content

Vijay: Implemented paper UI using R3 to make a layout for the image files, Revised Interface Design

Problem and Solution Overview

The "NewsToGo" story so far...

The employees of San Francisco public relations firm "Text100" currently revolve their work around online news sources, rarely delving into paper sources unless when necessary. With everything on screen, Text100 may just be the model "paperless office". However, even in this environment, there are people who do not wish to read 100% of their work on a computer screen - they would much rather print out a paper copy of news articles to travel with, or to read with high-resolution ease. The problem, however, is how to do anything with those paper copies besides reading them. With online articles, Text100 employees could share news between themselves by shooting off an e-mail on the spot. How can they be as productive while holding paper?

Our solution: To create a seamless experience between the on-screen world and the paper world using Anoto technology, as applicable to the employees of Text100. Starting from a Firefox plugin that can keep a list of online news articles in memory, to an external application that processes these articles into printable Anoto-enabled PDFs, to a final application that can take in the data gathered from the paper and process it in ways analagous to how Text100 works today. Entitled "NewsToGo" - From sharing an article with a co-worker to saving facts for later review, working with paper copies of online news articles has never been so productive.

11/21/2006, 11:15 PM - Note from Robert: I accidentally edited this section when using parts of it on the Problem and Solution Overview for our Final Prototype. I've put this back the way it was at submission; you can check timestamps if necessary. Sorry if this causes any problem.

Tasks

Easy Task: Adding an Article

The User, using Firefox, has come across a news article that he/she would like to read, but does not have the time to do so. He/she would like to save it for later, because it deserves a further review. It might even be useful to some of her co-workers in another team. To save this article, he/she needs to use buttons (as added by a Firefox extension) to add the article on the screen to a 'queue' of articles for later.

Medium Task: Highlighting (markup) a paper article

The User has now obtained a print-out of some articles that he/she wanted to read over on paper. The paper is Anoto-pen-enabled, so the User knows to use his/her Anoto pen to mark the special regions. The User sees a portion of the article that is particularly interesting, and wants to highlight it for reference later. Alternatively, he/she sees a particular fact about a company that could be useful later, and wants to save it as part of a running list of "Fast Facts" that he/she is building.

Hard Task: Sending the paper article to a co-worker with a message

Once the user docks the pen, the messages formatted to combine the original text and annotations, which is then sent to the user's Outlook application. According to requests from Text100, the articles and annotations are now preceded by an overall message comment, which serves as the article summary and action list. These articles are sent to the listed recipients if the option was checked on the paper document and also sent as a draft to the author for reference. At this point, the user is also able to go back to the firefox extension and view the sparkler notes extracted from the document.

Revised Interface Design

Changes from Low-Fi testing and Rationale Behind the Changes

Between low-fi testing and the interactive prototype, only a few changes were made. Change A: We took out the extra dialog box that appears after you've added an article; this dialog box confirmed the article had been added to the queue. Originally we were going to make a checkbox with the extra notification to be able to not display it again, but we've come to the decision that it would be better to remove altogether.See reference A below. Change B For the SparklerNotes and NewsToGo tabs, we added a help button, and changed "Cancel" to "Close". We also arranged the buttons differently to take into account the extra new buttons. We want the window to be less condensed, but this is currently limited by our knowledge of XML, which before the interactive prototype was nothing. We hope to improve on this in the future. Change C Help dialog box added. Has basic information about what NewsToGo is, adding articles, and manipulating the queue.

Reference:

Reference A: Confirm button removed

Reference B-1: Old queue window

Reference B-2: New queue window

Reference C: Added Help Window

Storyboard

Adding, printing and removing articles

Annotating articles and emailing them to coworkers

Finding SparklerNotes data in articles, and managing the data after docking the pen.

Storyboards 2 and 3 take place after the first storyboard and may occur more or less at the same time. That is, the user can annotate the document and look for SparklerNotes data in it at the same time.

Other Issues

Low-fi limitations: We tried to address the limitations of our low-fi prototype by adding more icons and color to our extension, to help users better associate buttons with functions. However, the biggest limitation so far has been that we have just begun learning XUL and Javascript, and we were having issues with getting the extension to appear exactly as we wanted.

Constraints of the Anoto Pen: NewsToGo does not require more of the Anoto pen than what it was designed for; in fact, our project does not even use the bluetooth functionality. We realize that transmission of a user's annotations can only be made once the pen has been docked, and the user may be a long time in getting to home or work where they may be able to dock the pen. Bearing this in mind, we have aim to make the process of transitioning from data in the pen to an email in Outlook as smooth as possible. Unfortunately for now this is a "Wizard of Oz" feature, as R3 does not yet support batch processing, but we have familiarized ourselves with the process of sending email from Java, which would appear in Outlook for the user to make last minute revisions to before sending.

Non-standard Interactions: There are no non-standard interactions required by the user as we see it. That is, our program works as it should, and there are no user-level kludges involved in getting it to run properly on supported platforms.

Prototype Overview

Overview of UI

The entire User Interface of this application is divided between the on-screen portions, the backend portions, and the paper interface portion. Up to this point, we have implemented nearly all of the on-screen functionality (the Firefox extension) and the paper interface portion. The backend portion is lacking, and we describe its shortcomings later. At the very least, it can perform the PDF generation, producing the paper interface.

We will walk through the interface from beginning to end, while referencing the files in the section below, to demonstrate the extent of completeness of this application.

The User begins at the Firefox extension. [See Screenshots and Firefox Extension Code] The extension allows him/her to add an article (the current URL when a button is clicked) to a 'queue', which is stored in Firefox's memory (even between sessions, very much like a cookie that does not expire). The User can add as many articles (saved as URLs) to the 'queue' as he/she wants; when the User is finished and needing to review the queue, he/she clicks on a different button and is taken to a window holding the contents of the queue. From here, the User can choose to delete entries from the queue, close the window to go back to surfing & adding articles, or begin the PDF-generation process for the articles he/she selected in the queue. [See Paper UI Generator Code] The PDF generation takes quite a bit of time, but the end result is a complete set of PDFs that (when printed) become the paper interface complete with Anoto functionality. [See Paper UI PDFs]

The User can then pick up his/her stack of paper with the selected news articles carefully divided and overlayed with the Anoto patterns. The pages (paper UI) also contain tools that allow the User to interact with the paper, marking it up and performing tasks such as sending the article to a co-worker or saving particular "Fast Facts" for later. (See above tasks.) These tools are displayed as tap-able Anoto "buttons" and (perhaps later) Optical Character Recognition-enabled Anoto "fields" on each page. To highlight some text in the news article, for example, the User would first tap the Anoto pen on the "Highlight" button, then draw lines under or through the text he/she wished to highlight. Later on, if that article is saved or sent, it would be seen with bright yellow highlights where the User marked the article with the pen. To send an article to others, the User would fill out the "Message" field, "Recipients" field, and "Tags" field, and then check off the Anoto-box marked "SEND". All these actions would not happen until the User docked his/her Anoto pen, which would initialize our backend to process the gathered data.

From there, the experience is essentially done. The requested e-mails are sent, with recipients getting a highly customized copy of a news article (complete with tags, highlights, and commentary by the User), and any FastFacts are saved into the same data storage as the User's queue is. The user experience process is set to repeat the next time the User wishes to add more news articles to use NewsToGo with.

What was left out / Why

There are two features which are not present in our current interactive prototype due to the lack of batch processing on instructional machines, where the R3 development took place. First, we are not able to send the email since we cannot properly obtain all necessary data. However, we have fully thought through the algorithm in order for such action to be implemented when batch processing is enabled. Also, we are not able to populate the sparkler notes browser due to again the inability to capture the data. In response, we have identified libraries to do the necessary OCR work and have the visual interface in place for this portion of the project. Please see the Wizard of Oz techniques section for more information about these two omissions.

Mockups of what was left out:

Low fidelity mockup of Outlook email

High fidelity mockup of SparklerNotes tab

Code to migrate articles from firefox to Anoto compatible JPEG

Image:Html-pdf-jpeg-conversion-code.zip

Wizard of Oz Techniques Needed

The first place where Wizard of Oz techniques are needed is to bridge the gap between the Firefox extension and the R3 page generation. Both of these parts of the application are fully developed: the Firefox extension can properly retain selected web site information, and the R3 page generation is capable of turning the webpage into a series of R3 documents, given a URL. However, the invoking of the R3 page generation is not possible under current security restrictions in Firefox. Java cannot be directly called by Javascript in Firefox, even if the script is running locally in the extension (This is possible under Internet Explorer through directX however). Another area which requires wizard of Oz techniques is actually obtaining the pen data. Streaming is ill-suited for our project, since we require all annotations to be present at a single docking time since we do not have any way to retain and archive this information. During development, batch processing was not available on the instructional machines (we requested it to be installed), and thus the email generation could not be implemented. however, with batch processing in place, the implementation of this portion would simply requires creating an HTML string composed of the annotation images alongside the previously generated page jpegs, which would then be sent through Java's email facilities. The last area requiring Wizard of Oz techniques is the sparkler notes. We identified OCR libraries to make such conversion possible; however, the lack of batch mode hindered are ability to further implement this feature. The interface however is in place to display the information in the Firefox extension.

Prototype Content

There is no single README.txt that covers the entire setup process to get our prototype working, but the extension and the PaperUI zipped files have separate README's included with them. Start by following the instructions as you read this section.

Screenshots

NewsToGo Toolbar

NewsToGo Queue

NewsToGo SparklerNotes

NewsToGo Help

See the sample PDF files at the bottom for paper-based UI examples.

Firefox Extension Code

Viewing the source code: Unzip the file, and "un-jar" the jar file in the chrome directory to see the code.

Installing the extension: Note this is also in the README.txt file that you can view if you unzip the extension. Download this ZIP file, rename it to a ".xpi" file, and use Firefox to open it. Follow Firefox's instructions to install it, and you'll see the interface within Firefox in a new toolbar. Note that adding articles to the queue only seems to work in Linux, specifically tested with Ubuntu 6.10 running Firefox 2.0. With other operating systems such as Windows XP the extension installs and the UI appears, but adding articles is not supported.

Image:Interactive-Prototype-GGiD-NewsToGo.zip

Paper UI Generator Code

Download this ZIP file, unzip it, and import it into Eclipse along with R3. The process to set this up is nearly identical to how the HelloWorldMap project was set up. Please see the README.txt file within the ZIP file for further instructions.

Image:InteractivePrototype-GGiD-PaperUI.zip

Paper UI PDFs

After the Firefox Extension invokes the R3-based Java program, JPEG images and PDF files are created based on the article located at the URL (passed in). The PDFs are to be printed out - they are the paper UI that the user will take home with him/her. Here is an example set.

• Image:InteractivePrototype-GGiD-page0.pdf
• Image:InteractivePrototype-GGiD-page1.pdf
• Image:InteractivePrototype-GGiD-page2.pdf
• Image:InteractivePrototype-GGiD-page3.pdf

Presentation Section

Presentation Slides

Group GGiD had their presentation in 330 Soda at 12:30pm on Wednesday, November 1, 2006. The slides from the demonstration are in the archive file below. (Need Microsoft PowerPoint to view.)

Image:InteractivePrototype-GGiD-Presentation.zip

---