PilotStudy-Group:Jigsaw

From CS160 User Interfaces Fa06

Contents 1 Introduction 1.1 Introduce the system being evaluated 1.2 State the purpose and rationale of the experiment 2 Implementation and Improvements 2.1 Fully implemented features 2.2 Implemented since Interactive Prototype 2.3 Improved since Interactive Prototype 3 Method (10 points) 3.1 Participants 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures (5 points) 4.1 General Measures 4.2 Negative Incidents 4.3 Debrief Questions 5 Results (10 points) 5.1 Time Per Task 5.2 Comments 5.3 Incident Analysis 6 Discussion (15 points) 7 Workload breakdown (5 points) 8 Appendices (5 points) 8.1 Demo Script(cut-n-pasted from Word document) 8.2 Project One-Sheeter (cut-n-pasted from Word document) 8.3 Informed Consent Form (cut-n-pasted from Word document) 8.4 Raw data (i.e., entire merged critical incident logs)

Introduction

Introduce the system being evaluated

The system our group has developed, named Jigsaw, is a software application to be used together with the Anoto pen which enables software and/or Web UI designers to integrate their initial paper sketches of a design onto a computer and into a format ideal for presentations. The design process with Jigsaw starts with the designer creating their sketches with the Anoto system. Each page of the sketches would be identified as either a Sitemap, Workflow, Wireframe or Note by checking the corresponding check box on the top of the Anoto enabled paper. From there the designer would start the Jigsaw application and create a new project at which point the Anoto pen data would be uploaded and organized into Sitemaps, Wireframs, Workflows, and Notes automatically. With the designers sketches imported, organized, and laid out for them they then have the ability go link and animate the sketches based on the conceptual model. Once the linking is accomplished, the design of the software/website is ready for presentation and/or output into other file formats.

State the purpose and rationale of the experiment

During the Pilot Usability Study we will accomplish the following:

• Determine which aspects of Jigsaw's design is confusing, time intensive, unintuitive, redundant, and any other issues that were not taken into consideration by the group during design and implementation of Jigsaw. It is the latter that perhaps would provide the most insightful feedback as it would be issues that were not previously considered. The data recorded during all three usability studies should provide consistent and insightful information regarding problematic features and tasks.
• Obtain both positive and negative feedback regarding the 3 tasks that we will have testers perform. Users will be asked to perform an easy, medium, and difficult task during the study. These 3 tasks were chosen to give the tester a good understanding of all the features and functionality of Jigsaw while at the same time guiding a user through out the actual design process where a finished product results.
• Get feedback from testers on what they: liked/disliked, think needs improvement, think is not necessary, think should be added to Jigsaw. This step in the usability study is to reinforce the data recorded during the study. But it is also to understand more a users needs and incorporate them into Jigsaw.

Most importantly, this usability study going to determine an accurate description of trouble spots in our application. Then, with that data, we can target such trouble spots in the application and incorporate specific changes.

Implementation and Improvements

Fully implemented features

The following is a list and brief explanation of features that are and have been fully implemented in Jigsaw since the Interactive Prototype.

• Unique and printable Jigsaw forms. Forms were created in PDF format with Anoto enabled active regions for sketch area and check box area.
• Create, Open, and delete Jigsaw projects. The ability to create, delete, and open Jigsaw projects in the Jigsaw application via a file menu option.
• Importing pen data. Importing of Anoto pen data into Jigsaw via streaming capabilities and drawn into images categorized by image type (Sitemap/Workflow/Wireframe/Notes).
• Navigation. Buttons to navigate between different categories (Sitemap/Workflow/Wireframe/Notes). These serve as the main navigation for a Jigsaw user.
• Page mapping. One-to-one mapping and page tracking of the physical pages a user sketched on and the image that is drawn in Jigsaw. This is accomplished by the unique page id given to every page and displayed semi-transparent on the displayed sketch in Jigsaw.
• Auto resizing of newly created pages imported into Jigsaw. When a user sketches a new page with the Anoto system, the pen data is streamed into Jigsaw and a new image is drawn into the correct category. If there is already pages in a certain category, the window will automatically re-size to display all images in thumbnail fashion.
• Creation of hotlinks. Hotlinks are used to link different portions of a page to another page (ie. Workflow to Wireframe).
• Drag and Drop. Drag and drop of pages onto other pages to create hotlinks (see previous bullet). A user can click and drag any page in any frame (there are 2 frames in Jigsaw) and drag it onto any page in the opposite frame thereby creating a hotlink on the page that was dragged onto.
• Active hotlinks. When a user clicks on a hotlink the associated linked pages are highlighted in the opposite frame. A double click on the hotlink will zoom in on the associated linked pages.
• Page highlighting and zooming. When a user clicks on any page it is highlighted. When a user double clicks on any page, that page is zoomed to fill the full frame window.
• Keyboard shortcuts. Shortcuts for every menu option were created to enhance productivity. The common Ctrl+O for open and the others were implemented.

Implemented since Interactive Prototype

• Linking wireframes to other wireframes. Wireframes can now be dragged onto other wireframes to create hotlinks. This reinforces the selection of an option on one wireframe that leads to another wireframe.
• Project Saving. The ability for a user to save their projects at any point and later open it just as any program allows.

Improved since Interactive Prototype

• Hotlink manipulation. Hotlinks can now be created, re-positioned, re-sized, and deleted. Resizeablity is useful when users want to cover an exact portion of their page with a hotlink.
• About Dialogue. Users should be able to get information on Jigsaw and their creators!
• Improved highlighting. A better approach to highlighting was taken which now creates a semi-transparent color overlay when a user selects and highlights a page.
• Navigation. Previously, when a users double-clicked a hotlink to zoom a page in the opposite frame the category button for that frame was not updated to reflect the new view; this was fixed.

Method (10 points)

Participants

Participant selection has always been difficult given our professional user pool and lack of resources in terms of compensating users for their time. We tried to select test participants who have a background with Web-based applications.

Marvin Doe: Marvin Doe is an experienced Web/UI designer with a graphic design background and 7+ years of professional experience. He has recently joined a stealth start-up as a senior visual designer, and maintains a freelance project designing an interface for an audio plug-in module.

Earl Doe: Earl Doe is a former Web applications software engineer who now works as a software architect at a major bank. In his professional life, Earl writes specifications/requirements and often functions as a liaison between technology and business personnel. Earl has over 10 years industry experience.

Lana Doe: Lana is a visual designer with over 5 years experience in technology. She has a strong graphic design background, and has a fine arts background, having started as print designer. She presently works as a freelance consultant for a Web solutions consultancy.

Apparatus

The following materials were used to conduct each test:

• Laptop, mouse, Anoto pen, USB bluetooth adaptor: used to run the Jigsaw application
• Sheets of custom Jigsaw Anoto-ready paper: used as the paper UI to the Jigsaw application.
• Test briefing sheet: each participant received a 1-sheet test briefing describing the Jigsaw application, general experimental goals, a list of tasks to work through, and debrief questions. Available in the appendix.
• Informed consent form: Available in the appendix.
• Demo script: A list describing the short demo given to participants before each test subject begins to perform the experimental tasks. Available in the appendix.

Tasks

Our user tasks were the ones selected for implementation in the hi-fidelity prototype. Below are the user tasks, with a breakdown of the steps (and application features) required to complete each step. These steps were also listed in the test briefing one-sheeter given to test participants at the beginning of each experiment.

1. Creating workflows/wireframes

1. Create new project
2. Draw bookstore workflow
3. Draw wireframes for each part of workflow
4. Save project

2. Linking wireframes to workflows

1. Link wireframe to each part of workflow

3. Export to standard format

1. Export diagram to JPG

4. Delete hotspot and diagram

Procedure

Each test begins with a greeting and introduction of team members. Our experiment called for a minimum team of 1 facilitator/greeter and two observors, one to take general notes and one to log incidents with timestamps.

The test participant is then given the experiment one-sheeter to read while the team sets up the application and enable Anoto pen streaming (usually on a spacious desk or table). We decided to provide test subjects with the one-sheet briefing to ensure consistent user base knowledge of the application - even though one of our participants was more knowledgeable about the application since he had participated in previous studies.

After the test participant finishes reading the one-sheeter, he is given a brief demo by the facilitator, following a script. The demo is designed to give the participant a quick introduction to the system without being too specific. As stated in the one-sheeter, the user is encouraged to explore the application UI.

After the demo, the user is asked to work through the tasks listed in the one-sheeter. As soon as the user indicates he/she is ready, we begin to time the tasks and log incidents.

When the user has completed all the tasks, we discuss the debriefing questions with him/her. We then thank the user and the experiment concludes.

Test Measures (5 points)

General Measures

• Time per task. Total time to complete all tasks.
• Negative comments per feature, positive comments. (note the target of the compliment)

Our general measures allow us to get an overall feel for how users perceive the application. In such a small sample size, we do not expect to draw any real conclusions from such measures from time per task, but it is an easy measurement to make and may provide some insight into the relative difficulties of each task.

Negative Incidents

• Moments when user appears "stuck" or confused (user pauses to think). (mark timestamp)
• User misnavigation (looking under wrong menu to find a specific feature) or surprise at application behavior (doubleclicking expecting a different application response). (mark timestamp)
• User is so "stuck" that he/she has to ask a question. (mark timestamp)

We count the above incidents and note timestamp to measure 1) overall intuitiveness of interface (fewer errors can be said to mean a more intuitive interface); 2) if frequency of errors decreases or increases as time goes on (how easy interface is to learn and retain - signs of good mapping and UI organization); 3) if specific areas of the application are "problem areas" - if different users consistently log a high number of errors while performing the same task or using the same feature.

Debrief Questions

1. What did you like best/worst about the application? What did you find most useful?
2. Suggested improvement to existing?
3. Export to EPS, single file project format, 'remote control' presentation, regular presentation mode (invisible hotspots and wireframe to wireframe view in same pane) - which would be most useful?
4. Suggested additional features

Our debrief questions allow the test participants an opportunity to give us free-form feedback about their experience and additionally, provide information about how to best prioritize features for future implementation.

Results (10 points)

Time Per Task

This data is not particularly useful given the small sample size and informal nature of the test (sometimes participants would pause to give comments). However, it may be interpreted as a rough indicator of the difficulty of each task.

Name	Creating workflows/wireframes	Linking wireframes to workflows	Export to standard format	Delete hotspot and diagram
Earl Doe	8 min and 45 sec	1 min and 1 sec	2 min and 1 sec	55 sec
Marvin Doe	13 min and 45 sec	2 min and 45 sec	6 min and 26 sec	32 sec
Lana Doe	16 min 4 sec	2 min 19 sec	5 min 23 sec	2 min 10 sec

Comments

Name	Positive	Negative	General
Earl Doe	Liked that he could work with sheets of paper in any order, and switch freely between them.	Creating a project was confusing, no feedback. The user started drawing before choosing paper type, since it was unclear if user should check a box first. There is no way to go back and erase if user made a mistake. There is no feedback after import is done. He expected a note of something like “Saved” or error message. User deleted a wireframe page but the hot spot associated to that page remained on the screen. No undo feature.	Question about what “workflow” means. Confused about pen vibrating
Marvin Doe	Liked that he could work with sheets of paper in any order, and switch freely between them.	Confused about creating new projects (suggested that it first creates an empty projects and you can save it later). When adding a new page, the button doesn't change to the associated layer. When the project is saved, the “*” is still there. User wants to zoom closer. Drag cursor doesn't appear if you drag too fast.	He wants a delete button on the screen. So he can select a page and click on that delete button.
Lana Doe	Pen/paper and application syncing is "cool". Diagram navigation and zooming after creating hotspots is "cool".	Confused about creating new projects, not sure what initial dialogue is for. Not sure what to do after new project is created. During export, confused about whether she was naming file or directory. Dialogue box unclear.

Incident Analysis

Below is a table that gives distribution of critical incidents on a per task basis. Each "X" represents an incident.

Task	User appears "stuck"/confused	User misnavigation or surprise at application behavior	User needs to ask a question
Creating workflows/wireframes	XXXX	X
Link wireframes to workflows	X	X
Export to a standard format	X	X	X
Delete hotspot and diagram		XX

Discussion (15 points)

We planned to use the pilot run as a way of finding out what improvements need to be made. By observing our test subjects, we were able to see how our application might be used by "real" users. We saw what they liked, what caused problems, and what could make things easier for them. While developing our application, we had in mind the general direction that we were working towards, as in new features to implement. In our post-experiment discussions with our subjects, we got feedback on these potential features, as well as new ideas that we hadn't thought of and bug reports.

Of our planned features, a single project file and presentation mode were the most requested. Our team knew that there would be issues with our current folder-based project system. We had predicted that our users would have problems with it, and we were right. This is a high priority bug fix for our next version. The presentation mode also seems like a natural feature to add. Being able to double click on a hotspot and have it open up in the same pane seems like a smoother flow while presenting. It also opens up new prototyping possibilities when linking wireframes to wireframes. Our subjects agreed with us that this could be a very useful feature. We were surprised to see little interest for our remote control and export to EPS features. While we thought the remote control could be a good application of the Anoto pen, our subjects preferred to use the computer if given the option. Our subjects also told us that JPG export was enough. Even our designer subjects felt no need for EPS, given the type of drawings most likely to be created with our application.

Our test subjects also gave us some ideas for other items to implement. All of our users complained about the general lack of feedback at certain points in the application. Since this is a key part of good UI design, we need to add this. One user suggested that we add more extensive panning and zooming features. Originally, we had this, but had taken it out early on. We thought it would be more confusing than helpful to have freeform navigation, but our users seem to like (and are used to) the freedom.

We noticed a strong correlation in how users responded and worked with the application to the applications that they used day-to-day. Keeping this in mind, any additional features or fixes we make will try as much as possible to be in line with UI conventions already established by common applications like Adobe Photoshop, Illustrator, or Microsoft Powerpoint.

Finally, we ran into a few bugs. Some we had encountered before and some we hadn't. Our team had been familiar with a bug that causes the images to load upside-down. Unfortunately, the bug appeared during one of our test sessions. We aren't sure of the cause, but we hope to fix it before the next release. One of our users pointed out that the selected button doesn't change even if the layer does, upon importing a new page. A user also pointed out that the drag cursor doesn't appear if you drag too quickly. Finally, deleting a page doesn't delete the associated hotlinks. All of these should hopefully be fixed for our final release.

Workload breakdown (5 points)

• Siu Pang Chu - Participation as observer in experiments. Results write-up.
• Scott Friedheim - Intro and Improvements write-up.
• Yen Pai - Participation in experiments, experiment one-sheeter write up, Method and Test Measures write-up, Results write-up.
• Patrick Rodriguez - Participation in experiments, Discussion write-up.

All: Experiment design, experiment results discussion and conclusions.

Appendices (5 points)

Demo Script(cut-n-pasted from Word document)

1. Open existing project.
2. Navigate between each of the different sections (Sitemaps, Workflows, Wireframes, Notes).
3. Draw a mark on a piece of paper to demonstrate correlation between numbered diagrams on screen and numbered pieces of paper.
4. Demonstrate how to create and resize a hotspot.
5. Demonstrate walking through a workflow once some hotspots have been created.

Project One-Sheeter (cut-n-pasted from Word document)

Introduction

You are about to test the Jigsaw application, a Web design conceptualizing tool using Anoto digital pen technology. Anoto digital pens use specially patterned paper to digitize your pen strokes.

Jigsaw is an application which allows you to digitize your paper drawn diagrams and notes, organizing them as Sitemaps, Workflows, Wireframes, and Notes.

Once diagrams are digitized, you can create “hotspots” (links to other diagrams) within each diagram. Clicking on a hotspot will show the linked diagram. In this way, a sitemap can be linked to workflows, or workflow elements can be linked to wireframes, or wireframes to other wireframes.

The Jigsaw application receives your digitized pen strokes over a streaming Bluetooth connection, so strokes are immediately visible within the application.

Each diagram is given a number that corresponds to the number on the specialized Anoto paper. Once a diagram is digitized, you can add to it by drawing on piece of paper with the corresponding number.

A Jigsaw team member will be doing a quick demo of the application before we begin. As this is an user interface test, the demo’s goal is to give you a general idea of how the application works and is not meant to address specific how-to questions.

You are meant to explore the interface, so do not worry if you cannot find a feature or find something particularly confusion – in fact, please be vocal about any perceived UI issues. We are testing the effectiveness of the UI, not testing your ability to figure out a confusing UI.

User Interface Tasks

You are asked to perform the following tasks. Please be vocal about what your thought processes as you work through the task list:

1. Creating workflows/wireframes

     1. Create new project
     2. Draw bookstore search workflow
     3. Draw wireframes for each part of workflow
     4. Save project 

2. Linking wireframes to workflows

     1. Link wireframe to each part of workflow 

3. Export to standard format Export diagram to JPG

4. Delete project items Delete a hotspot Delete a diagram

Debrief Questions

1. What did you like best/worst about the application? What did you find most useful?

2. Can you suggest any improvement to the existing features or UI?

3. Export to EPS, single file project format, 'remote control' presentation, regular presentation mode (invisible hotspots and wireframe to wireframe view in same pane) - which would be most useful?

4. Can you suggest any additional features to make the application more useful?

Informed Consent Form (cut-n-pasted from Word document)

Consent to Participate in Research CS160, Group Jigsaw

Researchers: Siu Pang Chu, Scott Friedheim, Yen Pai, Patrick Rodriguez

Project: Our project is a tool that utilizes the Anoto digital pen and paper system to help facilitate the initial design stages of user interfaces (specifically online applications). Designers will be able to draw workflows and wireframes on specially printed sheets of digital paper. These sketches will then be imported into our software, where they can be associated and linked with each other. The project can then be viewed as an interactive presentation.

Purpose: To test a prototype of our project and gather usability feedback.

Procedure: You will be participating in a casual session with three of our researchers. We will give you a few tasks to be done with our prototype and you will attempt them to the best of you ability. You may ask us any questions about your tasks and you should feel free to comment on any stage of the process. The total testing time should be between 30 minute and 1 hour, with some extra time at the end for final comments.

Invitation: As a professional designer, we appreciate the informative feedback that you may be able to provide to further improve our product. Risks: There are no risks directly related to participating in this research.

Thank you for your time.

I agree to participate in Group Jigsaw’s prototyping session:

Signature

Date

Raw data (i.e., entire merged critical incident logs)

Saturday Testing: Earl Doe

1. Time per task: 1. Task 1: Creating workflows/wireframes (Start: 00:00, End: 08:45)

Positive Comments: Liked that he could work with sheets of paper in any order, and switch freely between them.

Negative Comments: Creating a project was confusing, no feedback. The user started drawing before choosing paper type, since it was unclear if user should check a box first. There is no way to go back and erase if user made a mistake.

General Comments: Question about what “workflow” means. Confused about pen vibrating

0:06 stuck on how to create a project.

2. Linking wireframes to workflows (Start: 8:45, End: 9:46)

General Observation: The user tried to select a box in workflow and drag to a wireframe page at the beginning.

8:46 The user tried to select a box in workflow and drag to a wireframe page at the beginning.

3. Export to standard format jpeg (Start: 9:46, End: 12:00)

Negative Comments: There is no feedback after import is done. He expected a note of something like “Saved” or error message.

General Comments: Question about what is "selected."

4. Delete a hot spot and a page (Start: 12:00, End: 12:55)

Negative Comments: User deleted a wireframe page but the hot spot associated to that page remained on the screen. No undo feature.

Debrief Questions: 1. Positive: simple, easy to use. Liked the zoom function. Negative: can’t erase, confusing to figure out things first time around, not much feedback 2. Improve the create function 3. presentation mode ( first priority)

Testing ends at 17:00

Sunday Testing: Marvin Doe

1. Task 1: Creating workflows/wireframes (Start: 00:00, End: 13:45)

Negative Comments: Confused about creating new projects (suggested that it first creates an empty projects and you can save it later). When adding a new page, the button doesn't change to the associated layer. When the project is saved, the “*” is still there. User wants to zoom closer.

0:05 stuck on how to create a project.

2. Linking wireframes to workflows (Start: 13:45, End: 16:30)

Positive Comments: Zooming after creating hot links is nice.

Negative Comments: Drag cursor doesn't appear if you drag too fast.

3. Export to standard format jpeg (Start: 16:30, End: 22:56)

The wording on “Export Selected as jpeg to target folder” is confusing. He expected just "Save".

17:00 The wording on “Export Selected as jpeg to target folder” is confusing. He expected just "Save".

24:00 user question: did the export process finish?

4. Delete a hot spot and a page (Start: 22:56, End: 26:28)

Comments: The user wants to just drag the box out of workflow layer to remove it. He wants a delete button on the screen. So he can select a page and click on that delete button.

25:57 The user wants to just drag the box out of workflow layer to remove it.

Debrief Questions:

1. Likes: Ability to write on paper and the real time response from the screen. The page pop up after choosing the type. Zoom functionality. Features work as expected. Numbers on hotspots. Dislikes: Save, open, new project are confusing to work with.

2. Improve: want to have a status bar that shows the pen is still connected to the application. And able to export to .pdf format. Allow for more panning and zooming.

3. High priority: single file project format Low priority: remote control, EPS format, presentation mode.

4. other features: right click menu, status bar, help massager, color preferences( able to choose background color , box color)

Testing ends at 40:00

Monday Testing: Lana Doe

1. Task 1: Creating workflows/wireframes (Start: 00:00, End: 16:04)

Positive Comments: Pen/paper and application syncing is "cool".

Negative Comments: Confused about creating new projects, not sure what initial dialogue is for. Not sure what to do after new project is created.

0:24 confused about new project dialogue

1:00 stuck after new project is created. Needed to ask what to do.

2. Linking wireframes to workflows (Start: 16:04, End: 18:23)

Positive Comments: Diagram navigation and zooming after creating hotspots is "cool".

16:29: a little unsure what to do, we reminded her of the drag functionality.

3. Export to standard format jpeg (Start: 18:23, End: 23:46)

Negative Comments: confused about whether she was naming file or directory. Dialogue box unclear.

19:02: stuck on how to export file once selected.

4. Delete a hot spot and a page (Start: 23:46, End: 25:56)

23:50 misnavigation: user tries right-click menu, which does not exist. Figures out delete using delete key.

Debrief Questions:

1. Likes: Zoom and hotlinking functionality. Pen technology is "cool". Dislikes: New project and export features are a bit unusual and confusing.

2. Improve: New project and export dialogues. Right-click menus might be helpful.

3. High priority: single file project format (to eliminate confusing dialogues) Low priority: remote control, EPS format, presentation mode.

4. Other features: right click menu (as suggested above), improve export dialogue, allow export of entire project, have application "straighten" drawn lines.

Testing ends at 35:00

---