Pilot Usability Study

From CS 160 User Interfaces Sp10

Jump to: navigation, search

Due: before class on April 21st

65 Points



The goal of this assignment is to learn how to perform a simple usability test and to incorporate the results of the test into design changes in your prototype. In the real-world, this kind of "pilot" study would be used to redesign your experiment before running the study with a larger pool of participants.


You will be performing this test using the latest version of your interactive prototype. As you are working on the pilot study you should also be working to add more functionality to your prototype based on the feedback you received in your interactive prototype presentation. You should include a brief bulletted list in your study write-up of any changes made to your prototype.


You will need to find three unique participants (i.e., volunteers who are not in this class and who aren't your friends) to work through your benchmark tasks. Also remember that it must be voluntary. You should get the participants to sign an informed consent form and obtain other demographic information (e.g., age, sex, education level, major, experience with your type of tasks and application, etc.) You should avoid using subjects who participated in your lo-fi tests, unless it absolutely impossible to find others. If you use one or more of the same subjects, justify this choice in your write-up.

Benchmark Tasks

Your test will use three tasks. They should include 1 easy task, 1 medium task, and 1 difficult task. These tasks should give good coverage of your interface; if they don't then this is a good chance to redesign your tasks. These tasks do not need to be the same as the tasks in your lo-fi prototype study.

Measures and Observations

Although it will be hard to get statistically significant bottom-line data with only three participants and a rough prototype, you should measure some important dependent variables to get a feel for how it is done (i.e., task time, # of errors, etc.).

You should concentrate on process data. For example, you should instruct your participant to think aloud. You should make a log of critical incidents (both positive and negative events). For example, the user might make a mistake and you notice it or they might see something they like and say "cool". Set up a clock that only the observers can see (one or more of you should observe), and write down a log containing the time and what happened at that time when a critical incident occurred.

If you happen to have access to a video camera, it is fine to use it (but make sure your subjects consent) -- note the time that you start taping so that you can find your critical incidents later on tape. You may wish to use an audio recorder if you don't have a video camera. Screen capture utilities like Camtasia may also be useful for capturing the screen of a computer during your study.


You will give the participant a short demo of the system. Do not show them exactly how to perform your tasks. Just show how the system works in general and give an example of something specific that is different enough from your benchmark tasks. You should write-up a script of your demo and follow the same script with each participant.

The participant will then be given task directions for the first task that tells them what they are trying to achieve, not how to do it. When they are finished, you will give them the directions for the next task and so on. Each participant will perform all 3 tasks. You will want to keep the data separate for each task and participant.


You must report your results (values of dependent variables, summary statistics, and summaries of the process data) and in the "Discussion" section you should draw some conclusions with respect to your interface prototype. You should also say how your system should change if those results hold with a larger user population. This should be the most important part of the write-up. We want to understand how you would fix your system as a result of what you observed.


Your group should turn in one paper (before lecture on the due date) and on the Wiki. Your write-up should follow this outline with separate sections for the top-level items (number of pages per section are approximate). It should be at most 5 pages, plus appendices and sketches that describe what you did.

Introduction (5 points)

  • Introduce the system being evaluated and state the purpose and rationale of the experiment (1 paragraph)

Implementation and Improvements(15 points)

  • Bulletted list of feature improvements since the interactive prototype (1/2 page at most). The descriptions do not need to be detailed.

Method (10 points)

  • Participants (who -- demographics -- and how were they selected) (1 paragraph)
  • Apparatus (describe the equipment you used and where) (1 paragraph)
  • Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
  • Procedure (1 paragraph) describe what you did and how

Test Measures (5 points)

  • Describe what you measured and why (bullet points encouraged)

Results and Discussion (25 points)

  • Results of the tests (1/2 page)
  • What you learned from the user study (1/2 page). Document any changes that you plan to make in your prototype as a result of the study.

Appendices (5 points)

  • Materials (all things you read --- demo script, instructions -- or handed to the participant -- task instructions)
  • Raw data (i.e., entire merged critical incident logs)


Uploading Images

To upload images to the wiki, first create a link for the image of the form [[Image:image_name.jpg]] (replacing image_name.jpg with a unique image name for use by the server). This will create a link you can follow that will then allow you to upload the image. Alternatively, you can use the "Upload file" link in the toolbox to upload the image first, and then subsequently create a link to it on your wiki page.

Hand in Printout in Class

Print your assignment and hand it in at the begining of class on April 21st.

Add Link to Your Group's Page

Edit your group's page to add a link to a new wiki page for this assignment. The wiki syntax should look like this:

[[PilotStudy-Group:ExampleGroup-FirstNameLastName|Pilot Study (FirstName LastName)]]

Again replace ExampleGroup with your group's name and FirstNameLastName with your name. Look at Group:ExampleGroup for an example. Then click on the link and enter the information about your assignment. Be sure to clearly address everything mentioned in the writing guidelines above. Writeups for all group members should be posted here.

Add Link to Your Finished Assignment

One you are finished editing the page, add a link to it here with your group name as the title link. The wiki syntax will look like this: *[[PilotStudy-Group:ExampleGroup|Group:ExampleGroup]]. Hit the edit button for this section to see how I created the link for the ExampleGroup.

[add comment]
Personal tools