Assignment 4: Statistical Analysis of User Study Data

From Visualization Sp06

Contents

[edit]

Assignment Due Dates: Part 1 - March 6, 2006, Part 2 - March 14, 2006


Image:a4_bar.gif

In this assignment, we will conduct a graphical perception experiment as a class and you will be individually responsible for analyzing the experimental data and writing a report of your findings. The goal of this assignment is to better acquaint you with the methodology of controlled experiments, particularly analyzing experiment results.

[edit]

Part 1: The Experiment

The first part of this assignment asks you to take part in a graphical perception experiment. The experiment presents a series of data graphics and for each presentation asks you to quickly estimate which of two sets of graphic elements represents the greater quantity. The task could be to compare individual values or aggregates of values, using either a bar chart, pie chart, or table display. Each display sums exactly to 100 units. The y-axis grid lines on the bar chart condition are spaced at 0, 25, 50, and 75. Example stimuli are shown below.

Image:a4_bar.gif
The bar chart condition, with 7 data values and an A+B <-> C+D comparison.

Image:a4_pie.gif
The pie chart condition, with 7 data values and an A <-> B+C comparison.

Image:a4_table.gif
The table condition, with 4 data values and an A <-> B comparison.

The experiment has been implemented as a Java application, and can be launched using Java WebStart. To run the experiment, first make sure you have Java 1.4 or later installed (Java software can be downloaded for Windows, Mac OS, or Linux from http://java.com). Then just click the link below to run the practice trials or experiment. The software will be downloaded to your machine and run locally. You will need to accept a security certificate in order to run the software.

Run Practice Trials

Launch Experiment

The practice application consists of a block of 18 practice trials at a time. Afterwards, you can either try another practice block or exit the application (simply close the window).

The actual experiment consists of 180 trials, with rest breaks after each block of 60 trials. The experiment typically takes less than 15 minutes to complete. Hopefully you do not find this too tedious, as it is necessary to replicate the conditions enough times to get statistically significant results.

Please attempt to complete the tasks of the experiment honestly and as instructed, as this will result in a more valid and interesting data set for your later analysis. The experiment itself does not collect nor store any personally identifying information. When you finish the experiment, you will be given the option to submit your results automatically back to us. You will also be asked to enter in your name, so that we can give you credit for completing the experiment. Your name will be stored separately from the experimental data and we do not store any information allowing the two to be linked. If an error occurs while trying to submit your data, please copy and paste the provided results and e-mail them to us.

You must complete the experiment and submit a data set on or before Monday, March 6, 2006.

[edit]

Part 2: Experiment Analysis

The experimental data is provided as an aggregated, tab-delimited text file. Your task is to analyze the data using the appropriate statistical techniques and write a report on your findings. You are responsible for choosing 1 hypothesis of interest and performing a thorough analysis.

Experimental Data File

Here is the schema of the data file, a flat, tab-delimited text file:

  • id a unique participant identifier
  • task the comparison task, 0->A/B, 1->A/B+C, 2->A+B/C+D
  • reverse indicates if the comparions labels were reversed, false->A/B+C, true->C+B/A
  • parts the number of data values shown (either 4 or 7)
  • viz the visualization type, 0->bar, 1->pie, 2->table
  • targets array indices of the values being compared, listed in A,B,C,D order corresponding to the comparison task
  • values the actual data values visualized within the trial
  • time the participant reponse time, in milliseconds
  • error indicates the accuracy of the participant's response, 0->Correct, 1->Incorrect

Some of the questions you mightconsider are:

  • How does the number of values affect response time and accuracy?
  • How does the comparison type (A <-> B, A <-> B+C, A+B <-> C+D) affect response time and accuracy?
  • How does the display type (bar, pie, or table) affect response time and accuracy?
  • Are there any interactions between the above conditions?

Feel free to propose and investigate additional questions / hypotheses. Again, you are only responsible for investigating 1 hypothesis, but you are certainly encouraged to explore additional hypotheses.

Your report should include which statistical measure(s) you used and why, properly reported statistical results (e.g., including p-values and any other relevant statistical results or parameters), and accompanying text and graphics illustrating the result. You are free to use any statistical software package you like, including Excel, SPSS, or Matlab.

If you want a guide to performing an analysis, take a look at the Assignment 4 Example.

Your experimental data set must be submitted on or before March 6, 2006.
Your assignment must be posted to the wiki before class on March 14, 2006.

[edit]

How to create your wiki page

Begin by creating a new wiki page for this assignment. The title of the page should be of the form:

A4-FirstnameLastname.

Replace Firstname and Lastname with your real first and last names. You can create the page by entering a url of the following form into your browser:

http://eyeball.cs.berkeley.edu/cs294-10-sp06/A4-FirstnameLastname

To upload images to the wiki, first create a link for the image of the form [[Image:image_name.jpg]] (replacing image_name.jpg with a unique image name for use by the server). This will create a link you can follow that will then allow you to upload the image. Alternatively, you can use the "Upload file" link in the toolbox to upload the image first, and then subsequently create a link to it on your wiki page.

[edit]

Add a link to your finished reports here

Once you are finished editing the page, add a link to it here with full name as the link text. The wiki syntax will look like this: *[[A4-FirstnameLastname|Firstname Lastname]].


[add comment]
Retrieved from "http://vis.berkeley.edu/courses/cs294-10-sp06/wiki/index.php/Assignment_4:_Statistical_Analysis_of_User_Study_Data"