Usability and Evaluation

From Visualization Sp06

Lecture on Mar 2, 2006

Slides

Readings

  • Chapter 9, How to Plan Single-Variable, Multiple-Variable, and Converging Series Experiments. In Doing Psychology Experiments. Martin (handout)
  • Chapter 12, How to Interpret Experimental Results. In Doing Psychology Experiments. Martin (handout)
  • An empirical comparison of three commericial information visualization systems. Kobsa (pdf)
  • The challenge of information visualization evaluation. Plaisant. (pdf)

More Information on Statistical Tests (Optional)

Optional Readings

  • Chapters 4-5: In Task-Centered User Interface Design. Lewis & Rieman. (html)
  • An evaluation of space-filling information visualizations for depicting hierarchical structures. Stasko, et al.(pdf)
  • An initial examination of ease of use for 2D and 3D information visualizations of web content. Risden, et al.(pdf)

Contents

[edit]

Bryan - Mar 02, 2006 10:04:16 am

I often find myself asking after reading about visualization design or UI design, "what do users actually think about this?" or "which of these approaches is actually easier for users?" I think one dangerous pitfall of research in visualization or human-computer interaction is that it is very easy to invent grand schemes, code them, write papers about them, and show how beautiful they are, but it is very hard, if only in terms of paperwork, time, and hassle, to actually gather users and ask them what they think. I wonder if the field in general would benefit if every new technique required as a matter of course a comprehensive user study before publication. Then again, it might just slow things down.

One the one hand, I can imagine many papers written on fisheye distortion before anyone bothered to ask whether it was useful in a formal way. Was the time spent on those papers wasted? On the other, as a researcher I would probably find myself avoiding important problems because I'm too lazy to organize and execute a user study. Tough call.

[edit]

Noaa - Mar 02, 2006 08:59:22 pm

The whole user centered design thing really is important and it drives me nuts when I know people who develop software and just stick things in because it's a cool new technology or because they like it, but it has nothing to do with what the user wants or who the user even really is, and would in the end just end up confusing a user who isn't a techy like they are. And generally me saying this falls on deaf ears or I'm told that I'm not forward-thinking enough, but I think they're just inconsiderate designers!

[edit]

Mattkam - Mar 02, 2006 11:46:14 pm

To add, what would be an appropriate evaluation for a visualization or UI design? I mean, it would be an equally serious cause for concern if researchers go through the motion of conducting evaluations without questioning if the evaluation serves to answer the relevant questions.

In today's class, we covered quite a good variety of evaluation techniques, and I'm glad that Maneesh pointed out the value of qualitative and subjective research, as well as that of long-term ethnographic research. The evaluations presented in class today seem to be mostly controlled lab experiments, and I wonder if there were attempts to evaluate visualizations in more naturalistic settings? If not, is it because it's too difficult? Or viewed as less "scientific"?

[edit]

Cynbot - Mar 03, 2006 12:26:07 pm

In reading " Task-Centered User Interface Design A Practical Introduction" two very important points were made: 1. If management does not want to wait to ship a product for usability testing, shift focus from " TIME TO MARKET [to] TIME TO BREAK EVEN". It is true that if a UI for a product is unbearable, no one will want to use it and you will do more danamge than if you had just beaten someone else to the market. I am sure we have all seen products that were rushed to market and thought less of the company after. 2. While software algorithms and features can be updated post shipping, it is very rare and sometimes dangerous to change the UI. This places even more emphasis on the importance of user studies and sweating the UI design before release.

[edit]

Nchentan - Mar 04, 2006 09:29:40 pm

One comment I have about the Assignment 4 experiment is that sometime, the reverse in the alphabetical order of the left and the right choice can contribute to the error. For example, if the two choices are D, A. I seem to associate A to the left and D to the right, therefore, sometime I made error by pressing / even when I wish to answer D. Does the experiment intend to take this type of error into account? It does not seem that trial experiments help eliminating this kind of mistake.

[edit]

AaronHoover - Mar 05, 2006 11:46:14 am

Has anyone had any difficulty running the experiment on a Mac? I've tried running the experiment on my Mac several times now, but I keep getting a corrupted jar file error. I cleared the Web Start cache and tried again, but to no avail.

[edit]

Cynbot - Mar 05, 2006 12:54:23 pm

Nchentan - I had the exact same problem :P It's probably a good hypothesis to test. Some notes on my user experience: 1. I found that I ordered the letters a lot in my head spatially so if B was larger say than A, I would hit the right button even if the image "B" was on the left. 2. also I had a learned strategy, I would find the largest in the group first then look at the labels and combination of that would be larger, this did not work for examples where the largest was not in equation. at that point I would go to the next largest etc. 3. Also there was just alot of error, because of the time constraint I would get flustered and hit the wrong button.

So for me, this experiment is testing cognitive spatial awareness and which display conveys "greater than" relationships the fastest. It would be interesting to place the comparison tags closer so that the user doesn't have to scan as far to read what they are looking for.

AaronHoover - I have had no problems on my Mac running Tiger 10.4.5.

[edit]

Sharena - Mar 06, 2006 10:07:37 pm

I have learned and heard about usability and evaluation before but one of the parts of the lecture that stood out to me was the idea of doing a pilot version of the experiment first to test the instructions and the experiment itself. I have not really heard this before but I can see how it is very useful. I recently participated in a focus group and then after I was asked to complete a survey, it turned out that I was the first person to do the survey and I was able to talk to the researcher about the confusions I had with the survey and she was able to change it for the next time. So here it wasn't a formally conducted pilot but it turned into one, so I can see how it is very important to conduct these types of tests formally, each time a study is done.

[edit]

IvanTam - Mar 06, 2006 10:50:44 pm

One question I have about usability evaluation and visualization is how usability can be confirmed. For example, with interface testing, researchers can follow a potential users interaction with the interface and later on redesign the interation to better suit users. With visualizations, how is this done? Is this the part where we pull out the eyetrackers? While statistical analysis can tell is what "works" better, I feel that it doesn't really tell us where potential problems lie. It seems that we can only change the design and reiterate (trial & error) until we get the right numbers in the statistics.

[edit]

Brien - Mar 07, 2006 03:50:31 am

Bryan -- I wish it was very easy to pull together grand schemes ;)


One interaction technique never seems to strictly dominate another. In Kosba’s paper, the info vis software (InfoZoom) that displays all information up front works great until you need a different view, then wading through modes in Spotfire actually pays off. So I agree with Mattkam -- I'm sure every visualization technique could be shown to be dominant in an appropriately constructed experiment. How fragile that experiment is – how far its results can be extended -- is something worth studying. I guess that's part of the art of test design.


I thought the command logging mentioned in the lecture was interesting. Not only does it single out what interaction techniques need to be most intuitive, it also shows what algorithms/systems need to be sleek. It would be interesting to extend the logging to features not implemented yet. You could put some dummy menu items/buttons in the UI, give it to a test group, and then see in numbers which non-existent features are most requested. It may not line up with the obvious and would help prioritize development.

[edit]

Wait321 - Mar 07, 2006 10:16:41 am

Nchentan - I had that problem too and I know I pressed the wrong button a few times.

According to my manager, one of the reasons that user studies aren't done on products is because it's so hard to get a good study. The group isn't always a random sample of your customer base. In addition, there's the problem discussed in class: that users give sometimes give invalid input. As Kosba mentions, any experiment can also be easily biased, perhaps accidentally, due to the way it's set up or analyzed.

[edit]

Raymond - Mar 09, 2006 12:15:06 pm

Wait321- In my UI design class, I have actually heard that user testing is far more accurate that doing heuristic evaluations on prototypes. While they may costly and take a lot of time, it is still good to incorporate user input into the final design. What's actually the best way to go to alternate both user testing and HE.

In regards with the assignment, I too had similar problems where I knew the right answer but mistakenly pressed the wrong button (either due to reverse order and/or time limitations).


[add comment]
Retrieved from "http://vis.berkeley.edu/courses/cs294-10-sp06/wiki/index.php/Usability_and_Evaluation"