Qualitative Evaluation

From CS160 User Interfaces Sp09

Jump to: navigation, search

Lecture on Mar 9, 2009

Slides

Contents

Readings

Carolchen - Mar 07, 2009 02:29:37 pm

I found this week's reading to be useful, especially at this early stage of our project, when we need to begin evaluating our interface and revising it. A couple of interesting points in the reading were:

1. It takes time to decide between two ways of doing something. This indicates to me that a user interface designer needs to consider whether providing the user with more than one way of doing something is really valuable, given the time tradeoff. One common complaint about Photoshop is that there are too many ways to do the same tasks. The cognitive load required is inconvenient for the user.

2. Task-oriented evaluation methods such as cognitive walkthrough and action analysis are focused on a subset of all tasks. Because of this, it is inevitable that some action sequences are missed, and certain controls are not evaluated. This was one of my concerns about our lo-fi prototype. While on the one hand it is extremely time consuming to cover all predictable interactions and tasks, not being able to do so leaves giant sections of the interface unevaluated, unless you use heuristic evaluation or some other non-task oriented method of evaluation. As the reading also points out, task-oriented evaluation methods are also too modular, so that cross-task interactions are neglected.

3. I appreciated the example on how to do a heuristic evaluation. It's one thing to read a list of heuristics without ever having done such an evaluation. Seeing the types of comments made in an example evaluation and how they are organized with respect to the list is of immense value.

4. It was interesting to learn that the types of usability problems found through heuristic testing can differ depending on the fidelity of the interface. Knowing that it's harder to find missing elements in paper mock ups equips us to look harder for such holes.

5. It sounds like evaluation methods supplement each other and have different purposes. The main idea I got out of this was: do a heuristic evaluation to clean up glaring errors in the interface, then redesign, before conducting user testing. This cost-effective method of making the best use of users' time seems to have its merits.

Saung Li - Mar 07, 2009 05:55:19 pm

In testing designs in general, it seems that several different testing techniques that use several different subjects is best for uncovering both major and minor problems. Cognitive walkthrough is something the designer can use to imagine how the user will do something and why. If there isn't a good reason why the user will do something, then that action should be reconsidered in the implementation of the design. Other designers can also use this method to uncover other problems the one designer may have missed. A formal action analysis gets down to the gritty details of how many actions are performed and how long each of them take to carry out a task. This takes a ton of work just to allow the designer to shave off a little time, but saving a little time for actions that are done a lot will save a lot of time and money overall. The back of the envelope action analysis works similarly but does not go through the detailed calculations and can allow the tester to focus on the task at hand instead of the details. Heuristic evaluation is not task-oriented and not used on users. Several experienced evaluators should be used to examine the heuristics guidelines and see if the interface meets those criteria. Avoiding testing actual users can save time and money since there are usually some errors that can be found and fixed before presenting the design to the user. This saves the user time and gives a good impression for the user of the designer as a professional. After these types of tests, the results should be used to fix the problems. Major problems should be fixed first, while lesser problems may or may not be fixed depending on cost-benefit analysis. Testing on users should be done after a heuristic evaluation to catch more problems. These testing methods can and should be used throughout the design cycle, finding and fixing problems in each iteration. It seems like a good idea to use these methods in lo-fi prototyping to quickly uncover problems and fixing them. All of these methods can work with a paper prototype, and also later in the design phase. These methods have their own weaknesses; for example, the heuristic evaluation does not catch elements that are missing in the interface easily, so thinking about what elements are missing should be emphasized. Using a variety of methods, evaluators, and users produces the best results since different problems are uncovered and the interface can then be geared towards the general user population.

Jeffrey Patzer - Mar 08, 2009 11:53:12 am

I thought the most useful part of these pasts readings was the list of heuristics and also at the end of the Task-Centered User Interface Design article, their use of them. It was useful to be given a list of requirements and to be told how to evaluate them. By simply asking oneself questions geared towards fulfilling those requirements, one can learn a great deal about how their interface is working and functioning. Below are a list of the heuristics that I thought were the most useful:

  • Visible system status: this is such a simple idea, but an extremely powerful one. If we were to draw a metaphor with real life then we would say that when we look at things in the real world we are always assessing the state they are in. If we are using a program then we need to be able to do exactly the same thing by just glancing at it.
  • Aesthetic and Minimalist Design: I think that the use of an uncluttered workspace is extremely important when using a computer. If your desktop becomes too populated with folders and files then it becomes extremely hard to find anything. It actually becomes easier to look at it as a list. Finding the right way to layout information and in what view is should be laid out is key to getting the most from a system.
  • Help and Documentation: I am mostly putting this on here for mac users. Not that windows doesn't have a help window/button or whatever, but a mac's help feature is far more advanced. For those who don't know the system, the help button on the mac menu bar, when clicked, brings up a search bar right underneath it. As you type it begins searching in real-time, listing various guesses underneath your field (kind of like a search bar in your browser). The most useful feature of this system though is that when selected, if it is not instructions to do something it will pull up an arrow pointing to a menu item (ie copy, paste, etc...). It takes you directly to what you need. Very fast. Very helpful.

These heuristics and the others I think are great conversation starters for determining the effectiveness of your interface.

Denise Ngai - Mar 08, 2009 05:28:11 pm

I found these readings highly appropriate for our upcoming assignments. For example, with our Lo-Fi Prototype assignment coming up, the Task-Centered UI Walkthrough reading in conjunction with the heuristic reading provides helpful tips and how-tos regarding performing a walkthrough, such as the one we'll be doing via the lo-fi prototype. I especially found the "Ten Usability Heuristics" section of the heuristic reading helpful. Each point makes sense with regard to UI design. Also, what I like is how the usability heuristics stress the necessity for human language rather than computer language, so users from all backgrounds will be able to maneuver the UI. Clear, precise, and simple directions and error messages are definitely the way to go. I've witnessed many a person complain about error messages they receive on their Windows PC because they simply cannot understand the language used in the error messages. Only the extremely computer savvy individual would understand what's going on. Sometimes this even leads to the user having to research the error message online in order to determine what's going wrong with the computer. Simple language with thorough explanation that anyone can understand would help get rid of this necessity to look up error messages online.

It is helpful to read articles such as these because there are certain minor details that one would overlook when performing prototypes and walkthroughs that these readings be sure to mention. For example, in a walkthrough, I would have initially told the user what I'd want them to do and have them try to do it themselves rather than actually giving them the step-by-step instructions on how to perform the actions necessary. I originally thought this was the correct way to approach a walkthrough in order to determine how easy/difficult it is to maneuver the UI. However, I now know that a walkthrough is indeed literally a walkthrough. Provide the user with step-by-step instructions and observe whether he/she is capable of doing so.

Prahalika Reddy - Mar 08, 2009 06:34:20 pm

The three methods of evaluation described in Chapter 4 of Lewis and Rieman all seemed useful. I'd have to day the one that seems most effective to me is the cognitive walkthrough. Finding a reason for each action seems to be the best way to decide whther or not it's a good interface. It's the way to get rid of some of the arbitrary and unnecessary actions that are talked about in previous readings.

The heuristic evaluation method also seems like a useful technique, though it seems a bit more broad. From my understanding, the heuristic evaluation doesn't seem specific enough to find the smaller problems in the interface. If you just follows the principles stated in the heuristics, which aren't very detailed, I think you could miss a lot of the less obvious problems.

The action analysis method doesn't seem like a way to actually find problems as much as it is a timing method. Like in the previous readings, the action analysis measures the time it takes for certain actions to be completed, though in a much less math intensive way. This is useful in shortening the time it takes to perform some actions, but not exactly helpful for changing the interface for ease of use.

The readings this time were much more enjoyable than the previous ones, as they were easy to understand and about topics that actually seemed useful.

Kevin Huey - Mar 08, 2009 11:23:06 pm

I find it interesting how Lewis & Rieman use two of our most recent readings for two of their three design evaluation methods. Really all they've done is add Cognitive Walkthrough into the argument (which they credit themselves for, by the way). Nevertheless, I find the Cognitive Walkthrough portion to be the most useful method of evaluation. We can discover common bugs and design failures early without wasting the time of user testers to find what we can find ourselves. It allows us to create a better prototype, getting testers to focus on other elements that we can't find as easily, thus making beta tests more informative to us. The second method, Action Analysis, is based on GOMS. I still have little belief in it, other than the notion that small numbers are fast and large numbers are slow. Comparing two large or two small numbers is still my problem with it. Heuristic Evaluation makes sense, though many programs today still don't give the portrayal of using this method. As Nielsen said in his examples: in Excel, you make a table with 'Make Table' command. When you want to edit the insides of it, first-time users would look for 'Edit Table', but that only resizes the table. Instead, they must click on 'Format Cells', which don't have the word TABLE in it. A classic example of functions not being where we think they should be.

Sean Hansen - Mar 09, 2009 12:02:30 am

Did anyone else find it appropriate that the heuristic method's usefulness was tested by subjective means?

Anyway, I think that the latter reading should have been provided a bit earlier in the course, when we had to do all the individual design assignments; more effective means of personal evaluation of interfaces would have been quite useful.

I wonder how much the actual list of 9 or 10 (depending on the reading) heuristics has to do with the heuristic evaluation process. Given that the evaluators have some background in interface evaluation, you'd imagine that general problem areas like those would already be in their heads. If, however, the heuristics do help, then would a better set of specifiers yield better evaluations results? More generally, would a different set of heuristics yield consistently different results? If so, this suggests that there is a complementary set of heuristics that, when used in parallel with the suggested set, would yield evaluations that get much closer to finding 100% of the problems in an interface, and wouldn't that be just dandy?

Mark Dhillon - Mar 08, 2009 11:49:00 pm

Here's my ranking of the ten usability heuristics

10) Help and Documentation Yes you need this, but if your system is bomb, then they're not that important anyways 9) Flexibility and efficiency of use The idea of accelerators is cool, but not even close to being the most important. I think getting a solid functionality is more important. 8) Help users recognize, diagnose, and recover from errors This is important, but it's more important to prevent errors from happening in the first place I think. A user will usually figure out if something is wrong pretty quickly. 7) Aesthetic and minimalist design Something that I've noticed from the upcoming assignment is that it is important to not clutter the screen real estate and confuse the user. 6) Recognition rather than recall This is something that I strongly agree with. I like the idea of abstracting tasks and descriptions through pictures, and it seems to go well with this heuristic 5) User control and freedom This is really important, basically giving you an eraser... 4) Match between system and the real world Of course, you have to make sure users can understand what to do. I had a great time coming up with all sorts of button labels this past project, and it did require some serious thinking. 3) Consistency and standards Very crucial. There needs to be a guiding theme and principle in design choices. 2) Error Prevention Pretty much the holy grail of computer science, make sure you avoid bugs! 1) Visibility of system status Freezing and hanging up are unacceptable when it comes to applications these days. 'Nuff said.

Shoeb Omar - Mar 08, 2009 10:34:27 pm

Stupid chrome should ask to close tabs before closing a window. I typed a response and accidentally closed my browser trying to minimize it. ARGH. Talk about UI issues. Anyway, I preferred the second reading to the first because it was a brief interesting overview of three different user-less analysis techniques that didn't go into too much detail and sort of covered the first while it was at it. Here's my brief overview of the three techniques described:

  • The cognitive walkthrough was the idea of going through a list of actions to perform a task from a user's perspective. The example they highlighted here was the idea of the copier machine and whether a user knows to turn it on and the oddness of the push button as a switch. It was interesting how they said that the list of actions should be already existing but then when evaluating certain things they didn't really assume this. I think the problem is maintaining a balance between a list of actions and a list of detailed actions. They should be detailed enough you're not lost, but not too detailed that the whole process becomes useless.
  • The action analysis stuff reminded me of the last reading from Raskin. Formal analysis is screaming Raskin stuff and is boring and annoying. The back of the envelope stuff makes sense and I guess could be useful to make the list of actions to then perform the cognitive walkthrough with
  • The heuristic analysis with the 9 heuristics (depending on the reading) made the most sense as they were just basically design principles. I think that a formal analysis using the nine heuristics isn't really necessary but keeping them in mind both when designing and evaluating your design is a great idea to keep in check.

Chao Michael Zhang - Mar 08, 2009 11:51:44 pm

Heuristic evaluation seems like something our group will definitely need, since it will allow us to find problems with our interface before we write even one line for code to implement our design. Something I found interesting in the readings was how there is a quantifiable limit to how many evaluators you should recruit to use in heuristic evaluation. The fact that there is a formula that allows us to graph the tradeoffs between number of evaluators and cost is pretty mindboggling. I wonder how accurate the graph would be to our testing, since we have no monetary costs increases when increasing the number of evaluators.

Nielsen and Molich's Nine Heuristics also stood out for me, and I think our group will definitely be incorporating those guidelines while developing our low fidelity user interface.

Another thing I found interesting was in the Characteristics of Usability Problems Found by Heuristic Evaluation, and was the point that lo-fi prototype testing can often be better because evaluators don't get stuck on user interface problems and fail to move on to problems that are further down the road, as they are shown to often do when testing real systems. Instead, lo-fi testers simply move on to the next page on paper and test the interface there. Even more reason for our group to use lo-fi testing as our first main testing method.

William Cho - Mar 08, 2009 08:23:56 pm

I appreciated the examples and analysis of the real-world user interface designs using the three listed analysis methods. Many of the points are useful to keep in mind, especially for performing a user study in this upcoming group assignment, although I sometimes feel like I'd be overwhelmed trying to keep track of everything I'm supposed to keep track of during the session. "Formal" action analysis again seems like too much work for little gain. I would hate to wade through the sea of details; I'd rather stand back and look at the bigger picture, so the "back of the envelope" action analysis school of thought appeals to me. If I were to actually try one of these, I'd probably follow Nielsen's recommendation of alternating between heuristic analysis and user testing in order to get the benefits of both methods.

Stephanie Shih - Mar 09, 2009 01:00:04 am

In light of our recent assignments, I definitely found these readings to be relevant to my interests. Like others mentioned, the list of Ten Usability Heuristics seemed the most useful, but I also found the page on severity interesting in figuring out how to prioritize issues.

While finding the problems at hand in the first place is important, it is just as important to figure out what needs to be fixed most promptly, and the three factors they use to determine that - frequency of issue, impact, and persistence - are all part of evaluating the problems and thus dealing with them as well.

Chunwei Lai - Mar 09, 2009 01:22:13 am

Lewis and Rieman focused on using the cognitive walkthrough as a tool for finding improvements in an interface has many uses in a larger environment because people can easily confirm another's thoughts. I liked how the back-of-the-envelop has appeared again as it often does on CS topic but it seems to downplay all the analysis provided in previous readings through generalization. The idea provided through "Nielsen and Molich's Nine Heuristics" and "Ten Usability Heuristics", I think are a great point that can be easily overlooked. The use of "frequency", "impact", and "persistence" to evaluate a problem seems like a very practical approach in assigning the order for issues to be addressed.

Yin-Zen "Johnny" Hwang - Mar 08, 2009 10:44:59 pm

these are good procedure-based readings for conducting evaluations. however, i think instead of being in a reading form, it'd be so much more convenient and useful in a checklist form so that we can easily go through the list and check off the things we've done in evaluation, such as getting around 4 people for evaluation, doing rankings of severity, labeling major vs minor error, etc. but ya, it's kinda sad that UI people get no respect =(( even though everyone interacts through a UI. before there was the program, there was the UI. finally, it's good to see the methods most commonly used are considered the most useful. that means we're gonna be doing something right.

Dwij Garg - Mar 08, 2009 11:57:32 pm

I really liked the second reading by Lewis and Rieman. I think it really outlines the process that a designer should use to validate his/her design for the real world with real users and experiences. One step that can be taken, as stated by the authors, is to go through a cognitive walkthrough of the design, which is basically used the theorize and validate a user's thoughts about your design after initial implementation. This really helps outline what changes might need to be made to the final implementation, and also what a user might think and/or need while using the design. Another interesting step stated in the article is Action Analysis. Both the formal and "back of the envelope" approach are very beneficial for a designer. The formal approach explains and shows the designer finer details of the design, more specifically the amount of time it takes to do an action and what steps the user has to go through. The second approach is a larger picture analysis that tells the designer some of the bigger problems of the design. This is important because, as the authors state, many designers get lost in the finer details of the design and fail to look over the larger picture at hand. This reading is really appropriate to our project, as groups can definitely utilize both these techniques to make sure their interface is upto par with different users. The cognitive walkthrough should be done with rough sketches of the design, and the action analysis should be done before the final stages of the design. This will give us a better perspective of what to fix in the designs and what to improve as well.

Cuong Ngo - Mar 09, 2009 02:24:04 am

The readings turned out to be interesting, especially Nielsen's ten usability heuristics. We can avoid unnecessary details in our design by following such guidelines. For example, our error messages may be lengthy and somewhat cryptic. The user then wouldn't be able to "recognize, diagnose and recover from errors." Nielsen suggested that "error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution." Likewise, "users should not have to wonder whether different words, situations, or actions mean the same thing." This implies that we have a convention and we follow it consistently through out our product. Consistent design patterns also help users learn to use the system faster and perform the tasks more efficiently.

Kevin Nakahara - Mar 09, 2009 12:25:39 am

I found the Nielsen readings to be refreshingly precise and clear in describing a process that seems rather nebulous. I liken the study of user friendliness evaluation to economics because although both have fairly concrete principles and facts, both are still subject to the whim and recklessness of the human mind, especially with qualitative analysis. The process outlined in the first heuristic evaluation article is interesting in how it actually maps out a logarithmic function with real constants in rating the effectiveness of evaluators in finding UI problems. It also outlines an interesting cost benefit analysis at the end which I appreciated as a political economy major. The ten things to look for was also a nice thing to read since it reminds us of obvious and not so obvious things we should look for as we conduct qualitative evaluations.

Alan Young - Mar 08, 2009 11:41:52 pm

I think Nielsen's and Rieman's articles were very practical and I feel that they are important because they seem to be vital for the success of a UI design and an important part of the iterative design processes we learned in the beginning of the class. Nielsen's first article on Heuristic Evaluation was a good explanation of the process of having independent evaluators inspect the interface and compare it to a list of usability principles to find possible problems or bugs. I liked the effective use of graphs to show how many evaluators would correspond to number of issues found and even the practical aspect of how the ratio of benefits to costs would be for increasing evaluators. Rieman then goes on to talk about the 10 heuristics in depth which I felt made a lot of sense and probably serve as a good foundation for most interface designs. I did feel that his mention of "accelerators" under Flexibility was a little vague and in some scenarios, may conflict with the heuristic of minimalism. I feel that making advanced accelerator features hidden may make it too cumbersome for experienced users to bother learning and taking advantage of it since it might violate the heuristic of "recognition rather than recall". Rieman's "Severity Rating" was pretty typical and expected and I agree that individual ratings are not meaningful and not to be trusted, but the mean of many users' ratings provide some good notion of how "severe" a problem may be. Nielsen's final article on "uptake in industry" was interesting and was pretty unsurprising and expected in my opinion. I think that since the majority of people who took the course felt the need to learn it would lead to stronger motivation to actually use it and the fact that the majority modified the principles makes sense because real world scenarios often times have more constraints to take into account. I especially like Rieman's article on "evaluating design without users" because I agree with his concept of making sure the design is respectable before investing in the time and money of user-testing. I agree that this would lead to better user-test results and more efficiency in the iterative process in general. In my opinion, I feel this article would be perhaps more appropriate a couple weeks ago along with the article on apprenticeship testing since I think an emphasis on the step of testing without users is important in ensuring that the user-test would be most productive.

Sean Kim - Mar 09, 2009 01:59:43 am

After reading this web-page, I think that Ten Usability Heuristics suggested by Jakob Nielsen can be applied to the recent assignment for individual coding assignment for a taskmanager. Visibility of system status, Match between system and the real world, Consistency and standards; considered with the three points above, it determine which graphic item I used for a specific task and where the component is located for better consistency and standards. User control and freedom, Error prevention; And the next two points are more related with an error. We used validators for error checking and I inserted an additional comment for each component prone to occur an error. Recognition rather than recall, Flexibility and efficiency of use; And I think that these are a kind of advanced interface. I think that I can suggested a short cut key related with the tab name or task that users try to do. Aesthetic and minimalist design, Help users recognize, diagnose, and recover from errors, Help and documentation; I think those are generally affected to every process to develop them

Colin Downs-Razouk - Mar 09, 2009 02:31:43 am

This was a lot of reading. It seems to me that a heuristic evaluation of an interface would not work for all kinds of interfaces, or at least not how I came to understand heuristic evaluation from the reading. What if the application to be evaluated was for a very specific task. How can you be sure that your evaluators can correctly identify "usability problems" if they don't themselves understand the details of how the interface should be used. Do the evaluators in this case have to be picked from among people who already know how to do this task? If so, how is it different from regular user testing. It seems like the technique would be useful for certain usability problems, but it seems like it would miss a lot of important problems that are related to the specific task to be performed (if the task is a very specific task.)

Nalditya Kusuma - Mar 09, 2009 02:44:03 am

The second reading really opens my mind about several different ways to evaluate user interface design without the presence of testers. Cognitive Walkthroughs explores the interface by imagining people's thoughts and actions when they use an interface for the first time; sounds simple but the reading explores the topic more by reminding designers to keep asking detailed questions about each thoughts users might have during the interface usage. Action Analysis explores the interface by computing the time it takes for users to reach goals through formal and back-of-the-envelop models. Heuristic Analysis explores the interface by analyzing handy heuristics. These methods are going to be useful in developing group projects.

Phiroath Chan - Mar 09, 2009 02:57:13 am

The reading was an important reminder that we just can't rely on our users to catch the errors for us. They have their own lives and thus we need to also rely on ourselves to catch errors. The reading was also important because it described three ways how we ourselves can go about evaluating our designs. I think the cognitive walkthrough approach is really useful because it accounts for what the user thoughts and actions will be. The irony is that the designers are not allowed to use real users to do the walkthrough and that is where the real value of the cognitive walkthrough approach is. Without being depended on real users the approach still accounts for the users' thoughts and actions. Of the two action analysis approaches, I feel that the Back of the Envelop approach is more useful for our particular design project. We are not designing anything large or complicated enough to need extreme detail of the system and the evaluation of the system. Formal analysis is great for other large projects, but not really useful in my opinion for our projects. The more "big picture" approach of the Back of the Envelop analysis seems the correct way to go. Heuristic analysis has its worth because it has a checklist type format for designers to follow and evaluate their design. Having a concrete structured system is sometimes a plus for developing successful designs.

David Burban - Mar 09, 2009 01:03:00 am

I think that the reading about the Task-Centered User Interface Design would have been more useful had it been given earlier before we started doing our paper prototypes. Even though the self-evaluated design would not catch all the problems, it is still a useful thing to do, as catching any number of problems is better than catching none at all. Before we started doing our first user study, some of our group members objected saying that our procedure for adding a friend was overly confusing, but yet we persevered. And in fact, the first user found that, like some of us though, adding a friend was the hardest task.

As for the other readings from useit.com, a checklist of what we should follow would've been useful. I liked the page about the "Ten Usability Heuristics", since it is a nice summary and is easy to follow. However, for the Heuristic Evaluation, I was pretty confused as to were the monetary figures actually originated.

Ian Hildreth - Mar 09, 2009 03:05:07 am

I overall enjoyed the first reading more than the second. Although the second reading had a great number of ideas about how to conduct user testing, I found the principles in the first article to be much more important, especially at this stage in our design process. The article did a great job of describing the heuristic evaluation process and reaffirmed how important it was for our group to do use the idea. The ten usability heuristics were also very important because they identified key points in designs that I had overlooked in the past, as well as current problems with our interface we are currently working on. For example, the user control and freedom is especially important when designing games. A user must be able to have the power to go back and edit settings that they incorrectly set, or simply want to change, be it during the initial setup or midgame. Also, it is very important for the user to have the ability to exit during the game and simply return to the home screen, something that seems so simple yet is often overlooked.

Lucky Ongko - Mar 09, 2009 03:18:39 am

The important take point of the readings is how to evaluate the user interface of a program without a throughout testing. This can be done in three different ways according to Lewis and Reiman: cognitive walkthroughs, action analysis and heuristic analysis. Cognitive walkthrough shows different ways of imagining people's thoughts and actions when they encounter an interface for the first time while action analysis evaluates detailed steps that needed to be taken by a user to perform a certain task with the interface. An easier way of action analysis is by using back-of-the-envelope action analysis, which can be done by generalizing the steps that need to be taken into "natural" steps. The heuristic analysis consists of ten guidelines, in which Visibility of System Status and Flexibility and Efficiency of Use would be the most important heuristics to create an easy-to-use UI.

Timofey Titov - Mar 09, 2009 03:29:14 am

I think we could've benefited more from the first reading (Heuristic Evaluation), if it were posted before. The second reading says that "Every user will have a slightly different set of problems, and the testing won't uncover problems that the few users tested don't have." That's why you have to run user testing in conjunction with heuristic evaluation. It seems that Nielsen presents it as the Holy Grail of testing. This is in contrast with Rieman who sees it just as another kind of testing: "That's partly because there are kinds of problems that none of these methods will catch". I'd like to know what kind of mysterious problems these are. I really like Rieman's approach of evaluating an interface, before testing it. It's kind of like "debugging by thinking" and even challenges iterative cycle. Instead of revving through iteration cycles, one could think, plan and evaluate in order to come up with a better overall work. This will reduce the amount of iterations needed.

Moonway Lin - Mar 09, 2009 03:30:12 am

I also think that the methods described in the articles are practical and useful approaches to evaluating UI designs. While doing the design assignments, I've unknowingly applied several of Nielsen's 9 heuristics while sketching the layouts. I made sure to use simple language in the instructions presented to the user (heuristic #2), minimize memory load (#3), check for consistency between similar functions/screens (#4), and alert users with messages after every important action (#5). Same goes for cognitive walkthroughs: I put myself in a user's position and thought about what I would do to perform the program's desired functionalities. I've never done action analysis because the quantitative results are too precise for our needs in previous design assignments.

Ling Chen - Mar 09, 2009 04:09:59 am

This article actually taught us a bunch of useful things. Not only did it talked about cognitive walkthroughs, it actually gave examples. I always find it nice to have examples to go alone with the concepts we learn because then we can "see" the concept in action, and make it more applicable rather than simply having words on paper. It even advised us to use a group of people who are roughly at the same level in the company hierarchy in fear that the evaluation might turn into a show at the presence of higher levels. I especially like how the article points out to us the common mistakes people make during the walkthrough method, so we can know what to watch out for.

The of course, there are other analysis methods with other goals and purposes. Whether it's Action analysis, Back-of-the-Envelope Action analysis, or Heuristic Analysis, they each uncover the different kinds of problems with our interfaces. Even though with all of these methods combined, we could find a lot of the problems in our interfaces, they still can't catch all the problems. In order to create a truly user-friendly interface, it is our job as designer, to use these techniques and eliminate as many problems as possible before the user gets to test the system. These techniques are not meant to replace user testing, but rather as a guide and help to be used with user testing.

Adit Dalvi - Mar 09, 2009 04:00:20 am

The heuristics in the reading sounded like things you would be paying attention to if you were designing an interface, and if you weren’t paying attention to them (consciously or unconsciously), your interface would probably be impossible to use. I thought the cognitive walkthrough was a lot more interesting to read because it helps to really analyze your interface and try and weed out the bad aspects of it. The action analysis was pretty much the same as the last reading and I think that in a lot of cases, this analysis might be useful, but not efficient to perform on an interface. You’d probably be a lot faster to just apply the heuristics instead of doing action analysis, although back-of-the-envelope could be pretty fast.

Jason Lo - Mar 09, 2009 03:27:26 am

The heuristic evaluation article was pretty interesting. Although I found it talked a lot about heuristic evaluation without actually explaining what heuristic evaluation actually is. The Lewis article on evaluating designs without users gave a much better summary and large picture of what heuristic evaluation actually is and how to actually perform it. The article by Jakob Nielsen explains parts of heuristic evaluation and benefits, cost, and aspects without actually talking about the main thing. A lot of parallels between heuristic evaluation and contextual inquiry can be seen in the way the information is obtained and how evaluations are created. I thought the idea that experts can evaluate a design better on a series is good idea; however, it Nielsen doesn't explain many of the procedures of actually performing a heuristic evaluation.

Victor Lum - Mar 09, 2009 04:49:25 am

The heuristic evaluation stuff was pretty interesting. Especially useful are the list of ten recommended heuristics; these are all things I would want my programs to have, but wouldn't really think about too much unless it's laid out in front of me like it is. Given the low-fi prototyping we have to do for the next project assignment, using heuristic evaluation could help us get rid of some major errors before we test it on people. The second article's idea of cognitive walkthroughs is something I already do with other things, although not in as much detail. The action analysis seems a bit much, but I could see the usefulness of it. I'd much prefer using the back-of-the-envelope approach though.

David Jiang - Mar 09, 2009 05:11:25 am

I though the readings this week were very good and informative. I liked the second reading over the first one though. The first one talked about how to evaluate UIs. Basically had a couple of categories where users are able to 'grade' the UIs based off of these heuristics. Efficiency, ease of use, minimalist design, error prevention, there are just some of the several top UI design heuristics are based. It was pretty interesting, but some of the part just got kind of long and wordy. The second one wasn't too length, but the author got the point across. Cognitive walkthrough seems to work a great deal better than action analysis.

Chang Su - Mar 09, 2009 05:52:58 am

I find this week's reading rather fresh and helpful. Everyone knows that a design has to be evaluated even before being presented to the representative users for testing, but the articles defined the procedure systematically. If we were actually running a commercial firm, the findings of the usability studies presented by Nielsen would save us time and effort searching for the optimal methods, and consequently real money. The ten usability heuristics especially make an excellent checklist against which the initial design can be assessed. From the L & R writing the table of average times for computer interface actions is the most interesting. Although it appears almost ridiculously detailed, the collection of figures does provide a reasonable basis for estimating task time.

Meiying Li - Mar 09, 2009 06:43:54 am

I like the idea of having the debriefing session after the heuristic evaluation. It is important to involve the user in brainstorming anyway, plus, positive feedback helps designers in their next designs, which in some way are as important as negative feedback.

Though, I don't think the formula for the number of usability problems found in a heuristic evaluation makes sense.

I have used the severity rating before and I was confused about what I should rate with high scores and what not. It is also possible that even if we take the mean of a set of ratings from three evaluators, some evaluators might tend to rate everything too high and some too low. On the other hand, the designers/programmers might be more familiar with the market and the purpose of the program and can provide more useful ratings. What is the point of letting the evaluators do the rating then?

I like the cognitive walkthrough method mentioned in the second reading because it is more practical for small developing teams. The action analysis method is the same as the one we read last week. The heuristic analysis is just the same as the first reading.

Shendy Kurnia - Mar 09, 2009 06:59:42 am

I like the two readings. The title reminds me about heuristic function used in AI. After reading through, the meaning of heuristic function is almost the same with what I learn in AI. In AI, heuristic function is a function returning an estimation of the cost to get certain state. In UI context, it is also a value estimation of the design. We can get to know the "real" value after user interaction with the interface. Just like in AI, where the real value is discovered after the next state reached. The second reading also gives me an insight. It suggests UI designer to think thoroughly about the design before involving users. This somewhat contradicts previous lectures, but it can also be thought a complement of designing UI with user study.

Bernardo de Seabra - Mar 09, 2009 08:14:58 am

In the reading "Evaluating the Design Without Users" the authors describe 3 methods of evaluating interfaces without users. The first method presented is named "cognitive walkthrough" which is based on a prototype or elaborated design of the interface and on specific users. You then select a tasks to perform and develop a story involving the action. The story has to limit itself to what the users' know (general knowledge) and can see from the interface. The second approach is named "action analysis" which can be further divided into two sub-categories: formal analysis and "back of the envelop". The formal analysis is not quite easy to execute but can deliver great results, close to the actual interface performance. The main differences between these two is the fact that "back of the envelope" analysis leaves out details in favor of a quick overview of the big picture. The authors claim that this sloppier but faster technique can be very useful in deciding whether or not to add more features to an interface/system. Finally, the third and last approach is named "heuristic analysis/evaluation" are less formal rules (or rules of thumb) that can aid you in making sound design decisions. This topic is further developed by Nielsen and Molich's whom provide detailed analysis showing the pros and cons of this technique.

Eric Hernandez - Mar 09, 2009 10:21:06 am

In the Heuristic Evaluation reading, the chart showing the number of evaluators vs. the proportion of usability problems found is pretty useful to know. Having 5 evaluators is very good, but if not good enough, then 10 should suffice for most systems. The Design without Users reading made a good point with the Cognitive Walkthrough. In program analysis, it's always good to get high branch coverage (exercising all execution paths). The Cognitive Walkthrough seems to build on the branch coverage idea but more abstractly. If a set of tasks could be thought up to force the user through every screen and essentially every state of the interface then the likelihood of finding usability problems will be high if any exist. Going back to the GOMS analysis, Action Analysis was a valuable read as well, although not as in-depth as we have gone previously. The only real downside to this reading was that the Nielsen and Molich's Nine Heuristics fail in a way similar to other heuristics in that these general rules sometimes just aren't enough. Sure these rules can be applied to interfaces to yield better interfaces, but it doesn't evoke a magic or even sound method for creating better interfaces.

Sum Sum Wong - Mar 09, 2009 10:10:38 am

Walkthrough is an interesing idea that allows designers to evaulate the usability of an UI without having real users involved. It is true that users are not always free and willing to help designers to evaluate each new interface and there are some potential problems in the interface that might hardly be found by just a few users who is not really familiar with the interface. Thus walkthrough is a good method that can let desingers do a good evaluation on the interface by their own. The Ten Usability Heuristics in the first reading also caught my eyeball as it tells me what I should do(or not to do) when designing an interface. Of the ten, I am most impressed by the one - "Help users recognize, diagnose, and recover from errors", because it is something very important, but rarely did well among the user interfaces we are using daily.

Matthew Can - Mar 09, 2009 10:21:49 am

These are some useful methods for finding visibility errors. My favorites are heuristic evaluation and the cognitive walkthrough. Interestingly, I did a sort of cognitive walkthrough of my group's prototype after it had been created, but I realize now that my story about the user's actions was not at all believable. In my walkthrough, certain tasks were straightforward to complete. Testing with real users revealed that there was a lot of confusion about how to perform some of the tasks. My thoughts on what users would do was biased because I had designed the interface.

The method of heuristic evaluation seems like a good way to catch general UI problems. I would say that our group has been performing an informal heuristic evaluation of our game's interface for the last couple of weeks. But, I now know that we could have done it more efficiently if we did it individually instead of as a group. What I really enjoyed reading was the cost-benefit analysis of heuristic evaluation. A lot of the things we do in UI design are qualitative, so it's nice to see something that can be quantified. More importantly, I was surprised that heuristic evaluation could provide such a high ratio of benefit to cost as the reading indicated.

Derek Liu - Mar 09, 2009 10:30:35 am

I felt that these articles were useful in helping us to evaluate our own interfaces for our final projects, especially since my group is still in the stages of making design decisions. The ten usability heuristics were particularly useful for me, even though some elements of the list seemed obvious it was useful having the list spelled out to see just how everything is supposed to fit together. The second article had many ideas and techniques on user testing and good examples of walking through the system without actually testing with users. Both provide ways of catching interface design flaws and also mistakes in testing, on the user side and on the developer side.

Alexei Baboulevitch - Mar 09, 2009 10:25:03 am

Heuristic evaluation seems like an interesting method related to user testing that could help to uncover many usability problems. I was skeptical at first because quantifying usability problems into a discrete set seems like a potentially fallible methodology, but Jakob Nielsen appears to have done a lot of research to arrive at his results. In any case, it's clear that the users found heuristic evaluation and user testing to be very valuable (more so than any of the other methods).

The tasks themselves essentially summarize the information we learned in the previous articles: for example, "visibility of system status" relates to modes, "match between system and real world" relates to direct manipulation interfaces, and "consistency and standards" was touched upon in a few articles, especially in regard to common interface elements like widgets.

As for the other methods, action analysis relates to the "cognitive psychology" articles we read a week ago. Because of the level of specificity, it would probably only be really useful for organizations with lots of time and money. Heuristic evaluation and cognitive walkthroughs are certainly the most useful methods when developing an interface by yourself: they're things that we do anyway, and using optimized and well-researched versions of the methods will undoubtedly help to uncover significantly more usability problems.

Andrew Chen - Mar 09, 2009 10:37:22 am

As opposed to full-scale user studies or contextual analysis, I find that the heuristic evaluation method, together with the cognitive walkthrough and back-of-the-envelope action analysis of the second article, serve as much more realistic tools for a UI designer, and bear repeated use.

Cognitive walkthrough, to me, seems to be one of the most useful and fastest ways to clean up large usability problems, albeit only for novices (it is understandably more difficult to simulate a learned user in a person's mind). It seems the most natural to use, also, and I am glad that it is considered an actual method of analysis (since I consider myself pretty good at doing a cognitive walkthrough and used it on my designs even before reading the article).

Heuristic evaluation places most of the burden of making conclusions about interactions with the system on the evaluators instead of on the observer - who in a real user study would need to interpret all the pauses of a user's hand and eyes - and this seems to be a much easier way to analyze. When I was conducting contextual inquiry and task analysis with potential users in the project, I found that many users try to take control of the situation, generalizing and probing me for questions, and also moving too fast for me to really observe their actions (although this could come from my lack of expertise). Having the evaluators present the problems themselves would lighten the load on the observer. My question for this method, however, is who exactly would these evaluators be? They require knowledge of usability principles, however they do not need to be "domain experts." The only logical conclusions are that they are fellow UI people, or "professional evaluators" - whose existence I doubt. Could fellow UI people, however, serve as objective evaluators?

Siddharth Shah - Mar 09, 2009 10:37:43 am

I enjoyed these readings, and they seem like they'd come in handy for our final projects. I really liked the heuristic evaluation, the cognitive walkthrough, and the ten usability heuristics, and I would say those are the most important/applicable things I got out of these readings. I just really liked these readings, and hopefully they'll make me a better designer :)

Alexander Cho - Mar 09, 2009 10:36:00 am

All the concepts the readings suggests for improving interface design are agreeable and make sense. We should use multiple evaluators when doing heuristic evaluation because of the benefit of having different perspectives. I found it interesting how they actually had graphs and results from the benefits of more testers. Also this approach shows a good option for interface analysis when test users are not available. I found the list of 10 usability heuristics also quite agreeable, intuitive, but also something that I never thought about before. The ch.4 presented a cognitive walkthrough that breaks down all the steps of a user's thought process needed to execute an action it was useful to see it broken down this way.

Chris Thompson - Mar 09, 2009 10:50:32 am

Lewis and Riedman stress the point that although user interaction analyses are very useful for designing and debugging a user interface, the potential thousands of users will all have slightly different sets of problems, and the handful of beta testers one may get for a project won't find everything. Conversely, cognitive walkthroughs also won't find every issue with a user interface, but a combination of the two should significantly improve the end result. One way of evaluating a user interface without having access to users is to invent some imaginitive ones with a cognitive walkthrough. On larger tasks, it's often also a possibility to have other designers help out. It requires a description of the interface, including most main functions. It also needs a description of the task to be performed in this experiment, in order to direct it. The goal of the evaluation is to figure out what the user is thinking, and how they're responding to the system. Try to observe if they can find the proper components, and then understand how to use them, and see how confident or unsure they feel while using the interface. If you observe anything undesireable, make a note of it, and when you get back to development fix up the design. Action analysis, the next topic covered, is a simple idea: observe the sequence of actions people have to perform in order to complete a task. The rest of that article, and most of the next ones, focus on heuristics. Heuristics are a means of evaluation, in this case applied to interfaces. They go over some common and fairly obvious heuristics, and rate them by perceived importance. Then, the over all usability of a system can be estimated by summing up the values of the various heuristics.

Sean Ahrens - Mar 09, 2009 10:47:08 am

I enjoyed these readings very much. I especially liked the practical information given in the "Evaluating the Design without Users" chapter. I think it is a very valid point that many times in the industry (probably the majority) you are going to have to build stages of designs (even sometimes the whole design itself - for simpler systems) without the luxury of user testing. Being able to spot good design principles and thus make good design decisions without users can lead to much more efficient designs that are designed in less time. I enjoy heuristic evaluation because it cuts through the vast array of information regarding your design and gets to a few specific data points. You can then use these data points to make remarkably accurate assessments of your design. Both great readings.

Szu-Chun Mao - Mar 09, 2009 10:53:30 am

The reading for heuristic evaluation provides measurable methods and it’s concise and to the point, which seems easy to use as a practical application. I find the ten recommended heuristics especially useful and enable us to check the usable interface design for a quick evaluation. The Lewis and Rieman’s evaluating the design without users is a good supplement to last week’s reading. It indirectly explains how to apply GOMS analysis for the most frequently used actions and also how to do the back of the envelope calculation on a more complex user interface. Overall, I think this reading is very informative and it will be very helpful for our Low-Fidelity Prototype.

Salman Rahman - Mar 09, 2009 02:14:30 am

These readings were not only interesting but also provide practical value to us, especially for our lo-fi prototype design assignment coming up. The article on task-centered UI design describes performing a walk-through and that is something we will need to do for that assignment so our group will be paying close attention to the material presented and will make sure to implement that material, such as providing exact step-by-step instructions to the user as to how to perform a task. Another aspect brought up in the reading was the cognitive walk-through in which you find a reason for each action. That seems to be a highly useful way to weed out superfluous actions a user might have to perform in our UI.

The second reading provides a set of good heuristics to follow. For example, help and documentation ensures that the user has a contingency plan if he or she becomes confused as to where to go or what to do next. Also, the importance of an uncluttered workspace must also be considered because too much "stuff" can lead to confusion for the user.

Aaron Hong - Mar 09, 2009 12:22:29 am

This is an interesting reading about evaluating how well our design is doing. One thing I found interesting is the Action Analysis of the Task-Centered User Interface Design. They say that with proper analysis, you can predict the amount of time it takes for a user to perform a task within 20% margin of error. However, like they said in the reading there are many problems with it (the high cost being the main one) and we should not do it unless the pay off is huge (like time critical things that are repeated over and over again). For our purposes Back-of-the-Envelope action analysis is more useful.

Anjana Dasu - Mar 09, 2009 10:52:36 am

I thought the ten usability heuristics were really important and useful. They really linked to a lot of our previous readings.

  1. Visibility of system status- A user should always know what mode he/she is in. Although modes are not recommended, this is one of the ways to make navigating a "modal" system easier.
  2. Match between system and the real world- This is particularly relevant when we have a product that uses direct manipulation or design metaphors.
  3. User control and freedom- This advocates, like direct manipulation, for the user to be in control-- not the system. Also, as almost any one can testify to, it's very useful to have undo/redo.
  4. Consistency and standards- This is important on so many levels-- a program should have unique commands for each action, an application suite should be consistent across the board, and a company should maintain standards compliance across its products.
  5. Error prevention- It's obviously better to prevent an error than to have a user have to undo or restart.
  6. Recognition rather than recall- Ex. For a first time user who doesn't know cmd-X is "cut" the scissors picture is easily recognizable as cutting.
  7. Flexibility and efficiency of use- self explanatory.
  8. Aesthetic and minimalist design- The user shouldn't get bogged down by extraneous information or an overly complicated design.
  9. Help users recognize, diagnose, and recover from errors- A user should be able to understand his/her errors so they won't make them again. It's not enough for a user to just undo.
  10. Help and documentation- Ideally a user won't have to look through the help menu to do simple actions, but it's useful to have, especially for users to look up more complicated actions.

Joseph Tsay - Mar 10, 2009 11:17:09 pm

I found the readings on heuristics to be both interesting and useful in simplifying the way we analyze and think about different user interfaces. I found this reading to be related to the assignments we have completed so far, and the information would have been useful in both designing our interfaces and in conducted the user studies. I particularly agreed with the segments in the first portion of the reading discussing how the interfaces must be tested with a diverse group of many users, in order to discover the potential problems.


[add comment]
Retrieved from "http://vis.berkeley.edu/courses/cs160-sp09/wiki/index.php/Qualitative_Evaluation"
Personal tools