Qualitative Evaluation
From CS160 User Interfaces Fa06
Lecture on Oct 25, 2006
Readings
- Heuristic Evaluation. Nielsen. (Read the 5 linked pages under the heading: Jakob Nielsen's Online Writings on Heuristic Evaluation).
- Evaluating the Design Without Users. Task-Centered User Interface Design. Chap 4. Lewis and Rieman.
Chen Chang - Oct 24, 2006 12:08:20 am
Heuristic Evaluation: Nielsen - I think that this is a very helpful reading with regards to what we are going to do next with our projects after our interactive prototypes are completed. The fact that heuristic evaluation finds usability problems in designs so appropriate changes can be made as a part of the design process is essential to the overall design process. Rather than have only user testing, heuristic evaluation evaluators can spot those usability problems in advance by using techniques such as throwing out hints on how to use the interface and therefore save some time and resources of user testing. I can't agree more so with the reading that states that some usability problems are so easy to find that they are found by practically everybody, but then there are the problems that are found by very few evaluators. I think this is inherent in human nature in that each and every person has their own unique way of viewing and evaluating an interface, so each person has their own point of view and its worthwhile to get a spread out sample of evaluators to do heuristic evaluation. Like the article stated, 3-5 evaluators seem like the magic number to balance cost and productivity in finding problems and my guess is thats about the number we will be working with in this class.
Evaluating the Design Without Users: Ch 4 Lewis and Rieman - This reading provides two alternatives to heuristic evaluation which can be considered in the evaluation stage of the design process. Personally, I like the cognitive walkthrough technique as it is directly associated with the tasks that users are intended to perform versus action analysis which calculates the time it would take for the user to perform the given task. From my own experience, a mental walkthrough is very good because the possibilities are endless when trying to predict the steps the user will take through the interface; its essentially a brainstorm at first and then narrowing down until you only have good ideas remain which would likely be the "positive" steps the users will take in the end, but through this process designers can pick up on weak areas and add them to the list of fixes. Users and designers often think differently so something obvious to the designer may not be so obvious to the user and thus its a good idea for designers to be in the shoes of the user for a change. On the other hand, action analysis is a much more technical process with proper notation of the times it takes users to learn the interface, it requires extensive effort and resources to conduct and probably less feasible in our projects.
Jonathan Yen - Oct 24, 2006 01:52:31 am
Heuristic Evaluation: I think this reading provided some good information about the economic benefits of having evaluators as well as some useful heuristics to consider in designing an interface. A lot of the information presented seems redundant or otherwise trivial, but in practice, much of heuristic evaluation does not seem to be applied in improving user interfaces. I wonder if the issue is about whether evaluators notice usability problems or if it is about whether designers bother to deal with these problems given the costs and constraints.
Task-Centered User Interface Design: The part about common mistakes in doing a walkthrough was rather interesting. While the 3rd point of performing a walkthrough is quite intuitive, it's interesting how this step gets neglected many times. Also, I thought it was important to note that walkthroughs are not for testing real users, but for the designer to evaluate the interface. Like the last reading, the heuristics pointed out are either repeated or obvious, but no doubt are nonetheless useful if applied to the interface.
Robert Held - Oct 24, 2006 09:44:31 am
Nielsen:
The heuristic approach seems useful in that it sets out a well-defined list as basis for evaluation. Often when one assesses a product, the process seems disjointed and disorganized. The evaluator may be slightly confused about how to provide feedback. But heuristics provide a framework for descriptive comments and criticisms without being overly focused on a specific task.
Rieman:
Riemann's article did a good job of comparing the pros and cons of task-analysis-based and heuristics-based UI evaluations. He elaborated more on the work done by Nielsen et al that suggested using multiple evaluation approaches to reveal more areas for improvement within an interface. While heuristics are excellent for revealing potential issues within the program in general, task analysis is more focused and conveys specific issues involved in common processes users are expected to perform.
Ramy Ghabrial - Oct 24, 2006 07:41:39 pm
Nielsen: As stated before, the 3-5 evaluators figure is interesting. I am assuming groups will conduct heuristic tests on other groups' projects, which would fit this number very well. The suggestion for evaluators to go through the interface twice is also interesting; I wonder if we should have done this for lo-fi prototype testing? The ten usability heuristics along with their severity ratings seem very important and should be continuously considered in our interactive prototype. Finally, I was surprised by the higher usefulness rating given to user testing than heuristic evaluation, but upon further consideration it makes sense.
Lewis and Rieman: The cognitive walkthrough method sounds very similar to what we unknowingly used when first designing our interface and lo-fi prototype. The key issues here seem to be those of intuitiveness and feedback: interface elements should be designed so that users know how to find and use them when they need to, and so that users know they've correctly used them when needed. The action analysis is basically GOMS, if I'm reading this right, but the back-of-the-envelope variety seems to take normal users (not experts) into account, which is good. The heuristic analysis section goes into less detail than the other reading.
David Hoffman - Oct 24, 2006 09:02:32 pm
Nielson: This is a thorough description of the cost benefits of using external users to evaluate a product before its widespread release. He describes how asking the user to evaluate the product costs money and time, however they are more likely than the software creators to find usability problems. It is also important for the users to rank teh problems based on ther severity so that the development team knows what needs to be allocated to each problem to see if it is fixed. There is also an optimal number of users to use in one of these Heuristic evaluations. Using too many starts to just reproduce the same problems again and again, however, some individuals to uncover rare problems.
Lewis and Rieman: These authors discuss several methods for identifying usability problems in a piece of software. They choose as an example activating background printing on a mac. The methods they use rely mainly on following a thought trend through the full process of perfroming the goal. In this case, the area which I feel they omitted is somehow telling a user what background printing is. If I was not familiar with computers and all I wanted to do was print a document it is very unlikely that I would know to look for background printing, or would identify background printing if I had seen it. That aside, the techniques that they use seem to be restricted to basically very simlple small tasks. It would be difficult to scale this up to a large program. However, the general principle of thinking through actions and thinking about the time to do these actions is a good idea.
Maksim Lirov - Oct 24, 2006 10:27:58 pm
Nielsen: I was impressed by how complete the list of ten usability heuristics was. It appears that these heuristics cover the most important parts of any system and thus should be considered whenever possible while iterating through a design. Heuristic evaluation makes sense for many reasons but I believe the most important one in industry is the cost factor, like the example of the video conferencing system points out. At first it didn't make sense to me that "the lists of usability problems found by heuristic evaluation tend to be dominated by minor problems", but now it does because during the design period designers will concentrate on ironing out major problems and hold the minor ones to lower importance. I noticed that we already have been introduced to the severity ratings for usability problems, and from experience they were a good way to prioritize what should be changed in the design.
Lewis and Rieman: The walkthrough process that Lewis and Rieman describe highlights the important questions that need to be asked while doing the walkthrough. The four questions that they provide in section 4.1.3 seems to echo Don Norman's design principles - make controls visible, make mapping clear, provide feedback. In my opinion, the use of clear and intuitive labels is the most important for creating a "good" design, and these four questions help in identifying if the labeling will provide problems for the user. Lastly, it was interesting to read one of the author's heuristic evaluation of the Macintosh background printing controls using Nielsen and Molich's nine heuristics. One thing I noticed about the analysis is that while his comments made sense, I wonder if an evaluator from an entirely different domain (i.e. one that does not use computers very much) would have the same evaluation comments.
Robert Taylor - Oct 24, 2006 11:03:03 pm
Heuristic Evaluation: I'm actually surprised we didn't learn about this before prototyping because we seemed to do it, at least to an extent, in our lo-fi prototyping. Of course we didn't know all the heuristic principles at this time but I think some of the ones like aesthetics, error prevention, consistency, and visibility are fairly familiar. Other aspects of this reading, such as usability problems and severity ratings, are things we have already done exactly the same in lofi prototyping and I wonder why we read about them now. Basically, having gone over heuristics more thoroughly before prototyping probably would have let us create a better evaluation of our prototype.
Task-Centered UI design: I found this reading much more practical. Cognitive walkthroughs themselves are fairly intuitive, but I feel that this reading really helped hilight some of the questions one must ask while doing the walkthrough. The believable story first is a great and simple measurement of whether a feature is practical or should be included. I think the author somewhat glosses over the difficulty in identifying assumptions we may have, however. A formal system (I don't know if one exists) to give us a chance to identify our assumptions consistently would be nice. The only thing I can think of is going word for word through our cognitive walkthrough to try to identify if we've created an assumption. This is still not particularly effective, however. In the end I found this to be one of the most applicable and relevant readings we've had yet.
Yimin Yao - Oct 24, 2006 11:08:02 pm
Nielsen: I was kinda surprised by the number (4) of evaluators needed for optimal Heuristic evaluation; somehow I had imagined this number to be greater, since two people might approach the interface similarly and find similar usability problems. But given this number, it's now possible for us to conduct evaluation that will yield reasonable evaluation results while balancing our cost (time) and the completeness of evaluation.
I think the three criteria (frequency, impact, persistence) for severity rating are very useful. The first article mentioned about a debreifing session after the evaluation session. I think in addition, it would be helpful to combine the evaluation results from all the evaluators and survey their opinions on the severity of the usability problems that they did not discover, since something considered minor by one evaluator might be thought as major problem by the others.
Lewis and Rieman: I agree with one of the previous comments that such mental walkthrough proccess have been unconsciously practiced when we were working on the interface design or low-fi prototype. Because in designing features required by the tasks, or before giving users the description of tasks, we would go through the interface as an user but knowing what's the correct steps of actions, and try to look for places where confusions or errors might occur. The walkthrough process is indeed defined more strictly here, and I believe it's a very useful in doing a quick check for obvious problems or potential things to fix/improve. However we must keep in mind that what might be obvious or logical might not be so for users.
Bowen Li - Oct 24, 2006 11:59:04 pm
Heuristic Evaluation: I'm not sure I understand Figure 1. What makes an evaluator successful or unsuccessful? It doesn't seem to be correlated with the number or hardness of the problems they find. Am I the only one that read: “the observer .. has the responsibility of interpreting the user's actions” but also: “a possible observer .. does not need to interpret the evaluator's actions.” If familiarity with the system is not an important part of heuristic evaluation, then isn't it possible for the development team to perform heuristic eval themselves?
How does he define 'major' and 'minor' problem? He keeps mentioning that the users may have a hard time if they are not familiar with the domain. Why, one might wonder, would you want that person to test your system. They obviously don't fit into the target user group, and do not have the same mindset.
I am slightly disappointed by the 'rating' section. After all, in a method called heuristics, and after setting the user up with great information about the three factors that determine the severity of the problem, Nielson reduces the 0-4 to completely subjective ratings that depend solely on the assessor. I would have expected that the 0-4 ratings be more concretely defined by the 3 factors.
His take on the “usability of usability methods” is amusing, but very true at the same time. I think using heuristic evaluation is a great way to get some data about a program without having too much burden on the evaluator. It is also easier for the designer to find candidates to evaluate.
Evaluating without users: With respect to the cognitive walkthroughs, I know that I've definitely been doing something similar when designing our Anoto program. But the problem I run into most often is that different people have different opinions on what the users already “know” or don't know, what the user is capable of without explicit information given to them.
The walkthrough they described complements user testing well. One of them assumes the ideal and works down to the problems, while the other focuses on the problems and tries to get to the ideal.
Jason Shangkuan - Oct 24, 2006 11:39:15 pm
' Heuristic Evaluation:'
Heuristic evaluation is important in the design cycle because it is a way of verifying and confirming questions answered during the contextual inquiry. The contextual inquiry sets up a basis and parameters for a designer to work around and figure out what a user is looking for. Heuristic evaluation is a methodical process to analyze an interface to compare the expectations and interpretation. It is especially useful in the way that problems can be quantified by the severity. This is linked to concepts associate with brainstorming such as quantifying ideas. By approaching problem solving in a logical thought process, the larger problems of user interfaces can be identified and modified quickly.
Evaluating the Design without Users:
Cognitive walkthrough is an interesting concept because it sounds similar or can be related to the action cycle that we learned earlier on in the semester. A user or designer first starts at the high level of trying to understand the end result or goals. If the goal has not been met then they enter the execution path, which is quantified by the gulf of execution. The user then reaches the output of the interfaces and then the user enters the evaluation path, which is quanitified by the gulf of evaluation. If the interface is simple and clear then this distance is small and user will be able to identify whether they achieved their goal. If not, the cycle repeats until the user has achieved their ultimate goal. These are the mental thought processes that a designer steps through to interpret what a user might do, which helps with refining an interface.
Anirudh Vemprala - Oct 25, 2006 12:06:57 am
Neilsen:
A good description of the technique. I found the ten usability heuristics that he pointed out in the article to be a good summary of UI principles we've been discussing to date. His exploration of rating severity of usability problems looks like a very practical thing as companies often make trade-offs in what bugs to fix and which ones to leave as fixes in for future roll outs. Lastly, his analysis of the cost/benefit ratio of multiple evaluators was great research on his part.
Lewis & Reiman:
The cognitive walk-through process seemed a bit similar to the idea of persona creation we discussed earlier - just flipped around. Are they related in actuality? It sounded like the cognitive walk-through was designed more towards a general sort of user. One thing that did bug me about this process was that it was being run by the designers themselves. Though it is important for the designers to do this sort of evaluation, I'm not sure how benefitial it would be because of their "closeness" (emotionally and intellectually) to the design of their interfaces. As a side note, the examples in this reading were quite useful in understanding the concepts at hand.
Antonis Mannaris - Oct 25, 2006 12:10:06 am
Heuristic Evaluation: I was suprised that this reading came after the user testing readings. As the article suggests, it makes more sense to perform a heuristic evaluation before user testing to catch and fix obvious problems. Even designers themselves will be surprised when running through their whole design as a whole. Overall the articles are very clear and comprehensive, and heuristic evaluation is another method that will help any system during its design phase. Even though it seems extremely time consuming, if the cost efficiency figures are true, it is definitely worth it. One thing which troubled me was the strict heuristics and guidelines. If evaluators are aware of these they may be focused into finding problems that fit one of the categories. Perhaps it is best not to do any classifications during the evaluation period.
Lewis & Reiman:
In chapter 4 of the book, Lewis and Reiman describe three valuable tools about evaluating a design without users. This is particularly important since users are hard to find and costly to work with. I was particularly interested at how each of the three methods uncovers separate problems, so they each complement each other. It seems imperative for a good and thorough design to follow all three methods as soon as possible with your prototypes. These methods provide a concrete mechanism which a designer can follow so that the knowledge of his own design will not prevent the discovery of some serious problems.
Randy Hilarbo - Oct 25, 2006 12:35:48 am
Heuristic Evaluation At first, I was a little unsure how different this type of evaluation compared to the previous methods we learned. But I did understand how effective this method is. It's interesting how studies of this evaluation method reveals a curve in a graph that associates the number of usability problems correlated with the number of evaluators. I would assume that the number of usability problems found would continuously increase as you add more evaluators (at least more than 3 evaluators) but I guess this article proved me wrong. I'm also a little surprised of how much are spent in usability evaluation and how much the method described decreased this cost.
Their list of ten usability heuristics are pretty complete. This would be helpful to consider in further implementing our projects. Also, I think that rating the severity of problems is a very effective technique since it allows developers to easily prioritize what they need to implement.
Evaluating the Design Without Users I personally think that evaluating your own design is somewhat difficult since you yourself designed the system, and you already spent a lot of time thinking what it needs even before you implemented it. This kind of evaluation, I think, is perfect for checking if your design meet the specs but as far as if it really addressed the intended users tasks, more effort may be needed for this kind of evaluation to be very effective.
Out of the three methods presented in evaluating a design without users, I found that Heuristic evaluation would be the most effective. This method, I think, makes sure that you consider all aspects of your design. Unlike the cognitive walkthrough and the action analysis, where you are mainly focused on tasks, leaving opportunity for you to miss other parts of your system that needs to be evaluated.
Andrew Tran - Oct 25, 2006 12:49:58 am
Heuristic Evaluation: I believe these readings were excellent. Before today i only knew the definition of heuristics were to be rules of thumb when problem solving; however, as i found out after reading the articles heuristics were much more than that. I've always knew there were testings needed to be conducted to test problems in how users use the product, i didn't know there was a specific process in going about that testing, or techniques that would be quite helpful. I find that those ten heuristic principles are quite valuable in interface design. The severity ratings for usability problems is a good way in measuring how bad the usability problems are. Although it does seem like common sense that if the problem occurs quite often, is persistence, and has a major impact on the user, then it should indeed to fixed right away and the product should not be released. The first article mentions how the experimenter can help the user when he encounters a usabilty problem and has no way of figuring out what to do. I found myself doing this alittle during the lo-fi prototyping.
Evaluating the Design Without Users: Out of cognitive walkthroughs, action analyis, and heuristic evaluation mentioned, the most interesting thing i found from reading this article was the cognitive walkthrough. This is because heuristic evaluation i just read from the other 5 articles, and action analysis is somewhat analogous to KLM. So far was have talked about lo-fi hi-fi prototyping, contextual inquiry and task analysis, and wizard of oz technique. All these methods that help improve our interface requires interaction with a user. The cognitive walkthroughs doesn't require a user, although it can be used with a user. All you have to do is make up a story that a typical user would do with your system, and this will help draw out potential problems with the interface. I really do believe this method helps in finding problems that users might face when using the interface for the first time.
Tak Wong - Oct 25, 2006 01:04:35 am
Nielsen The 10 Heuristics mentioned basically summarizes what we learned so far in this course. In particular, many of them relate to the Human Processor (mapping and recognition). What I thought was interesting is that it mentioned about doing the analysis in small sessions for complicated UIs. I don't think this simple point was mentioned in the previous analysis. It also suprised me that the Heuristic evaluation would be very effective with only 3-5 people. I would think a lot of their ideas would overlap and won't really find enough significant problems. I guess having the evalutors doing their separate evaluations help to let them not conform to each other's ideas. I agree with the point that people won't use these evaluation methods until they are introduced in the middle of their project. It seems like people generally either don't care much about the design or want to focus on what they try to accomplish with the application rather than making it useable.
Lewis and Rieman I thought the cognitive walkthrough was pretty interesting since we usually don't try to convince ourselves that the application is useful. The "action analysis" was really the GOMS analysis we discussed in class in a less technical form. I thought this less technical analysis of GOMS is probably enough for our analysis instead of going into how fast we perceive things and where things are stored and such. Their heuristics evaluation is also more general than Nielsen (with 1 point missing). It seemed like the common theme in these 3 analysis is to do all the analysis we can without the user first. This is going be cost saving and will lead the designer to be more concious about their later designs and improvements.
Yang Wang - Oct 25, 2006 01:53:53 am
I have some questions for the heuristic article. I like the idea of heuristic evalution but I don't really the much point to it and how it differentiate from user testing. To me these two sounds very similar and almost have the same functions.
There are couple reasons that the author give to justify heuristic evaluation instead of user testing. But to me, it seems that these reasons don't really stand
1. The users are hard to get and we don't want to "waste users", but who says that the evalutors are easier to schedule. And why it is perfectly okay to waste the evaluators? Since a lot of times that evalutors are experts in the domain, I will think it is harder to get the evaluators to evalute the product.
2. The user test and heuristic evaluation find different problems. This is even more ludicrous to me. How does that support doing both user test and heuristic evalution, since that even different heuristic evalutions with different evalutors reveal different problem. Even if all evaluation give the same result, the difference can be result from the amount of instruction that moderator gives in different settings.
If both of these two points break down, then heuristic evalution is nothing more than an user testing with limited instructions.
Cheng-Lun Yang - Oct 25, 2006 02:54:29 am
Nielson
Heuristic evaluation is a critical component of user interface design. More is better cliché does apply to heuristic evaluation. The more users evaluating the design, the more problems can be found. It is interesting to note that there is no one person who is the best heuristic evaluator. One person may only find one problem but the problem is hidden so well that no one else found it. Another person may find the most problems in a design but those problems may all be considered easy and are found by many other evaluators. There is no set definition of a good evaluator so every single one of them are important.
Rieman
I found the cognitive walkthrough a little bit redundant. I think every designer is constantly walking through their design while making it in the first place. It is rare that a designer just act like a robot while designing an interface. Human beings all learn and modify while they are doing certain things. Also, as mentioned in previous readings, the designers themselves sometimes overlook the problems of their design since they just spend a lot of time doing it. That is the reason the designers need to have evaluators who are not part of the design process to evaluate the design.
Kang Chen - Oct 25, 2006 02:31:25 am
Nielsen
I personally found the graph depicting the number of usability found vs number of evaluators involved to be pretty interesting. In the beginning, it's intuitive that by adding more evaluators, each having an average probability of 42% in discovering major problems and 32% in discovering minor problems) would drastically improve the number of issues being identified. Then the number of problems identified levels off despite adding more evaluators. This, I think, agrees with the design and software testing/debugging cycles where the number ideas/number of bugs discovered decreases and eventually levels off as time increases. At some point, it's no longer worth the time and effort to find more bugs.
Lewis and Rieman
The three techniques, cognitive walkthrough, action analysis, and heuristic evaluation mentioned in this reading certainly helps in eliminating the more obvious usability issues beforing performing a wizard of oz interview with users. Cognitive walkthrough is fairly easy to perform as designers can simply write up a story and go through it step by step and note and flaws in the design. Since the designer should have some knowledge of the environment, the user, and the tasks being performed, it's relatively easy to put the designer in the user's shoes. As with the action analysis, the less formal or back of the envelope method seemed to be a lot more effective to me mainly because it's quick and dirty. It's easy for the designer to keep in mind which tasks have shortcomings and assign a priority to them. As mentioned in the Severity Rating and Usability Problems article, it's more important to assess how frequent the task is performed, the impact of the problem, and the persistance of the problem rather than just how long it takes to perform the task(GOMS & KLM). If the task is performed rarely, maybe it's not worth spending time trying to do a KLM on the task and optimize it. Finally, I really liked the heuristic evaluation checklist since it can be done quickly and applies to nearly every interface.
Simon Tan - Oct 25, 2006 02:00:15 am
Nielsen
Evaluating an interface seems like a wise idea, even considering the lo-fi prototypes we made a few weeks ago. Having a pre-evaluation would have made the official demo a lot easier and more polished - we wouldn't have had as many questionable points in our interface and would probably have been able to make options of interfaces for our real testers.
I agree that the list of ten recommended heuristics is an invaluable set of guidelines that user interface designers really need to memorize.
"Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution." -- SO important. I'm working on some error messages right now, and I realized that a lot of older ones I wrote are completely meaningless to the average user.
Lewis and Rieman
"Users with Mac experience will probably think that they need to click the "OK" button, but there isn't one."
I thought this was a bit odd, because today's Mac OS X prides itself on not needing the user to click "OK" buttons when changing settings. In fact, when I first tinkered with Mac OS X's control panel, I found it disconcerting that there was no "OK" or "Apply" like in Windows. I'm not sure which is better - having your options set as you set them or having a confirmation step.
Hrm, I thought at first that Action Analysis seemed a lot like the GOMS model... then I saw the link to the GOMS article. I must say, this summary of the technique seems a lot more straightforward than the things we read a few weeks ago. This article goes even further to simplify it by introducing the "back-of-the-envelope" method, which (frankly) is how I would do calculations of UI-time.
Utsav Shah - Oct 25, 2006 09:34:16 am
Nielsen
Heuristic Evaluation is an interesting concept and very useful one too. I think Neilsen does a good job mentioning some easy-to-follow guidelines. It seems fairly obvious but it's amazing to see the results provided by Neilsen in the reading. Infact, I've seen it done at my workplace over the summer when they were launching a new product. I think keeping those 10 usability heuristics in mind can solve most of the problems, if not all.
Lewis & Rieman
Here Lewis & Rieman provides couple other methods for evaluation in addition to Nielsen's Heuristic Evaluation. The concept of cognitive walkthrough is very fascinating because believe it or not most of the designers do it in their head without realizing it. What I mean is that everytime I'm writing a particular program, I always think about different scenarios and walkthrough from others' perspective. Several people mentioned that why is designer doing the walkthrough as opposed to the user. I think it's important for designers to think from users' perspective and cognitive walkthrough is the best way to go about doing it.
Edward Karuna - Oct 25, 2006 09:49:15 am
Clearly, as this reading shows, user testing is not the only way to go when working out the kinks in a new product. Nor should it be left by the wayside either, though, but rather used in concert with the methods outlined in today's readings. Looking at the charts presented, it seems that most methods are clustered within a given benefit rating, save for heuristic evaluation and user testing, thus, while all methods should probably be employed for every product, perhaps it is best to start with those.
The cognitive walkthrough concept presented in "Evaluating the Design without Users" is a compelling one. However, I would expect the creation of a proper cognitive walkthrough by a group of interface designers, developers, etc. to be a difficult task. This is not because of any unfamiliarity with the software, but, on the contrary, I would think that their "tech-savvy" would cause them difficulty in envisioning the actions of an unexperienced user.
Tabassum Khan - Oct 25, 2006 09:25:35 am
Nielson: The Ten Usability Heuristics seem to cover all the principles of good designing that we have learned so far in this class: make controls visible, provide clear mapping, immediate feedback, direct manipuation, direct engagement etc
The persistence factor which constitutes the severity of a usability problem cannot be measured by just one user-testing or by one heuristic evaluation. In order to categorize a problem as persistent or non-persistent more than one heuristic evaluation needs to be done by the same set of evaluators.
Lewis and Rieman: I believe some sort of cognitive walkthrough is done by the programmer while developing the interface because programmers are also users. The reading recommends to engage at least 3 to 5 experienced evaluators to catch all the "heuristically identifiable" major problems, isnt this approach really really expensive and influenced by their knowledge. It doesnt takes into account actual users and may therefore miss simple problems. I wonder if alternating between User testing and Heuristics evaluation can produce a more cost-effective result.
Ypai - Oct 25, 2006 10:13:19 am
Nielson: The article provides good information about how to do heuristic evaluation but Nielson also focuses on promoting heuristic evaluation as a very "usable" HCI technique. I think the point about making HCI processes more "usable" is very valid. HCI can provide benefits to any design but it must be adopted and integrated into existing development processes in order to realize those benefits. And while "user testing" is well accepted as a beneficial UI evaluation technique, the article points how user testing can be more effective if other HCI techniques are applied prior to user testing, which can be expensive and difficult to organize. In addition, Nielson does a good job of detailing how theory and practice differ, as well as trying to outline the monetary benefits of applying HCI for the bean counters.
Lewis and Rieman: This article does a good job of outlining in detail different techniques for evaluating a UI. Useful to me were the pros and cons of quantitative analysis and how to "guesstimate" quantitative analysis using action analysis. The part on heuristic analysis reiterated the Nielson article and also made it clear to me how heuristic analysis can be used to either quickly iterate and/or focus other forms of interface evaluation.
Sean Carr - Oct 25, 2006 10:52:42 am
Heuristic Evaluation: It seems like a lot of the usability problems found by heuristic evaluation are also the same problems we found during our lofi prototype interviews. What heuristic evaluation does is guide the evaluator about specific rules to follow where as in the prototype interviews the evaluator/user has to come up with everything on their own. I think this guidance is helpful, but you have to make sure you have good heuristics. Having a list of heuristics makes the evaluator less likely to report problems not covered by the heuristics. This is a sense can guide what type of evaluation is received, but this problem is probably insignificant compared to the benefits of the guidance provided by the heuristics.
Evaluating the Design Without Users:
One of the things I like about the cognitive walkthrough idea is that it can be applied at various scopes. Like the chapter suggests, a single developer can do it informally in their head while developing or the design team can do it in a more formal way, say after each milestone. I would hope that all designers use at least back of the envelope action analysis techniques when choosing between multiple options in their interface. I think this type of analysis should occur frequently during the actual design and development and not just at evaluation stages. Of course there is always the keystroke level model but that has to high a cost for being the method of frequent analysis during little decisions about small tradeoffs.
Johnathan Hawley - Oct 25, 2006 10:59:16 am
Heuristic Evaluation - I can see the advantages of heuristic evaluation of traditional evaluation. With the traditional forms of testing we've conducted so far, the user is dropped cold into this new interface and forced to struggle through with as little help as possible. This seems ineffective to me since almost all interfaces are difficult to use at the very beginning. The information you gather at this stage of the evaluation might not be relevant. Heuristic evaluation invites more dialogue between the user and observer. This helps the user overcome any initial hurdles that any user would have with looking at an interface for the first time. Heuristic evaluation also suggests running through the experiment multiple times with the user back to back. This again helps to locate interface problems that are not directly associated with the initial shock of your new interface.
Evaluating the Design Without Users - Conducting cognitive walkthroughs seems to be something we have already been doing this whole time: walk through the interface, and as why as much as possible. Nevertheless, I think a lot can be gained from taking cognitive walkthroughs seriously. They are cheaper, they can be conducted anytime without appointments, and can help solve a lot of interface problems before you try to reserve a test user's valuable time. With regards to action analysis, I can see KLM written all over that. This method forces the designer to break everything down into what incremental steps the user would take to accomplish a task, like a computer program as they mentioned. This seems like it would be a very effective way to evaluate a given interface. Forcing yourself to look at your interface at that deep of a level is bound to catch errors.
Melissa Jiang - Oct 25, 2006 10:19:05 am
Nielson Neilson stated that an observer cannot truly do user testing because they are required to observe user actions as well. This point seems to contradict what has been taught to us before. The interaction between the user and the object should be important because it hints towards what flaws the object may have even if the user does not point it out in the end. The Heuristic Evaluation itself seems like a good evaluation to do, could we not simply intergrate observation as well as Heuristic testing.
Reiman
Like many others have pointed out, it is nice to be able to do something before having to involve users. And like many others have pointed out again, this article suggests an idea similar to that of GOMS. This method would help us keep the user's limitations and envioronments in mind while producing the product and testing it. I do agree that the users' time is very precious and thus, this is a good evaluation to do before having users test out the product. My only question is, how come we weren't assigned this reading before we started making our prototypes...
Patrick Rodriguez - Oct 25, 2006 11:31:25 am
Nielsen: I think that the list of 10 Usability Heuristics are some good guidelines to follow for usable interface design. It puts down in words, explicitly, what we should be striving for as designers. Sometimes we strive to make a "cool" design, and forget about these design principles, and end up making something less than usable. It would be wise to always keep these 10 principles in mind. For example, aim for "Aesthetic and minimalist design." If ever you get the urge to ignore this rule, your heuristic evaluators will serve as a check and balance of sorts.
L&R: I like the Back-of-the-Envelope Action Analysis over the more formal GOMS techniques. For some tasks, a bigger picture overview is more useful than a key by key speed measurement. How confusing would it be to calculate the time taken to use the 35-mm camera that is given as an example? Someone should probably take the time to figure it out, but a Back-of-the-Envelope action analysis would be more useful at first. It would give us a picture of what could possibly cut and what could possibly be confusing. Very important info.
Eric Yoon - Oct 25, 2006 11:23:24 am
Heuristic Evaluation. One thought that stood out to me was the counterintuitive willigness of evaluators in heuristic approaches to answer the questions by the testers. When I was doing user testing, I just let the people struggle, in order to imitate a real-life setting. But it makes sense to help out and address the issue rather than wasting valuable time watching the person get frustrated. The emphasis on the importance on getting a lot of data from lots of different, isolated testers makes a lot of sense as well.
Evaluating the Design Without Users So much of the testing so far has been based on looking at what users have to say, so this title made me wonder how effective non-user-based methods could be. But some do make sense to me. The key to uncovering their value, it seems to me, is to clearly understand what each kind of test offers and what it doesn't. For example, the cognitive walkthrough doesn't test whether or not the proper steps are known or can be easily recognized. That's a task better left for user testing. What it does test is how challenging it is to actually figure out how to do a necessary step.
Michael Moeng - Oct 25, 2006 11:26:08 am
- Evaluating the Design Without Users:
I thought the section on cognitive walkthroughs had a somewhat cyclic argument. Usually problems one needs to catch in evaluation are exactly the ones the designer did not think of in the initial design--analyzing one's own walkthrough to find these problems would probably not catch a majority of the issues. From the article: "Users often aren't thinking what designers expect them to think." This is exactly why user evaluation is so valuable. I did think the "back-of-the-envelope" analysis was useful. Listing all the actions the designer expects the user to perform, then going through each step and checking for complexity or lack of knowledge could be very useful.
- How to Conduct a Heuristic Evaluation:
Alternating Heuristic evaluations with user evaluations seems like it would be quite valuable, as each user evaluation allows the evaluators to further emphasize or de-emphasize certain heuristics they used (if many users complained about aesthetics, and many were able to easily recover from errors, then more emphasis could be placed on the aesthetics heuristic).
One problem is that repeated heuristic evaluations can lead to unconscious de-emphasizing of important heuristics--like documentation or "recognition rather than recollection," as the evaluators get used to the interface.
Rayhan Lal - Oct 25, 2006 10:55:05 am
Heuristic Evaluation: I like the idea of having a more general audience evaluate the heuristics of an interface. Nondomain experts can still have a very good idea about what makes a usable or problematic interface. As the reading demonstrates the approach can be extremely cost-effective and one is able to evaluate the design without pestering the end-user. The method is a wonderful complement to the user testing we have been doing but certainly not a replacement.
Evaluating the Design Without Users: This article exemplifies my thoughts on the first reading. I initially felt like we should have done a cognitive walkthrough, action analysis or heuristic evaluation before prototyping with the end user. However, it seems like one would have to do a cognitive walkthrough of sorts (since we didn’t formalize the process) when doing the UI sketch after the contextual inquiry. I find it extremely informative that the article describes when each method is appropriate. As other have mentioned the list of heuristics is short (but still important), so it is important not to limit oneself to heuristic analysis alone.
Tony Yu Tung Lai - Oct 25, 2006 11:25:38 am
Reiman: when I read that it takes 2 to 3 seconds to perform an action, even a simple one that such as 'choosing a cursor', I didn't believe it too much. In my mind, I gave it half a second, may be a second. But after reading that and look at other people when they are using the computer, it is funny how from start to finish of an action, it does take a couple seconds. I am still not completely convinced that it takes more than a second for simple tasks, but for all tasks that starts from a 'complete stop', it seems to be the case.
Nielsen: I find it interesting how Nielsen pointed out that testers participating as heuristics evaluators are more willing to actively evaluate the interface than tradition user study participants. Even though they are essentially doing the same thing, but the role of being the 'expert' evaluator- someone who is finding the problems, rather than being the novice evaluator- someone who expose problems with the interface through his/her mistakes made the difference in the participant's mindset, which effect their willingness to participate. Amazing how rewording can be so powerful.
Sung Yi - Oct 25, 2006 11:57:06 am
Heuristic Evaluation:
Heuristics evaluation is performed by having each individual evaluator study the interface alone and gather their findings together afterwards. Independent and unbiased evaluations may be ensured this way, but I think this on the other hand also limits the variety of findings since discussing and sharing thoughs with other observers can certainly greatly improve the outcomes (this may slow down the evaluation process however).
Evaluating the Design Without Users:
During the walkthrough, there are 4 questions to keep in mind as you critique the story:
1. Will users be trying to produce whatever effect the action has?
2. Will users see the control (button, menu, switch, etc.) for the action?
3. Once users find the control, will they recognize that it produces the effect they want?
4. After the action is taken, will users understand the feedback they get, so they can go on to the next action with confidence?
I think one more questions should be added in:
5. After the action is taken and the user wishes to roll back, will the user see how to go back?
Michael Mai - Oct 25, 2006 12:03:06 pm
As a whole I found these readings informative and light on the reading compared to others we've had. I liked the fact that there were a lot of visuals for understanding of the various processes.
Heuristics: In regards to evaluators, the reason that about 5 or 6 are needed is because different people approach different problems (the UI) differently. People tend to see the problems that affect them while they are going through the analysis of the UI. Because of this, I do feel that it is important to have at least 3 testers from the pool of possible users. Perhaps have them be a different age, gender, location, etc. In regards to the ten heuristics, these rules are nicely laid out, but unless the evaluators have these memorized and are professionals at these kind of analysis, trying to have each evalutor all ten can become very cumbersome and time consuming. Perhaps splitting the tasks up between evalutors may be a useful idea. Comparing the number of major vs minor usabiltiy problems, the number of minor ones should, in my opinion, always be significantly greater than the major ones because many of the major ones can be coded out/seen by the original creators of the UI.
Evaluating the Design without users: One of the ideas that struck me in this reading was eliminating unknown actions. Although it would be nice to eliminate most/all of the actions performed by a user, I feel that eliminating the little steps is not the best road to take. I am sure that some of the actions are completely unneccessary, but just because the user doesn't see a reason why an action should be done does not mean it should be eliminated. Perhaps having signals or directions to indicate the action needs to be performed is a better outtake. The formal action analysis seemed like a complete and more difficult flashback of the KLM process. I hope that just a basic understanding of the concepts and numbers from the formal action analysis will be sufficient for most testers, otherwise someone should be highly trained from the group to do it. Combined with user testing, I believe that these techniques can be used to find a high percentage of the usability problems for an interface.
Ming Huang - Oct 25, 2006 12:03:13 pm
Like many previous posts suggest, the heuristic evaluation techniques Nielson writes about are closely related to direct-manipulation principles, and a usable interface is considered to be one that manifests directness to its users. Although today any piece of usable software necessarily includes a large number of components and interface elements that I doubt whether the guideline of having 3-5 evaluators per product/interface is enough.
Lewis and Rieman have sound methods for evaluating user interfaces designed around the task-centered paradigm. The cognitive walkthrough seems to repeat whatever is already in done in the design phase. When we are designing with task-centered approach, we think about scenarios of usage and try to fit our designs to best suit the common case while allowing room for occasional special cases. The cognitive walkthrough is a step in simply verifying our assumptions, and to me this task would best be done with actual, unbiased users than without them. Basically all of the listed methods encourages the designer to rethink the interface by “using” it, without aid from formal user testing or minimal aid from quick-and-dirty evaluations.
Patti Bao - Oct 25, 2006 12:05:01 pm
Nielsen: Nielsen's heuristics definitely have their uses - it's easy to find problems quickly, the resource requirements are low, and you only need a few evaluators to get a representative group (assuming a relatively homogenous group of users). However, it seems to focus more on identifying problems that are easy to spot as opposed to those that can only be found through in depth or repeated use. In another class, the recommendation was made that usability testing should involve a few sets of heuristics: a general set (such as Nielsen's), one for the same kind of application (such as Bruce Tognazzini's for websites), and one for the application in particular (such as company guidelines or competitor heuristics). I wonder how this practice actually plays out in the industry. At any rate, while most heuristics seem to be based on experience (as rules of thumb often are), Usability.gov has a huge set of heuristics based entirely on research, which I found interesting as well.
Lewis & Rieman: I wish this reading had been assigned earlier, given how hard it has been to find users so far! We definitely could have benefited from doing a more complete cognitive walkthrough as there was one fairly important control on our interface that none of our lo-fi prototype testers spotted. However, the problem still remains that at the end of the day, we are not our users, and there is always going to be something wrong that we can't see. The hardest part for me is to distance yourself enough to be able to objectively and critically identify what works and what doesn't with your design, but perhaps that's something that only comes with more experience.
Joe Hart - Oct 25, 2006 11:45:54 am
Heuristic Evaluation The magic number of 3-5 evaluators is interesting. You would think that as the number of evaluators goes up the return for the investment would rise as well. Even though the diminishing returns makes this not very costeffective important bug/usability problems can still be uncovered. The major issues may have been solved in an application but there is a point at which too many small errors buildup to a problem or give an end user the feeling that the application can not be trusted. A designer must be sure enough evaluating has been achieved toavoid this.
Task-Centered User Interface Design Isn't this what everyone does when designing something? If you are not designing with the end user in mind then what are you designing to? I guess just stating the fact that a designer should actually think before acting (coding) is good to recognize. Coding can't be done in a vacumm and this is a great place to start when putting your ideas into action.
Roland Carlos - Oct 25, 2006 12:03:38 pm
Heuristic Evaluation: Interesting balance you have to play with heuristic evaluation. You definitely don't want just one evaluator for an interface, since they will most likely only find a low percentage of the problems in your interface. However, you don't want too many because by that point, you're involving too many people which may cost too much real money in the real world, at which point, the ratio of benefit acheived from evaluation from cost is too low to be worthwhile.
Of course, with a lot of the stuff we've been finding out in class, even this fact can be summarized in a few graphs and math equations.
This is definitely a useful reading for our projects though. I kind of wish we had this before the low-fi prototyping, so I would have had a more detailed view on what to look for when evaluating the user's comments.
No surprise either, that user testing ranked so highly on the usefulness scale, pretty much putting every other testing method (besides heuristic) in the dust. Just more support for how important user testing is.
Evaluating the Design Without Users: I guess I'll echo a lot of the previous comments from students and ask why we didn't get this reading earlier in the semester. I feel like we may have done some of the practices mentioned in this reading in our discussions with our respective groups, but to have a formal process to work with would have accelerated the process. It's just like with brainstorming, everyone knows how to do it, but until we got some of the pointers from lecture, we didn't really know how to do it at maximum efficency.
Still, quite an interesting read. It definitely won't replace testing *with* users, but there are definitely situations when this practice may be worthwhile, like early on in the design cycle or when cost/time is an issue. The main thing is that we have to try and think as much like the user and get out of the designer mindset. Many designers feel their designs are infalliable, but as we learned previously, a designer's vision and the user's vision usually have a large disconnect in reality.
Hiroki Terashima - Oct 25, 2006 11:55:12 am
Reiman: I agree with Chen; the usability heuristics will be useful for us later on this semester. And yes, as Tony mentioned, this reading does shed light on important but often neglected issues of usability. I found the rating of usability part very useful. Severity, impact, and persistence. Since becoming a webmaster of wise.berkeley.edu, I've had to consider all 3 issues numerous times, and I've come to develop a good method of determining which bugs to fix first. The consistency and standards is important but hard to get right, especially if you have a bunch of people working together asynchronously.
Nielsen: The mental walk-through part was interesting, and I know from experience that it has some advantages and disadvantages. I don't think there is one right way of doing things, but a combination of multiple things that come close to the right way (otherwise things would be boring), and this reading reflects that: cognitive walkthough, action analysis, and heuristic evaluation together make a good design evaluation process without the users. However, as with any new skill that you have to learn, each of these tasks are very arduous and unattractive at first. For example, fixing the interface after a walkthrough sounds like a no-brainer, but it gets really cumbersome after several iterations...you want to just present it to a user and have them test it too, to get a real, lively response. It would be really dumb to do everything without having the user involved because we all have some bias/tendency towards our own work, even while following the heuristics or planned actions; it's just human nature.
Michael Udaltsov - Oct 25, 2006 12:33:55 pm
Heuristic Evaluation - The article states "Heuristic evaluation involves having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the 'heuristics')" but that seems somewhat incorrect, since the presented heuristics are just common suggestions for usability, not strict rules that everything can be judged against. It would also be pointless to have evaluators just go through each heuristic at every step of evaluation - instead the heuristics should be presented as general guidelines, and each user should evaluate the problems that they see, not necessarily specific to any category. Another thing that seemed odd to me is that the evaluators get to rate the severity of the problems. While it's true that combining several ratings would create a more meaningful result, it does not take development time into consideration. What seems to be a bad problem to one user may be insignificant to another, but it's really up to the developers to determine how long fixing it may take, and how many resources to allocate to it. At the same time, enough priority should still be given to less important problems, otherwise the final product may end up with no major issues, but with many smaller irritating bugs that ruin the users' experiences.
Evaluating the Design Without Users - A cognitive walkthrough seems like a great way to test the usability of a design without additional users, but at the same time, it's a difficult task for the designer. Since the designer knows all the specifics of the system, he or she has a lot of existing knowledge and assumptions about it. So when thinking about the imagined user, it may be difficult to include every little detail of the interaction. This could lead to only identifying the more obvious and bigger problems, and missing the smaller ones that real users would notice. Action analysis is much more detailed, but is really used for different results - timing of actions and general issues instead of specific problems with the interface. As the article points out, it's rarely used because its complicated, but some parts of it may give ideas about how a user should think when using the interface, which may lead to improvement. The Back-of-the-Envelope analysis is a generalized version that doesn't include every minor detail, but just like the regular action analysis, it doesn't always point to any specific problems, and does not necessarily show any possible solutions. In a way, it seems to be just like criticism with no positive feedback - only the errors are shown with no suggestions on how to fix or improve them.
Huangnankun - Oct 25, 2006 12:38:49 pm
Nielsen Heuristic evaluation make use of subjects to test out an interface. This reminds me of the low-fid protoype stage where the protoype can be used to gather further data to test out the actual interface. Given the rules in the "how to do heuristic evaluation" section, this can be done perfectly at the low-fid prototype stage. The heuristics which are used int he evaluation can be mostly modeled at the low-fid protype stage. The visibiliy and Match ( Another word for mapping ) and consistency. However error recovery might be hard to model at the low fid prototype stage. The comparison of user testing vs heuristic evaluation does not actually outline the significance between them. After all, one can argue that Heuristic testing is a form of user testing as well.
Lewis & Rieman This article talks about user interface evaluation without users by basing actions on Tasks. I think the content of this article is very similiar to teh GOMS process which we have already learnt. Both process makes use of task/goal centered actions and analyse the efficiencey of the interface based on how well it is tailored for performing these tasks. In fact the Keystroke level analysis is an essential part of the evaluation process as described by the article. The article also talksa bout heuristic analysis, which makes use of guidelines on usability. I think the entire evluation should be a cyclic process, where heuristic analysis and KLMS are used in parallel one after another in order to make sure the interface is good.
Jason Lee - Oct 25, 2006 12:27:33 pm
Nielsen: After reading these sets of articles, I'm still unclear as to what the difference between a heuristic design and a user test is. (The subheadline "Alternating Heuristic Evaluation and User Testing" seems to suggest that there is a difference between the two). The best I can come up with is that heuristic evaluation seems to be evaluation based on the ten guidelines presented, while user testing is just users testing the system in general. In a practical sense, there doesn't seem to be much difference between these two in my opinion. What was interesting about the read was the ten guidelines that were pointed out. Certainly these are all things that seem painfully obvious when designing a userface, yet it is difficult to design an interface on the first try that adheres to all of these. Though we try to consider all aspects of users and the potential problems they may face, it is inevitable that we as designers will forget something that is quite important, and so evaluation and testing is needed.
Lewis and Rieman: For me, a cognitive walkthrough would not be very practical, as I would find it difficult as the designer of an interface to get into the mindset of someone who has never seen the interface. The persona I would design would inevitable end up thinking exactly like me and any design choices in the interface would instantly make sense to me. This is a dangerous assumption to make and one that I probably would not be able to avoid making. On another note, it was interesting to come across the heuristic guidelines again, only this time with nine instead of ten. Even so, in so many words, they still managed to capture the essence of the Nielsen document, with a corresponding heuristic to each Nielsen heuristic (with the exception of "Help and documentation". Perhaps Lewis and Rieman felt that a UI should be designed well enough that extra documentation should not be necessary. Either way, these are good guidelines to keep in mind and would have been handy earlier in the semester when we were still designing our lo-fi prototypes.
Vijay Rudraraju - Oct 25, 2006 12:36:33 pm
Heuristic Evaluation This article is timely because in developing our interactive prototype it is necessary to make some informed design decisions without having to resort to user testing at every step. Although the interactive prototype is simply that, a prototype, it is important at this stage to be more careful about design choices because from this point on mistakes and corrections start to become exponentially more expensive. Heuristic evaluation gives some guidelines to avoid major design mistakes in the interactive prototype, which would most definitely lead to user frustration and reduces the cost of the next design iteration.
Evaulating Design Without Users I find the cognitive walkthrough technique to be a much more humane form of evaluation than the strictly heuristic evaluation technique because it finds flaws in the actual story of the interaction and user intentions rather than the disconnected interface elements. Often it is the actual process as a whole that is created by an interface more so than the discrete elements of the interface, which can lead to frustration and are more difficult to catch. The walkthrough presents a technique for addressing some of the these shortcomings of heuristic evaluation.
Scott Friedheim - Oct 25, 2006 12:24:43 pm
Heuristic Evaluation: The questions that Heuristic Evaluations answers would have been nice to have during our Contextual Inquires. Specifically, the ten recommended heuristics. I was kind of impressed by how dead on these ten recommend heuristics were. Usually I can find a few rules that just really don't seem practical or achievable but this list is a very good summary of a designers design goals; This will definitely be a list to use later on in the design cycle. As far as the actually heuristic evaluation of a design goes, the most important idea, and the one that seems very logical is to disallow evaluators communicate and have their findings aggregated until all of their evaluations are done. If communication was allowed then one might mention to the other a flaw that really wasn't important to another thereby generating false evaluations.
Task-Centered UI design: A Cognitive Walkthrough does have some good aspects in terms of generating ideas or a sort of brainstorming of how and what a user will encounter. It allows for an analysis of all the assumptions the designers made; this being the most critical component because usually designs fall apart when assumptions of a user are not accurate. But I found, as many others have already mentioned, that this is still somewhat of a weak method of evaluating a design for flaws. It just isn't practical to think that the designers could jump into the roles of the users and *poof* they are now newbies to the system with. Even more, there is a hard line to determine the knowledge and experiences that users bring to the table before using the design. Overall the Cognitive Walkthrough is valuable in further analysis of a design however, it should not replace a formal contextual inquiry with the users.
Siu Pang Chu - Oct 25, 2006 12:19:04 pm
Heuristic Evaluation:
I like this method since it is quick, cheap, and easy evaluation of the user interface. Heuristic Evaluation use a evaluators , a small set of testers examine the interface, and judge its compliance with orginally designed usability principles. The goal is to identify the usability issues so the it perform correct function. I believe Heuristic Evaluation will be very useful in Web development circles because it requires few resources like time or code. So any developer can enjoy the benefits of usability testing. Heuristic Evaluation need several evaluators to do it well, choosing evaluators is also an important issue for this method. And figure shows that 15 evaluators are enought to find out all the usability problems.
Evaluating the Design Without Users:
The article describe three methods for evaluating an interface without users including Heuristic cognitive walkthrough, and "back of the envelope" method. The article also gives the order to perform the evaluation. In general, the cognitive walkthrough will be performaed first, it will give you the best understanding of problems it uncovers. We select one of the tasks that interface was designed to support. Then we try to tell a path of the actions that the user has to do to perform this task. Then , we can perform a heuristic analysis that is a good way to catch additional problems but we need several good evaluators to do it. Back-of-the-envelope action is the last analysis that can make good "sanity check" as the interface becomes more complex, and this method is also useful in the earily system design stage to help find out whether a systems's functionalilty are doing well or not and whether it is easy to learn.
Dexter Lau - Oct 25, 2006 12:54:52 pm
Heuristic Evaluation:
The evaluation process described is essentially what we have been doing thus far with our own projects. Having a more informal personal evaluation, the interview can more effectively use her notes in order to refer back to them quickly. The interview can also note body language or other cues that might not be otherwise seen via other reporting methods. The top ten recommendations for heuristics are helpful general rules to follow when iterating through an interface's design. Many of the points note to a mapping for the real world and system world in order to minimize the gulf of execution and evaluation. When iterating through the evaluations, it is important to use heuristics to order and prioritize the major and minor problems using a 1-5 system. It is also important to note that more evaluaters can provide better results, but they are also more costly -- a good link to real world applications. The method has been adopted quickly due to its ease of use with free-form comments and graceful degradation
Evaluating the Design Without Users:
Again, this is another evaluation method that we have been using a lot already. In this method, it discusses the GOMS approach where interfaces can be evaluated much more scientifically. Each keystroke, homing, reading, and every other action is accounted for when executing a task to complete some sort of goal. This method has its benefits because testers are not an unlimited resource. Moreover, testers are fairly inconsistent and may not catch every flaw in a interface. As thorough as this method is, it makes a lot of assumptions, such as perfect use.
Tom McClure - Oct 25, 2006 12:54:38 pm
Task-Centered User Interface Design
Of the three of the activities described -- cognitive walkthrough, action analysis, and heuristic analysis, not to downplay the importance of the other two, but the heuristic guidelines seem like the most bang for your buck with regard to making a UI actually useable. The guidelines laid out seem so simple and obvious, but so many projects get waylaid with feature creep and unrealistic deadlines that sometimes good design principles are the first thing to suffer. Designers should keep the nine guidelines printed out and taped above their monitors as they work to remind them to constantly ask themselves these questions. This is especially important if the project, like so many, has not budgeted time (or has cut its budgeted time) for QA and testing with users.
Of course, one or two iterations of the design cycle should naturally bring most problems of this nature to light, but the design can only get better if these types of things are taken care of before they even hit the users in a first iteration.
Vahe Oughourlian - Oct 25, 2006 12:21:16 pm
Heuristic Evaluation
This set of tutorials is basically the practical application of several of the previous readings: a systemic approach to finding and eliminating usability problems. I like that it puts empirical data to use, and it does not shy from pointing out the flaws in it's method, such as applicability to certain situations, since not all user interface problems may be a problem in one context versus another. The rating system for the quality of the user interface error I found intriguing, as well as its justification. That people would simply gloss over the severity of the user interface error is common while looking for other user interface errors to fix. However, I find it amusing that, according to some employees I know at a large software firm, this exact method is used to justify the cost of fixing an error or not. It works great in a business sense, but my opinion is that it leads to laziness in regards to actually fixing the errors.
Evaluating the Design Without Users
This document tends to move backwards in the sense that we seem to have been taught to take into account a user's opinion on the user interface, moving away from the idea that the designer knows best. While it does give guidelines (heuristics, if you will) on how to be self-evaluating, it makes little mention of this technique in addition to the other techniques we've observed.
I liked the walkthrough section, because, like the previous reading, it seeks to apply empirical data to help redesign the interface. Theory is all fun and good, but using data we have is much more satisfying. I especially like the observation that bad interface control design can be hidden by good visual design for the interface. I can think of many examples, OS X among them, where making it look pretty makes up for obtuse user control. Also, the feedback issue is prevalent in the difference between OS X and Windows. Whereas OS X gives you a bouncing dock icon (which users use as a metric for the performance of their computers), Windows gives you no indication as to if an application is launching or not. Having just switched back to my PC after my Mac died, I noticed this slight frustration immediately.
Andrew Hao - Oct 25, 2006 01:25:45 pm
Heuristic Evaluation
Good ol' Jakob Nielsen, here to save the world from the the wrongs of interface design. His ten heuristics allow complete idiots as I a way to come up with a semi-decent interface without having to really do any costly user testing. I remember reading his heuristics a few years ago and thinking to myself, my God, the man is a genius! And I suppose in reality, the heuristics aren't floor-shaking as they are trumpeted to be, but simply good common sense.
(An aside--I have always wondered how Nielsen became UI's Big Man on Campus [asides from being a UI pioneer and all that -- his useit.com website has always struck me as an eyesore])
Task-Centered Design: Evaluating Without Users
The author identifies the main problem with the cognitive walkthrough: simply, there are still no real users. As good as these general self-checking methodologies are, we still cannot accurately place ourselves in our users' shoes until we actually get them in the driver's seat. Though I do acknowledge the author's disclaimer that the method is used more to "monitor your designs as you work", it does not fully replace user testing.
Aleksandr (Sasha) Ashpis - Nov 01, 2006 12:25:39 am
Lewis and Rieman
1) I surprised and impressed that when discussing the part on who should do a walkthrough, the author clearly states that is should be done with people that are roughly at the same level in the company hierarchy, because if this isn’t the case then other issues come into play as the author points out. It is interesting to me how scientific pressing a key on a keyboard or learning a single step in a procedure can be. It makes me question on how they concrete time measurements are derived and how wide range of sample pool was used to conduct the experiments.
2) The Heuristic analysis or guidelines setup by Jacob Nielsen and Rolf Molich seem to be well encompassing and cover a broad array of issues. If a user interface takes account of all the heuristics mentioned it should be fairly complete. The one thing I think is missing, but it is not a major category, is the way or the how to become an expert user. For example, under the provide shortcuts section, it states that shortcuts should be provided in order to help experienced users avoid length dialogs and informational messages that they don’t need. This is all well and good, but how does one learn the shortcuts, it is my experience that shortcuts are not listed in any easily reachable location, and most shortcuts are learned through poking around or word or mouth. I believe a side note to the providing shortcut heuristic is also providing the ability to learn short cuts.
Nielsen
1) While describing how to conduct a heuristic evaluation, he does not mention what an ideal number or even a suggested number of evaluators needed. Naturally one would think that the more the better, but in business money is always a factor, so the question I have is when does it stop making business sense to invest in more evaluators? I understand that this heavily varies by situation, but this author and the ones from the other reading seem to be able to quantify everything else.
2) I believe that severity rating are critically important but they carry with them some bias of the person setting the rating. This dilemma is intended to be over come with a formula like approach to assigning severity ratings but still the person setting the number has a big sway. For example, a small problem may be pushed to the top of the list merely because someone in a place of higher authority has said to do it. Furthermore, a more common case in industry is if a company has a big client, that client’s problems will automatically gain severity merely because they are from a big client, making other problems with actual higher severity be put on the back burner.
Charles Lee - Nov 01, 2006 02:01:08 am
Heuristics Evaluation 1:
- Regarding "ProblemsFound(i) = N(1 - (1-l)^i )", it appears to me that the "l" proportion is hard to measure, hard to define, and as a result, is as reliable as voodoo. It represents the proportion factors each user finds. That would be a fraction, where A.) the numerator varies depending on user...use an average? and B.) the denominator varies a bit because some issues are only problems for some people. The result is an exact-sounding calculation based on non-exact measures. Precision of the answer is as good as precision of the smaller parts, so the calculation is not very useful, especially as "i" increases.
Heuristics Evaluation 2:
- The severity ratings in heuristic evaluation is a topic I have seen in many places. This tool is used a lot in industry to track problems, and deal with the most important ones first. Depending on how understaffed the company is, the lower priorities can sometimes mean "ignore". This is unfortunate, because sometimes one person's afternoon can solve several hundred easily fixed typos/low-priority small problems. Another column for rating of ease-to-fix or ease-to-implement should be used as well.
Evaluating the Design Without Users 1:
- It seems to me that evaluating the design without users is pretty similar to being a considerate person - some people just are better at it than others. The designer cannot think of a process that is supposedly intuitive to the average users, he can only design a process intuitive to _his perception_ of average users. As such, success is largely reliant on an accurate perception of people, which not everyone possesses. This can be practiced, perhaps, but I would say an unsurmountable advantage goes to people who are normally "considerate" on a day-to-day basis, meaning that they easily and often consider others.
Evaluating the Design Without Users 2:
- The article outlines 3 "failure stories." Of those, 2 of them ("What is the user thinking" and "Can the user identify the control/feature") pertain directly to my first comment. However, for "Can user find the feature," this failure story stems from planning out a good interface, balancing the layout between confusing and having as many features available as possible. This is the failure that has the least room for excuse. Although it is slightly subjective, this is the bare minimum of what people should be drilled with.
Bryce Lee - Nov 01, 2006 11:18:37 am
Nielsen
Nielson highlights important considerations for heuristic evaluation. However, his points are weakened by the number of details he tries to incorporate into his argument. For example, the point that the best evaluations happen under a specified amount of testers may have been applicable to his clients, but I think it's unreasonable to assume that this is universally true.
While the specific details detract from his argument, he does recognize the real world applications and adoption of heuristic evaluation. However, again the details behind his reasoning of real-world adoption is plagued by a lack of data. He bases his arguments off of the responses of less than a 50% feedback from a small group.
Lewis and Rieman
The second article makes a good point of being critical about the instructions themselves. Many designers believe that the development of a process, which is clear and concise to them, is all that is needed for the user to easily use the interface. However, putting these steps themselves under the microscope can reveal serious misconceptions and assumptions made about the user, not the system. Also another good point made is the workflow between purpose, identification, and action. Often times, designers take for granted the first two.
However, there seems to be a lot of subjective decisions made in the examples. In the Mac Background Printing, two actions are merged into two since the author says that "the analysis should reveal pretty much the same things." I don't see any benefit other than an easier drafting of the walkthrough for the developer - a mere shortcut. Also, at one point they conclude that faster hardware speeds would trump any decrease in user activity time. This is a critical flaw of the example; even in programming, algorithms are designed to be efficient despite the architecture or future improvements. I don't understand how user interfaces would be different.
Jonathan Chang - Nov 01, 2006 12:24:40 pm
Reading 1: We tried this out in discussion yesterday, and I actually found it a very useful thought process to go through. They weren't necessarily all things that you are most concerned with when you're developing an application, either. For example, exiting smoothly from the application is something I don't think I've ever considered while coding a project; I've mostly been concerned that thing worked properly once inside. However this can be a cause for anxiety for users that aren't familiar with the system.
It was also interesting for our sub-group, and actually most of the section, to note that not all of these heuristic evaluations applied to all types of applications. We were evaluating woot.com and realized that things like error messages, state feedback, or clear exits were difficult to show on a simplistic webpage. While we sort of thought this was just a characteristic of it being such a simple interface, its also important to consider the heuristics in terms of what its designed to do, not just the letter of the particular rule. Exiting can refer to backing out from an online purchase, or knowing you're about to make a purchase.
Reading 2: A cognitive walkthrough is a great idea, because it forces you to consider all aspects of what a user might be thinking, instead of potentially ignoring interface problems. This goes so far as not assuming that, if a printer is turned off, a user would instantly know to turn the printer on: there is little precedent for such a situation, and some kind of informative feedback would be instrumental in helping them along in their process.
I think performing a cognitive walkthrough on our own application will lead to more changes, just as the transition from lo-fi to interactive prototyping made significant changes in our interface design. It's hard for us to step back from the design process when we're in the moment, especially when aspects of the design are challenging, to take into account exactly what a user with no experience with our application might be thinking about it.
CharlesLeung - Nov 01, 2006 11:31:15 am
Nielsen
1) The listing of heuristics is very interesting because they seem simple, but they actually are very important concepts. For example, the first one stated is visibility of system status, and after I thought about that concept I realized that our interactive prototype actually needs a more accessable system information. For example, if we included a clock in our interface, then our program would become a lot more user friendly. Keeping these simple heuristics in mind will be very helpful for future projects.
2) I thought that the two curves relating the number of evaluators and the problems found and the ratio of the users and problems found very interesting. I think that it was especially interesting to note that in the second graph the ratio starts off very low and then hits a global maxima before slowly tailing off. At first this struck me as strange because I thought that the curve would be monotonically decreasing. However, I realized that teamwork and the externalities gained by working together can account for the global maxima not intersecting the y-axis.
Lewis and Reinman
1) I think that evaluating your own design probably won't be that much help because if we were the ones to evaluate our own design, then we would probably only look for things that we have already thought about and implemented. Although it may be impractical to try to bring in others to do a cognitive walkthrough for a very small project, I still feel that running through a cognitive walkthrough by yourself would not do very much good. Especially if we're looking at a description of the project that we ourselves wrote down, I fear that it would be too hard to have a lot of useful feedback.
2) In terms of action analysis, I like the "back of the envelope" approach far more than the rigid HCI approach we learned earlier. Although the HCI approach does seem useful and apparently is a fairly accurate predictor, I would have to agree with the authors in that for the few tens of a second that can be saved through very rigorous analysis, it does not seem to be worth the designer's time. In fact, it seems that just learning basic principles about design would be a much more effective use of time.
Qingyun Tang - Nov 01, 2006 12:24:40 pm
- Heuristic Evaluation
It is quite common sense for us to know that different people have different views, just like different users have different habits and perspectives when use your software. What surprises me is each heuristic evaluation session for an individual evaluator lasts one or two hours. I thought it would take less time to do so. Each evaluator taking an hour means 8 evaluators will take a whole work day for someone to do the evaluation. However, heuristic evaluation is often easy to generate a revised design according to the guidelines provided by the violated principle for good HCI systems. Since each evaluator has limited views of finding usuability problems, it is a great idea to have severity ratings. The ratings can help evaluators access various problems easily even if they have not found them in their evaluation session. Many companies have just recently realized the urgent need for increased usability activities to improve their user interfaces. Thus heuristic evaluation becomes popular soon after showing its effectiveness.
- Evaluating the Design Without Users
The title is quite shocking to me that it is telling us to evaluate the design without users. After reading, I feel evaluate this is quite reasonable to do it before hand. Users' time is usually limited and we cannot always expect to find people to evaluate the program we design, especially there is a lot of bugs that we can find by ourselves. If we can fix those bugs before letting the user evaluate for us, it would help much more because users don't have to catch our technical errors but design errors. This article talks about Heuristic evaluation again. It is good to hear that Nielsen and Molich have developed developed and tested a procedure for using a short list of general heuristics to evaluate a design. However, if none of the evalutors have experience, it would take approximately 15 of them to find 75 percent of the problems. The science of heuristic evaluation doesn't promise the results to be correct; more evalutors and more results only increase the accuracy of some specific problem to be identified if it is mentioned more than one evaluator. I didn't agree with the article that more evalutors increase the overall accuracy, because there is always bias.
Jae Chang - Nov 01, 2006 12:50:39 pm
Heuristic Evaluation
Comment 1: Heuristic evaluation is a good tool to find usability problems in UI or program design. It was very interesting in the fat that heuristic evaluation is the most popular of the usability inspection methods. In common, people might think that as the number of the evaluators is increased, benefit is increased. However, as the number of the evaluators is increased, the cost of evaluation is increased. The formula for the number of usability problems found in a heuristic evaluation, ProblemFound(i) = N(1 – (1 – l) ^ i), leads cost-benefit model. 3- 5 evaluators are very cost effective as well as benefits. Even if there are many free evaluation copy software in internet, I believe that very few users reports bugs in a program. When a company needs to evaluate their product, hiring 3-5 evaluators for heuristic evaluation can be more cost-effective and benefit-effective than internet evaluation copy distribution.
Comment 2: The ten general rules for user interface design called heuristics gives very good guidelines how to build or design system. The ten usability heuristics approach from user perspective not from designer or developer perspective. Developers tend to think generating correct error message is important because the error were generated by user. However, the rule tells us that even better than good error messages is a careful design which prevents a problem from occurring in the first place. By this approach, developing cost and time may increase, but many errors can be eliminated so that the system or application may be used with short period of evaluation. I believe that the ten rules are very crucial and should be followed by every designers or developers.
Evaluating the Design Without Users
Comment 1: This is a method when designers or programmers design a system. The method is widely used in the industry. For cs160 project, every members of our team followed a cognitive walkthrough even if we did not learn the method. The article shows why the cognitive walkthrough is important and how to apply it so that we can make our design better. The benefit of this article is very high for every one in the industry. It was also very interesting how author evaluators are more willing to evaluate the user interface than other user study.
Comment 2: As author say, the cognitive walkthrough gives us the best understanding of problems it uncovers. The cognitive walkthrough also gives us to think the designed interface in walkthrough terms as we develop it. However, the walkthrough will make the design changed more rapidly which is very hard because developing product usually should be done in short time of period. The three techniques, cognitive walkthroughs, action analysis, and heuristic analysis, should be combined very effectively so that developers can meet the developing time due.
Keenahn Jung - Dec 07, 2006 10:59:59 am
Nielsen
1) The 10 usability heuristics listed are very important. A lot of UI's in production now miss a lot of these. For example, leveraging standards and conventions. There seems to be no conventions for the location of the "options" or "preferences" dialog box. It is only through much practice that I remember that Firefox's "options" menu is located under the "Tools" menu (which makes little sense, but is at least consistent with MS Word), while the "configuration" menu in Ultraedit studio is located under the cryptically named "advanced" menu. If designers could solve these large problems, we would save a lot of time.
2) You need a team of evaluators to find all the bugs. I used to work in Quality Assurance, and often times I would outright disagree with my coworkers on what is a bug. Certainly, there are diminishing returns on the number of people in the team, but it is still incredibly useful to have more than one person evaluating the interface. Also, with better communication among users, they could skip the bugs that were already found and focus on finding new ones.
Lewis and Reinman
1) It is definitely important for designers to "eat their own dog food." While we need user feedback as well, conducting periodic sanity checks is a great idea. Often times, we may come up with an idea that we think is incredibly novel and genius, but when we go back to it, we realize that it sucks. If it's obviously a bad idea, we do not need to waste precious user interviewing time with it. Also, other designers can act as users in your prototyping, so it does not have to be a closed loop of feedback.
2) While GOMS is effective at giving rough estimates, often times simple qualitative measures are sufficient. For example, if a user feels that your interface is slow because of its nested menus, you don't really need to perform GOMS analysis to know that you need to pull out some of the options in the nested menus and expose them to the user. The vast majority of UI's out there have QUALITITATIVE problems that need to be improved before we ever hope to shave off a few miliseconds of time. However, GOMS is still very useful in comparing two qualititatively similar interfaces.
Siyan Wang - Dec 07, 2006 03:11:14 pm
Heuristic Evaluations 1: These guidelines for heuristic evaluations seem pretty comprehensive and straightforward, although some of the guidelines seem a bit unintuitive as well. For example, giving the evaluator hints seems a bit counterproductive, since their job is to find the flaws. If they get hints, they can easily bypass those flaws. However, it seems this method is in agreement with the iterative design cycle. since it makes use of representative tasks of the user for the evaluator to go through to evaluate the interface. This makes heuristic evaluation seem like a good fit in the design cycle.
Heuristic Evaluations 2: The actual heuristics used all seem like common sense but so many times it seems many interface designers bypass them. Most, if not all of the error messages I remember getting all seem to have some sort of technical goblygook that I can't make heads or tails out of. I really don't understand the interface designers' intent in displaying user messages that way, as it would seem to be a very common thing to avoid. The only thing I have a bit of a problem with is their heuristic to keep things minimalistic. Maybe it's just me, but I prefer a lot of options and whatnot. I'm always afraid in a minimalistic design that information I might use will not be displayed
Evaluating the Design Without Users 1: This cognitive walkthrough procedure seems like it would be pretty useful in conjunction with the heuristic evaluation described in the first articlbee. It seems that after making sure the interface satisfies the heuristics, it should go through the walkthrough for a more complete evaluation. The walkthrough procedure itself seems to be intuitive and obvious, and also fits quite well with the task-oriented design iteration cycle we have been studying.
Evaluating the Design Without Users 2: The keystroke analysis of the design walkthrough seems very technical at first, but then it seems like it would be very useful for more fine tuning of the design. However, both of these evaluation methods seem very broad and vague, not meant for precise measurements. I suppose this keystroke analysis is a good method to make it more precise so that one can organize specific improvements.
Robin Franco - Dec 15, 2006 12:23:01 pm
Heuristic Evaluation 1: In reading the ten usability heuristics, I found that it related to much what we've learned in the course. The one that stood out was “Recognition rather than recall”. One can also apply this to a reason to limit the use of “modes”. In a mode, one has to recall that they are in a mode, instead of recognize it. However, something I considered during this reading is that its possible for a mode to change the perception of an object. For example, if a keyboard displays the letters of a key on tiny screens, upon pressing the caps-lock key, the keyboard could change to display capital letters instead of the usual lower case letters. This type of “recognition” instead of “recall” would vastly improve the keyboard-user interface especially for novices.
Heuristic Evaluation 2: “Help and documentation” was a second heuristic that caught my attention. There is a fine line between being helpful and being annoying. One good example is the infamous Microsoft Word Paper Clip. This little animation would pop up with suggestions whenever it notices that one is typing a letter or other format of a document that the program recognizes. Much like other instances in which warnings or advice are eventually ignored, I personally ignored the help to an extend where I no longer payed attention to it. So when there came a time when it would actually be helpful, I was so accustomed to ignoring it, that it was of no use to me. So I think the author in this article should have pointed out that help should be voluntary and not forced onto the user.
Evaluating without Users 1: “Be consistent” is one of the tips given by this article. As much as its important to be consistent in a software application, the problem many developers run into is that every piece of software is “being consistent” with itself, but not necessarily with others. The problem with this is that it can be a jarring change in usability from one software to the next. Its very difficult for all developers to come to a consensus as to what is best for the users. To mitigate this, major operating systems such as Windows and Apple have released usability guidelines to direct their developers into creating a more consistent interface across all applications. Therefore, the author should mention that being consistent within one's own application is as important as being consistent with the whole user experience in general.
Evaluating without Users 2: “Provide shortcuts” is another tip given in the article. For the most part, I agree with it; shortcuts can greatly reduce the time spent on many tasks. However, its been known that presenting multiple ways of doing a single task to users can be more detrimental than helpful. When first learning a task, one wants to be a master of doing it one way before thinking of it in two different ways. So I believe it is important to hide the shortcuts from a novice user in order to keep them from being confused. After a user has gained some experience with the software, it would be good to expose them to simpler ways of doing it. Much in the same way we learn to multiply and divide by hand before we move on to using calculators.
