From CS160 User Interfaces Fa06

Contents 1 Introduction 1.1 System Introduction 1.2 Purpose and Rationale of the Experiment 2 Implementation and Improvements 2.1 Brief Summary 2.2 Bank of America / Wells Fargo 2.3 Gas Stations 2.4 Multiple Sheets 2.5 Improvement in error cases 2.6 Improvement in information 3 Method 3.1 Participants 3.2 Apparatus 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 5.1 Variables: 5.1.1 Ease of Use: 5.1.2 Accuracy: 5.1.3 Appearance: 5.1.4 Matches your needs: 5.2 Time Results 5.3 Task 1: 5.4 Task 2: 5.5 Task 3: 5.6 General Comments: 6 Discussion 7 Workload breakdown 7.1 Kang Chen 7.2 Melissa Jiang 7.3 Andrew Tran 7.4 Tak Wong 8 Appendices 8.1 The script 8.2 Instructions attached to the interface 8.3 The interface 8.4 Consent Forms 8.5 Usability Surveys 8.6 Notes from the Interview 8.7 Results Charts

Introduction

System Introduction

The system being evaluated is called Simple Searching System or SSS for short. It is a paper application empowering users to easily find information on ATM's for all the major banks, gas stations, and receive directions on how to get to a destination both by private vehicle and public transit. The main differentiators between the simple searching system and the usage of fancy, expensive, battery-dependent, and internet reliant laptops/PDAs are mobility, durability, extremely long battery life, instantaneous results, and internet independent. From the start, SSS was aimed and designed to require little or no technical background as well as a low learning curve. This is achieved by the few steps required to perform each task. For instance, to request information for ATM's, all the user has to do is circle a region on the map and tap the button corresponding to the bank in question. As for the other tasks, the only extra step necessary is to two draw circles in lieu of one, first corresponding to the starting location and the latter representing the destination.

Purpose and Rationale of the Experiment

The purpose and rationale of this pilot usability study is to test the changes we've made since the low-fidelity prototype, flush out usability bugs that went unnoticed in the prototype, and getting a benchmark for our system. This will serve to help breach the gap between the designers' model and the users' model of the system. It's necessary to test the changes we've made since the low-fidelity prototype in order to validate the necessity and benefits of the changes. This is due to possible bias from the low-fidelity prototype introduced by the selection of our users. It's certainly possible that the users from our low fidelity prototype suggested changes that do not generalize well to the general public. In that case, the pilot usability study will provide a larger user base to reduce such bias. On the flip side, it's also possible that all users from the lo-fi prototype missed some critical usability bugs during the interview. Again, a larger sample size of users, each having unique characteristics and participating in the study independently without collaboration, will be more likely to point out the bugs. Lastly, all we've done so far is theoretically imagine how our users would be using SSS yet this might be completely off from the actual scenario. The pilot usability testing will feature various measurements on our users' performances which would yield more scientifically sound benchmarks for SSS. These benchmarks will play a big role in helping us determine which parts of the system still require some finishing touches in the next cycle of development.

Implementation and Improvements

Brief Summary

The functionalities we implemented for the interactive prototype were finding Washington Mutual ATM locations, trip planning by public transit, and trip planning by private vehicle. For the interactive prototype, our system only supported one sheet with downtown San Francisco. We have implemented many new features since the interactive prototype assignment. Our system’s functionalities now include finding Bank of America and Wells Fargo ATM locations, and gas station locations with their current prices. Our system also now supports the entire city of San Francisco, now which we have 13 sheets total. We also implemented a usability error proof for our system. We still have not implemented sending the information to the user’s cell phone. We are still using the cell phone GUI for this assignment. The new features are further described in more details below.

Bank of America / Wells Fargo

In addition to giving the users Washington Mutual ATMs locations, we now implemented Bank of America and Wells Fargo ATMs as well. After circling an area on the map, the user now has the option of choosing any of these banks to find their ATM locations. The ATM locations include both single machines apart from the bank and machines that are attached to the bank itself. The user would not know if the ATM machines are with the bank, he would just receive information stating ATM machines exist at those certain location.

Gas Stations

As proposed earlier, our system now includes the feature of finding gas station locations with their prices. When the user circles an area on the map and checks the gas station box, gas station locations and their gas prices are sent to the cell phone GUI. The location of gas stations is given by the address of the station and the cross street as well. The gas price given to the user is the price of regular gas of that station. The gas station locations that are sent to the user are the ones closest to the user’s drawn circle. Closeness is defined by the zip code of where the user drew the circle.

Multiple Sheets

For the interactive prototype, our system only supported one sheet and the downtown area of San Francisco. We’ve now implemented our system to support multiple sheets. Now the users is not bounded to downtown SF, but can find ATMs, gas stations, and do trip planning within the entire city of SF. Our system includes 13 sheets; a cover page, and the divided 12 regions of SF. The cover page includes the entire map of SF, a reset button that would clear all the maps, and all the buttons the user would make check to request information. The map of SF on the cover page shows the 12 divided regions and what page each region corresponds to. The 12 individual maps include a region of SF and a clear button. The clear button would clear that page only. To find the area the user wants to perform a task, he would look at the cover page to find the page that corresponds to that area of SF. Then he will perform the task on that page, and go back to the cover page to make a check mark on the appropriate box. For the trip planning features, the user can circle his source location and destination location on different pages.

Improvement in error cases

Our system before handled all error cases where the user would draw too few or too many circles for any task. We now improved handling error cases with the case where user messes up in drawing his destination location. As a scenario, the user would draw his source location and then his destination location. He realizes that his destination location was wrong, so he clears the page with the destination circle and redraws it elsewhere. Then he would check the box for trip planning. We have accounted for this scenario in our system, making sure we distinguish his intentional destination correctly. However, we cannot support all errors users would make with our system if they do not read instructions. One error case that we have not yet handled is the case where the user needs to redraw his source after he already drew his destination. Currently our instructions state that if the user ever messes up in drawing his source location, clear the page and redo the task. In the future we plan on supporting this scenario. Another error case the user might make is if he draws his destination location before his starting location. We cannot support this mental error of the user.

Improvement in information

As stated in the interactive prototype, our maps include grids for the user to find streets and locations more easily. In addition to just giving the user the locations of ATMs and gas stations, we now included giving the user the grid of the locations as well. For example, if the user is seeking gas stations and the location of a gas station given to him is 1500 Market St. and 9th St., the grid containing that intersection is given to him also, such as (H8).

Method

Participants

Subject H: Subject H is a female Civil Engineering undergraduate student at Cal. She is 20 years old and currently resides at Berkeley. She works at San Francisco, so she commutes to work by public transportation a few days a week. Her work nature also requires driving within San Francisco, so this interface can possibly benefit her. She was selected through a friend who is a coworker of her and also because she is not native to San Francisco, but is required to go unfamiliar places in San Francisco. She is fairly familiar with both paper maps and online maps. Note that this is not the same subject H as the one in Low Fidelity Prototype.

Subject Ti: Subject Ti is a male intended Computer Science undergraduate student at Cal. He is 20 years old and currently resides at Berkeley. He is native to San Francisco, but is only familiar to a few places within San Francisco. Subject Ti is among the friends that one of our friends recommended because he does not drive, which leaves him less familiar with San Francisco in general. He is fairly familiar with online maps, but not too familiar with paper maps.

Subject To: Subject To is a male Information Systems undergraduate at San Francisco State University. He is 21 years old and currently resides at San Francisco. He is native to San Francisco and commutes from his home to San Francisco State University daily. Subject To was a classmate of one of our friends and was chosen because we want to see how our interface perform compare a person who normally takes the bus and is somewhat familiar with the bus routes.

Apparatus

The equipment we used are the Anoto pen, a stack of 13 sheets that includes the map of San Francisco and its subsections, one of our laptops that supports Bluetooth and the Anoto system and our implementation, and a location with internet connection. We also gave a paper consent form and survey to the user to fill out. For all 3 interviews, the location was at the interviewee’s room with a table so the user can mark on the map accurately.

Tasks

We start with a consent form that the user has to sign so they are aware of what this study is about. There are three tasks the user has to perform. The first task is to find gas stations of an area of their choice within San Francisco. The user has to find the top two returned results on the map. The second task is to find driving directions from one place to another within San Francisco. They are also allowed to freely choose their desired locations and have to show us the route that is returned. The third task is the find transit directions from one place to another within San Francisco. They can also freely choose their desired locations and they are told to find where they should get on and off each bus. Finally, they are given the usability survey to fill out and give some feedback on.

Throughout each task, we observe how each user performs the task, what they do when they are stuck, encourage thinking out loud, and improvements that they think will be helpful. We also ask them what they think about our potential improvements to make sure the new features are worthwhile.

Procedure

We start the interview by telling the user the brief overview and the purpose of our interface without telling them the details. The purpose is the find the closest ATM and gas stations in a given region in San Francisco. We also support giving directions from one place to another by their desired mode of transportation. Before we start, we hand the consent form to the user and ask him or her to read and sign it if he or she agrees to participate.

Before the user starts to perform his or her first task, a demo was given by the presenter. The presenter says he wants to find Wells Fargo ATMs in Chinatown, then goes ahead and identifies Chinatown on the cover page, then go to page 3 and circle Chinatown. Then the presenter taps the Wells Fargo ATM box and the computer shows the user the output message in the simulated cell phone display.

After the demo, the user is given each of the three tasks described in the “tasks” section above. During each task, the note taker observes where the user has trouble, what the user says, especially the thinking process. The observer also makes note of how long the user takes for each task, the questions that the presenter asked and the answers of the users, and any addition feedback the user provides.

Finally, we have the user fill out the usability survey to find out how much the interface matches the user’s needs. We also want to collect demographic information of the user to see how they fit in with the general target user group.

See the "appendix" for the script, consent forms, and the usability surveys.

Test Measures

For teat measures, first, we asked the users to complete the three typical tasks then fill out a usability survey. The usability study measured their satisfaction with the system. We used four variables to measure their satisfaction. The four variables are:

Ease of use: A good design must incorporate the users’ needs into their system. One of the biggest needs of users is easy of use. One characteristic of our target users is not being too tech savvy. If our system is too complicated, then the users will not use the system. Thus, with this variable, we can find out whether our system is too complicated or just right. If the users feel that the system is too complicated, then we will need to focus time on making the system simpler.

Accuracy: We are constantly improving the system and the accuracy the system. First of all, we want to test how accurate our system currently is. We constantly improved our system while we were testing it on the different users and this measurement is a good way of keeping track whether our progress can be seen or not. There are also external circumstances (the way the user holds the pen and thus, putting the camera in different angles) that also causes a lost of accuracy. However, if the accuracy rate is fairly good , then we know whether the system is mainly inaccurate due to external circumstances. However, if the accuracy rates are really poor, then we would know that the problem probably will come from our own code and thus, need improvement upon.

Appearance: Aesthetics may not be important to a designer’s point of view but it may be very important to a user. If the system does not appeal to the user aesthetically, they may not use it either.

Matches your needs: We want to measure how much our system matches what our target users want. This could be one of the most important items we measure. This will hopefully let us know whether we are giving the right information, giving enough information, giving useful information, etc. If we see that the user is not satisfied with our system, then we can ask which part they are most unsatisfied with and improve on that area. This should also provide an overall sense of how well our system is.

Secondly, we also took into account for how much time the user took in completing each task. First of all, if the system takes too long to retrieve the desired results from, then the user will not see the system as an efficient alternative to what they may normally use. Also, this is another way of measuring the easeness of the system. If the user takes too long to find the location, then we could assume that the map is not high resolution enough or that we should improve our grid system (A-I, 1-12).

Thirdly, we measured some background information for each user including age, major, gender, educational level, and level of familiarity with different types of maps (online and paper). These variables may or may not influence the variables we have selected and thus, we felt it's best to include them.

Results

After completing all results with the three participants, we charted the data we collected and placed the charts in the appendix section.

Variables:

Ease of Use:

Mean = 6.66667
Standard Deviation = .471

Subject H rated us a 6 for ease of use because she cannot find the streets after being shown the intersection of where the gas stations. She felt that the map was not clear enough to locate the streets. Also, she felt that it was “difficult to follow when location is on another/separate page”. At this point, because there were a few bugs in the system, the cross street was not included in the output and hence, it became more difficult for the user to find the intersections. Subject also tapped the wrong button for finding gas stations.

Subject Ti rated us a 7 for ease of use. Once again, the system did not return the cross streets and thus, made it somewhat difficult for the user to find the location returned. However, the subject had to use some background knowledge to help him locate where the streets returned were.

Subject To rated us a 7 for ease of use also. At this point, the bugs were fixed and the proper cross streets were displayed to the user which makes locating the difference intersections much easier. However, the subject still did not like having to flip back and forth between the different pages and commented about wanting one big map.

Accuracy:

Mean = 6
Standard Deviation = 1.41

Subject H rated us a 5 for accuracy. At this point, the project’s coordinate system was somewhat buggy as well. Therefore, when the subject circles one area as a destination, the subject will return directions for a really off area. Thus, the user became confused when she circled one area and was told to go somewhere 10 blocks away.

Subject Ti rated us a 5 for accuracy for the same reason. Though the user was able to find the final destination presented by the system (it was far away as well), the user took a really long time.

Subject to rated us an 8 for accuracy. By this point, we have fixed many of the coordinate system’s errors. Therefore, the accuracy rate grew because the blocks are now only 2 blocks off.

Appearance:

Mean = 7.6666667
Standard Deviation = .471

Subjects H and Ti both rated us an 8 for appearance while subject To rated us a 7. Parts of the numbering grids were hand written in.

Matches your needs:

Mean = 7.333333
Standard Deviation = .471

Subjects Ti and To ranked us with an 8 for matches your needs while subject H rated us with a 7. Discussion for why will be given in the discussion section.

Time Results

    Subject H(mins:secs)    Subject Ti(mins:secs)   Subject To(mins:secs)
Task 1            4:43                  4:45                         5
Task 2            5:56                  12:53                       9
Task 3            5:43                  6:52                         5

Task 1:

All subjects had a difficult time with this task when asked to locate the cross streets. However, each subject completed the task and found the necessary location in a decent amount of time (all in 5 minutes or less).

Task 2:

Subjects H and Ti had a more difficult time with this task than Subject To. Subject H and Ti had difficulty following the directions because the coordinate system was off. Also, Subject Ti could not find one of the streets returned and thus, took a really long time to complete the task. Subject To had a difficult time because he could not locate where “King” street. He had to spent a long time looking for the freeway exit before being able to proceed. After locating the exit, subject completed the task fairly easily.

Unfortunately for Subject H and Ti, the results returned were not very close to the destination.

Task 3:

Subjects were more comfortable with the system at this point. All subjects became better at locating the different intersections. Thus, all three subjects completed the task fairly easily (even though, for Subjects H and Ti, their destination was still off).

General Comments:

Two of the three subjects sprawled their papers out so they do not have to flip back and forth from the page they were working on to the front page. Two of three subjects never realized that there were instructions there.

Discussion

From the pilot run, we learned some positive things and usability bugs needed to be fixed about our system. Positives include a low learning curve, ease of use, and matching the users' needs. Meanwhile, usability bugs or suggested improvements include higher resolution maps, one big map instead of dissected regions, icons for each of the features in lieu of text, more help with finding which region is on which page, better directions in the returned results which ultimately led to a much longer period of time spent performing each task, and better accuracy with the directions returned.

Our users commented that SSS was easy to use because of the relatively few steps required to perform each task. This was encouraging because one of our main goals was to make SSS fast to request information with. Also, due to the relatively few steps required to perform the functions, it was also easy to learn. All of our users had little difficulty performing the tasks after the brief demo Tak performed. Users were also pleased that SSS matched their needs of querying for instructions.

For the "real" experiment, it would be difficult to implement or fix all of the usability bugs learned during the pilot run. However, we would definitely fix the most crucial ones such as higher resolution maps, icons representing each feature in addition to text labels, more help finding which region is on which page, and better accuracy with the directions returned. The reason why we choose the above to implement/fix is because despite having only two to three steps required to perform each one of the features we currently support, users are still taking longer than expected to actually perform them. The reason behind this seemed to be locating the streets returned by the request when the locations are on different pages. These changes should yield times more geared towards actually performing the task and decrease the time spent on flipping through pages to find the locations because they weren't very accurate.

Based on the results of the study, one of the changes we should make is to increase the resolution size of our dissected map regions. Two of the three users expressed interest in having a big and connected map much similar to commercial maps as opposed to our dissected version. In regards to this, we would really like to fulfill our users' requests but the absence of a big printer capable of producing high quality prints is a major issue and one that we are unlikely to solve. The alternative would be to further enhance the resolution size of our map regions so users will have an easier time finding the locations they desired. This should reduce the time users spend on locating the streets indicated by the returned results.

Another change we plan to make is adding icons representing each of the buttons. As one of our insightful user pointed out, users can associate the function of the buttons more readily comparing to our current text labels. For instance, we currently support ATM's from Bank of America, Wells Fargo, and Washington Mutual. That's a lot of similar buttons all of which are closely positioned on our paper interface. A consequence of this is that users might accidentally tap the wrong ATM button. By having icons of each bank alongside the text labels, users can immediately recognize the link between the button and the corresponding bank and would ultimately reduce the chances for error.

Another adjustment to our interface would be the addition of more graphical labeling to help our users find which page the regions are in. As mentioned above, it's difficult to offer users a big and connected map which means users will be forced to flip through pages to locate the streets in their respective regions. This results in significant time wasted on locating which region the streets are in, finding the corresponding page the region was printed on, and finally flipping to that page. In addition to the number of extra steps, there is also an increased memory burden on the user requiring them to remember which page their source location was circled on. As a solution, we proposed adding labels for the adjacent regions on every page. This eliminates the steps of flipping back to the front page to locate the street and the page the region is in.

Finally, we will be spending a majority of our time on improving the accuracy of our directions. As learned from the pilot run, the returned results were far off from what the users intended. Our users expressed that they would be able to get to their intended destination once in the general area as indicated by the returned results. However, there were cases when the returned result was referring to a location on another page thus caused our users a lot of pain in finding it. A higher accuracy would significantly reduce the time and headache our users experienced while attempting to locate the streets.

Workload breakdown

Kang Chen

Kang assisted in the implementation of MapSwingUI, MapPaperUI, and KMATMap. The modifications in Swing includes handling 13 different sheets, implemented the new features of Bank of America, Wells Fargo, and gas stations. Kang also helped implemented Swing to handle most of the errors the user would make with SSS. The modifications in KMATMap include handling the 13 different sheets with their different dot patterns. Kang helped refine MapPaperUI to handle the 13 different sheets as well. During the pilot study, Kang filled in the role of observer and timer. In addition, Kang helped update the implementation and interface between Swing and the database. For the wiki, he was responsible for the introduction and discussion sections.

Melissa Jiang

Melissa is responsible for implemention of the parsing the different websites and retrieving ATM locations, gas station locations, driving directions and transit information. Parsing required the use of regular expression. Assisted in creating, converting, and editing many of the maps including figuring out and resizing the maps to a size that would fit. Melissa was also responsible for using php to access the database and returning the desired locations and directions onto the cell phone.

For the wiki, Melissa handled the excel charts, the results section and the test measurements.

Andrew Tran

Andrew assisted and wrote MapSwingUI, MapPaperUI, and KMATMap. The modifications in Swing includes handling 13 different sheets, implemented the new features of Bank of America, Wells Fargo, and gas stations. Also implemented in Swing to handle mostly all errors the user would make with our system. The modifications in KMATMap include handling the 13 different sheets with their different dot patterns. Andrew refined MapPaperUI to handle the 13 different sheets as well. He was the computer in the interviews with subjects H and Ti. Andrew was also responsible for testing the system, handling any errors and debugging. He also measured the range of the coordinates x and y for each map, querying these numbers into the database. For this wiki, he was responsible for the implementation and improvements section.

Tak Wong

He is responsible for making further changes to the database side to support multiple sheets. This includes converting a raw data point from the paper to the data points in the database. He also supports generating the grid from the database given an intersection. There is also further refinement of the database queries to improve accuracy and eliminate more error cases from finding the closest intersection in the database. He also modified CS160PaperUI.java to support multiple pages, with some refinements by other group members.

For the interviews, he prepares the script, survey and test measures for each user. He is also the presenter during each interview. For the report, he is responsible for the methods and appendix section.

Appendices

The script

Script: The purpose of this interface is to conveniently find the closest ATM and gas station in a given region in San Francisco. We also support giving driving directions to a location by your desired mode of transportation.

Before we start, please read over the consent form and sign it if you agree to participate in our pilot study.

Demo: As a demonstration, I will give you a general sense of how to find a Wells Fargo ATM using the SSS. First, I look on the cover page to find approximately where I want to find the closest ATM locations. Then I go to the appropriate page number and circle the exact area of where I want to find the ATMs. Then I tap the appropriate box. The information will be returned to this cell phone interface. I can use this information to find the exact ATM locations on the map.

Your first task is to find gas stations in a region. Pick a region within San Francisco and use the SSS to find the closest gas stations. After that, find the gas stations on our map.

• Ask clarification questions, ask for feedback

Your second task is to find driving directions from one region to another. Pick a starting point and try to find directions to another region within San Francisco using the SSS. After that, try to follow the route on the map.

• Ask clarification questions, ask for feedback

Your final task is to find directions from one region to another by public transportation. Pick a starting point and try to find directions to another region within San Francisco using the SSS. After that, try to find the stops you should get on and off the bus on the map.

• Ask clarification questions, ask for feedback

Please fill out the survey to help us enhance our interface further. Thank you for participating in our pilot study.

Instructions attached to the interface

Instructions for using SSS:

To find an ATM in an area:

1. Circle the area where you would like to find an ATM

2. Tap the box next to the bank of your choice

3. Look for a text message on the simulated cell phone

4. Find the location of the ATMs on the map based on the text message

To find a GAS STATION in an area:

1. Circle the area where you would like to find the closest gas stations

2. Tap the box next to “Gas stations”

3. Look for a text message on the simulated cell phone

4. Find the location of the gas stations on the map based on the text message

To find directions to a location:

1. Circle the intersection of where you want to start your trip

2. If you draw start location by mistake, use the clear page button.

3. Circle the intersection of where you want to reach

(The following 2 sections are indented, but cannot be done on wiki) To travel by car:

4. Tap the box next to “By Private Vehicle”

5. Look for a text message on the simulated cell phone

6. Follow the driving directions on the text message

To travel by transit:

4. Tap the box next to “By Transit”

5. Look for a text message on the simulated cell phone

6. Follow the transit directions on the text message

The interface

Consent Forms

Subject H

Subject Ti

Subject To

Usability Surveys

Subject H

Subject Ti

Subject To

Notes from the Interview

Subject H

Find a gas station in a user defined region:

The subject is given the instructions

• user tapped the wrong function button
• The subject chose region 5
• Had trouble finding the place on the map
• The subject didn't realize there was instructions for the tasks she was trying to perform
• The subject had considerate trouble finding the return location on the map
• Don't see the place returned on the page listed as
• User couldn't find the 1st place returned
• User couldn't find the locations
• The street names are really tiny and blurry
• user wasn't happy about results returned weren't on the same page

Time: 4:43

Finding driving directions: Pine & Stockton -> 16th & Bryant

• User had to look back and forth between the cellphone and the map
• couldn't find some of the places returned by the results
• User had to look back and forth repeatedly
• the returned results was far from where she wanted to go
• had to use some background knowledge to find the places returned by the results
• user had never been to the area she chose to do
• user was really bothered by the fact the results were off
• user rather have a big connected map
  • some of the maps don't have labels

Time: 5:56

Finding directions by public transportation

Taylor & Market -> Lincoln and 26th

• direction on which direction to turn before heading to the first point returned by the results
• user starts to get the hang of using the maps
• user found the locations relatively faster now
• user didn't notice the grids appended by in the directions returned
• Couldn't find some of the locations returned by the result
• User happened to have been to the area before
• good thing they were big streets so it was a little easier to find

Time: 5:43

Post interview: Pro:

• Liked the nifty pen

Cons:

• Multiple pages are okay as long as the streets are clearly labeled
• Had to look at the big map if the results are on another page
• Would be more helpful if each page has the adjacent sections listed by page
• definitely need higher resolutions (1. clearer 2. bigger)
• drive some of the time
• Not sure if she would like the idea of having to always carry around a hard copy of the maps
• Like getting pictures in the msg
• Is okay with receiving multiple msgs on the cellphone
• Has unlimited text msg so it was okay to receive more msgs
• Thinks most people would have some sort of text messaging package when getting deciding to use this system.

Subject Ti

Task 1: Find a gas station in user defined region

Gas station in Silver(region 12):

• Can I search the street?
• User didn't like reading the directions
• User didn't have any trouble performing the task
• User liked the fact that the results included the price
  • User couldn't find the location because there was no cross street returned
• User relied on background knowledge to help him find the location returned in the results
• User was able to locate the second gas station returned
• Wanted cross street or approximate house number
• Reasonably close results were good

Time: 4:42

Task 2: Find driving directions from one place to another in SF

Silver, 3rd street -> Chinatown, Montgomery

• User had trouble finding the destination on the map
• User didn't know the cross street of where he wanted to go
• The map is kinda hard to read
• Labeled directions
• had trouble following the directions returned
• Had trouble when the directions require going to another page
• Couldn't find some locations
• Want a bigger connected map
• The directions weren't really clear
• There was no label for a street
• Used some background knowledge to find the locations in the returned results
• Confusing because the returned result was near the destination he requested
• It was not helpful that the returned destination was different from what he requested
• Can probably go to the requested destination from where the map left off.
• Question about the directions (speed) of the returned results.

Time: 12:53

Task #3: Find directions to a destination by =public transportation

Silver, 3rd -> Richmond (Geary and 3* something street)

• Has difficulty finding the destination
• Wondered where the desired page was
• Performing the task much faster now
• User had to redo the task multiple times before the system successfully recognized the inputs
• User had trouble finding the location on the map
• The street wasn't labeled on the map (google's fault)
• What was "H8"?
• User realizes that H8 realized that "H8" referred to one of the grids on the map
• User had trouble finding where to get on the bus
• User liked the fact that the results provided the time of the next bus
• User gets hang of using the grid system to help him locate the locations on the map
• User found the location significantly faster after using the grid system
• User successfully found the destination returned by the results but it was different from where he requested
• Wanted walking directions from where the bus dropped him off to the destination he requested
• User wasn't very familiar with the area
• Commented that the grid system was really helpful in finding the locations on the map
• Wanted more detailed maps: clearer street names
• Some pictures would be nice
• Map of the bus routes might be nice but could be confusing to read.

Time: 6:52

Other comments:

• Usually walks
• Would like to have the function buttons on every page
• Images would really help
• Wanted to actually write the names of the locations
• Wanted restaurants also
• Had trouble switching between pages
• Had trouble finding the streets if they are on another sheet

Subject To

Gas station Begin: 1:45pm Subject finds the region they want. They picked page 12. But gets lost after that. Subject asks: Do I circle closest place? Do I poke or circle? Decides to circle. Goes back and forth between the current page and the front page and hits the gas station.

Facilitator asks user to find the location of the gas stations. Subject had a really hard time looking for San Bruno. Flips map back an forth, moved it sideways, etc. Gas station returned H9, user tried looking for H9 but realized that it was not in the same page (page 12) as they are currently on. Had to flip through a lot of pages looking for H9 but gave up and use the tip top of H10 instead.

Facilitator asks user to find the second gas station. User realized that it was in H9 also so flips through all the pages again trying to look for the pages. Finally finds it and locates it.

Facilitator: What was difficult about finding the gas stations, how should we make it more easier? User: Bigger map will help. Had to search for H9 myself but the indexes are a good help.

End time: 1:50pm

Transit Begin: 1:51pm Facilitator asks user to find directions by driving. Where start? Decides to start at home on page 12 again. Tapped the transit button instead because user says that his hand was covering the transit word so did not realize. The results are returned. User cannot find the street once again. User commented that they cannot see the streets that well on the maps. Finally finds it and see their final destination on the map and comments that they will be able to find it.

User: The Map was hard to read because it is slanted and the streets are hard to see. Facilitator: Would you need to bring the map with you? User: Yea, that would probably be best. I would feel more secure

End: 1:56pm

Vehicle Begin 1:58pm User looking for some place. Starting at Visitacion Valley playground. User taps on private vehicle. Interface died. Ask the user to pick another place. User picks another place, City College by Balboa Park and picks Chinatown as destination. Direction shows up and then the user finds the entrance to the freeway. However, user had a really hard time finding the exit because there were no region guides in there. Had to refer back to the cover page a few times to follow all the maps containing the 280 freeway. Finally finds King exit then they were following the streets fairly okay. Arrived at Green Street and realized that they arrived.

This one doesn’t have the region guides so it was much harder. I had to manually look for King street. Also, does not say how many miles I had to go on the freeway also.

Facilitator: Did you notice that the direction is off? User: It gets me to the general area so it is okay. Facilitator: If you had the map then is it easier to find where to go? User: Yea, I can find it. End: 2:07pm

Other Comments:

They only show one route. I might want more routes. Would multiple text messages be confusing? Not really but the system should pick the best for me. Prefer minimum walking distance.

Facilitator: Why rank 8 for accuracy? It was like 5 blocks away so that’s still in general area.

User wants indention or indexing on the directions.

• • User had the page they are using as the top of their pile but also has the cover page on their ride side for easy access. User did not do that for task 1 but once they got more familiar with the system, then user begins having the cover page on the side.

Would different colors help differentiate the boxes? Nah, the words are good enough.

Trip planning, by Mode. Put an icon into the boxes itself. People usually recognize the logo and not the name.

What are the biggest error you think? No feedback on whether it was detected or not. Because even if I circled it, I do not know whether it has detected it or not.

Would prefer the big map instead of having something like this.

Subject never realized that there are instructions in the package

Results Charts

---