PilotStudy-Group:Group Made of Smiles-NirAckner

From Cs160-sp08

Contents 1 Introduction 2 Implementation and Improvements 3 Method 3.1 Participant 3.2 Apparatus and Test Location 3.3 Tasks 3.4 Procedure 4 Test Measures 5 Results 6 Discussion 6.1 Suggested Changes to Experiment 6.1.1 Changes to Test Environment/Equipment 6.1.2 Changes to Test Script 6.1.3 Return to Multiple Evaluators 6.1.4 Addition of More Test Subjects 6.1.5 Evaluation of New Tasks 6.2 Possible Changes to Interface 6.2.1 Selectability 6.2.2 Notes Saving 6.2.3 Note-Taking Efficency 6.2.4 Shared Notes and Reviews 6.2.5 Adding Logos to Reviews / Prices Pages 6.2.6 Changing Price Size 6.2.7 Continuous Scanning 7 Appendices 7.1 Raw Data

Introduction

When at discount retailers or grocery stores, customers often find it difficult to tell what the best deal is for a given product. Without access to a computer and internet, one must actually visit several stores (and potentially online retailers) to determine the best deals on items. It is hard to know whether the price on a bottle of sunscreen or a dress is the lowest available. Especially on impulse buys, it is either difficult or impossible to do this research beforehand.

Our mission is to allow consumers to make informed buying decisions by providing instant product information to their fingertips. Our mobile price comparison system addresses our users' needs by automatically comparing prices on items at nearby stores and websites. The user can input an item and instantly see reviews and prices on related and companion items at nearby stores, allowing the user to quickly gauge the best deals around.

The experiment we will be conducting is to determine the usability of our application's interface. Using a prototype that provides interactive functionality, we will gauge how easily our test users can navigate our application and use it to perform main tasks such as price comparison, finding companion products, and looking up price histories.

Implementation and Improvements

Based on feedback from our interactive prototype demo, we made the following changes and additions:

• Favorites
  • Favorites list displays price over timestamp
  • Implemented add/remove from favorites
  • "Remove from Favorites" after adding to favorites

• Notes
  • No error if no note in database

• Barcode Entry
  • Added a text field to enter in a UPC instead of scanning a barcode, linked into DB
  • Feedback if no results
  • Implemented automatically adding the product to search history on search

• Overview Page
  • Added pictures by URL, modified DB schema appropriately
  • Calculate overall rating from average of individual ratings

• Ratings
  • Added numerical rating per review (users wanted in lofi)

• Price History
  • Built this page (wasn't needed for lofi)

• Store Pages
  • Built this page, which shows store phone, address, website, and adds links appropriately

Method

Participant

The study participant was Georgia Crane, a 22 year old female currently studying Molecular Cell Biology at UC Berkeley. She is a new iPhone user, and says she "discriminates more on quality than on price". However, she is still a heavy comparison shopper, looking at a large number of products and prices before buying.

In order to ensure accurate results, the participant was selected to be a distant acquaintance who would be willing to provide honest feedback. In addition, tech-savvyness, and heavy use of a mobile device were important criteria. The other key part of the selection criteria was a focus on comparison shopping, which was evaluated by asking questions about shopping habits.

Apparatus and Test Location

Due to the constraints of the Android emulator, testing could not be done on a real mobile device. Instead, a laptop was used. The location of the test was a table in a local bookstore, so lighting and distractions were similar to some of the usage scenarios for the product (though not comparable to a supermarket, where the environment is far more busy). The evaluator was seated next to the user, to allow both demos of the device and logging of critical incidents without interrupting the user. The tester wore a watch, which allowed discrete time check when logging.

Tasks

• EASY: Compare prices: The user wants to see prices for a product at nearby and online stores and determine what store has the lowest prices. The user enters the product into the application either by taking a snapshot of the product barcode or typing in the product name. The application then shows a list of nearby and online stores, their distance, and their prices. The lowest price is highlighted on the list.

• MEDIUM: Check reviews: Prices are not everything, and the user wants to investigate a particular product further by reading reviews. After a product is entered into the application, it displays different reviews pulled from websites such as CNET or Amazon.com. This allows the user to quickly browse reviews and user testimonials in-store without needing to perform product research beforehand.

• HARD: Take notes on a product: After doing research or buying an item, the user wants to make a note on an item for future reference. The application allows the user to access a product and go to a note viewer/editor which stores notes per-item. At any time in the future, the user can go back to the product page on the application and view/edit the associated note.

In addition, our prototype provided the ability to find companion products, view the price history for the product, and mark items as favorites.

Procedure

1. Participant is welcomed, and the mission of the product and purpose of the testing is described.
2. Informed consent form, which participant has reviewed in advance, is collected.
3. A walk-through of the price-history feature is performed. The script for the walkthrough and the rest of the testing can be found in the appendix.
4. The user is asked to imagine themselves thinking about purchasing a new iPod. In sequence, the participant is asked to compare prices, check reviews, and make a note about the product. User is encouraged to talk through what they are thinking throughout the process.
5. Participant is prompted for any other feedback, thanked for time, reminded about policies documented in informed consent, etc.

Test Measures

The following quantitative measurements were observed:

• Number of key presses (Used to suggest how efficently users worked with the interface)
• Number of times user was confused and/or went off-track (Used to suggest how confusing the interface is to use, especially in direct comparison with the lofi interface)
• Time to complete task (Used to evaluate feasibility of using the product, what the initial time cost would be to users)

In addition to the quantitative measurements, we also observed how the users interacted with the system during the test and debriefing. More specifically:

• What tasks users had problems with
• How users were having trouble
  • What mistakes users were making
  • What misconceptions did they have about specific functionality such as menu options, buttons and displayed items on page
• What actions were intuitive and easy for the user
• Whether users performed a task in the way we envisioned
• Any side comments or suggestions that the user made
• Critical incidents

Results

	# Key Presses	# Times Confused/Off-track	Completion Time, with Explanation to Tester (seconds)
Task 1	10 (10 optimal)	0	45
Task 2	4 (3 optimal)	0	30
Task 3	8 + text (4 optimal)	2	120 (mostly for entering note text, letter by letter)
Mean, Standard Deviation	7.3, 3	0.66, 1.15	65, 48

• Task 1
  • Unclear how to get from main page to product page since UPC is already entered from demo
  • User remembers feature is in menu and assumes prices is there -- doesn't even look at overview page
  • Discover individual stores lead to store page ("Cool!")

• Task 2
  • Clicks back, doesn't pull up menu
  • Unclear as to whether items on main screen clickable
  • Pulls up menu, even though they could have done this on the previous screen
  • Not sure what reviews are sorted by
  • "Can I move or edit ratings?"

• Task 3
  • This time, remembers the "Edit Product Note" line, clicks on it
  • Wants ability to jump to the end of the text box
  • Seems to have trouble entering lots of text (partly emulator)
  • Still not sure if note is saved, assumes "written to disk somewhere by now"

• Overall
  • User generally seemed to like the interface
  • "I have no suggestions for improvement -- it just seems to work"
  • Likes look and feel

• Ideas:
  • Add sync to web-browser and desktop interface!
  • Popular user searches, global statistics on products
  • User product ratings

Discussion

Interactive prototype testing revealed a number of usability and testing issues, as well as interesting new ideas. Changes to correct these are discussed below.

Suggested Changes to Experiment

Changes to Test Environment/Equipment

Rather than test on a computer, it would be more ideal to have a physical device to test with in a real store, interacting directly with products, if this becomes feasible.

Changes to Test Script

All of the user tests we performed involved first introducing the user to our program, which requires doing the scanning / UPC entry for them, so we never tested this part of our interface. It would probably be much better to simply show the user Android semantics by having them play with other applications for a little while, and then let them loose on our application.

Return to Multiple Evaluators

It would be good to go back to the multiple tester model, to allow more than one set of observations -- instructing and taking notes was very difficult to do well at the same time.

Addition of More Test Subjects

Obviously, testing on more than 3 users would allow more generalized results, since if 2 of 4 testers find some usability issue, it is still not necessarily statistically significant. This is especially important for things like our save functionality, where there are direct tradeoffs between feedback for new users and efficiency for experienced users.

Evaluation of New Tasks

Testing on the representative tasks was a good choice for evaluating interface changes and usability, but they do not test all of the interface, since features like the search history are in a completely different part of the UI. Adding new tasks would help to test these parts of our design.

Possible Changes to Interface

Selectability

Some users were still unclear what was selectable, even in the interactive prototype. Part of this likely has to do with familiarity with Android, which could be changed in future testing, but perhaps adding more visual cues like making text into buttons rather than selectable list elements would help if this remains an issue.

Notes Saving

Even though time and number of errors when saving notes went down from our lofi testing, indicating some improvement, users were still unclear as to when their note was saved. Perhaps providing text that says "Saving..." and then "Saved." or an animation of a disk is necessary. Even though the device saves instantaneously, users are still used to the paradigm of software like MS Word, which animates the saving process.

Note-Taking Efficency

When typing out a note, the user wondered whether there was a button that would jump to the end of the textbox. Perhaps adding hotkeys of this nature for expert users would be a logical next step. Also, providing predictive texting like T9 would help to make taking notes more feasible.

Shared Notes and Reviews

One suggestion that came up from users and evaluators in class was sharing notes and reviews. This could be a great concept, but further experimentation is needed to decide whether this would be useful or a case of feature creep.

Adding Logos to Reviews / Prices Pages

Another potential change is to add merchant / reviewer logos to the reviews and prices pages, so that users can visually see who is selling the product, rather than having to read the text. There probably isn't enough room for this, however.

Changing Price Size

Since users mostly seemed to be looking at relative prices, one change could be to size the different entries on the prices page so that the cheapest stores have the largest entries, and size are inversely proportional to price. Again, this likely would make the interface too complicated.

Continuous Scanning

Another suggestion that seems like it would be useful is constant scanning, rather than forcing the user to "take a picture". Thus, if a barcode is recognized by the software at any time, the product page will be opened. This definitely would make more sense on a faster, future phone, but is not feasible with today's technology.

Appendices

• Informed Consent Form
• Test Script

Raw Data

Task 1
Unclear how to get from main page to product page since UPC is already entered from demo
User remembers feature is in menu and assumes prices is there -- doesn't even look at overview page
Discover individual stores lead to store page ("Cool!")

45 seconds



Task 2
Clicks back, doesn't pull up menu
Unclear as to whether items on main screen clickable
Pulls up menu, even though they could have done this on the previous screen
Not sure what reviews are sorted by 
"Can I move or edit ratings?"

30 seconds



Task 3
This time, remembers the "Edit Product Note" line, clicks on it
Wants ability to jump to the end of the text box
Seems to have trouble entering lots of text (partly emulator)
Still not sure if note is saved, assumes "written to disk somewhere by now"

120 seconds, mostly typing note 



Overall
User generally seemed to like the interface
"I have no suggestions for improvement -- it just seems to work"
Likes look and feel

Ideas:
Add sync to web-browser and desktop interface!
Popular user searches, global statistics on products
User product ratings

---