PilotStudy-Group:Group3-HenrySu

From Cs160-sp08

Henry Su (Group 3)

CS 160 Section 101

Pilot Usability Study

1 Introduction
- 1.1 System overview
- 1.2 Purpose and Rationale for Experiment
2 Implementation and Improvements
3 Method
4 Test Measures
5 Results
6 Discussion
7 Appendix
- 7.1 Materials
- 7.2 Merged Critical Incident Logs

Introduction

System overview

Our application is a mobile translator. Basically, it starts in Camera mode, where the user can take a picture of some foreign text (be it a sign, a restaurant menu item, etc). With a click of a button, the application will progress to Manage mode, where the user can verify the to and from language of translation, and crop the newly-taken picture, if the user wants to specify a region to translate. Once this is done, the user presses the center button, and our application displays the translation into the language of choice.

Purpose and Rationale for Experiment

The purpose of this experiment is to gather some statistics that will reveal whether our interactive prototype design is effective, especially for completing the three tasks. From the study, we can come up with ideas on what to change for the next design cycle.

Implementation and Improvements

We have made many changes to our interactive prototype since the previous submission. Based on feedback from the interactive prototype presentation, we decided to remove the Options screen entirely, and put in all necessary features in the Menu. As a result, the Menus for each mode have changed. All modes now have a direct link to Favorites, as well as Manage Dictionary (a not-yet-implemented feature that will let the user download a translation dictionary for a particular language). We felt that these two activities are universal in the sense that a user may want access to them while at any mode. Also, in Manage mode, the Menu now has "Select Language" instead of "From" and "To". This was to make it more intuitive and concise. In addition, we implemented the Select Language screen for this iteration. The user can now select a language, and the changes are reflected on the bottom bar in Manage mode. When the user presses the translate button, there is now an error dialog if the wrong "From" language is selected (right now, if anything other than Korean is selected). In Crop mode, we changed the cursor to make it more visible and visually appealing. The color even changes once the user presses the center button. Another important improvement is that the correct region is now highlighted once the user finishes cropping and re-enters Manage mode. Furthermore, when the user presses the Translate button, the actual region selected is translated. There is a catch, however. Because the translations are hard-coded for each sign, the algorithm will "add" a sign's translation to the final translation if and only if the center of that sign is within the cropping rectangle. That is, even if you only select the center part of one sign, the translation for that sign will be given. For the most part, though, the translation respects user action much more than in the interactive prototype presentation. As implied, selecting multiple signs will cause a run-on, where each sign's translation will tacked on to the final translation (unfortunately, in no particular order right now). For the sign that has multiple translations, two copies of the final translation are made, which are identical except with each containing a different translation for that sign. In Translate mode, we got rid of the "From" and "To" in the Menu, since this feature is only useful in Manage mode (it was also taken out in Camera mode for the same reasons). There is also no more "Send" button, since we are postponing indefinitely the implementation of that feature. Also, in Translate mode, pressing "Save" now actually saves the translation to Favorites. Similarly, clicking on the desired translation in Favorites will now lead to a screen containing the picture (including the selected region, if any), with the translation below (if there are multiple translations, only the saved one is displayed).

Method

Participant

The participant was a 26 year old male EECS undergrad at UC Berkeley. He had previously evaluated our low-fidelity prototype, but since that was a while ago, and we had made many changes to the interface since then, I thought he would still be a good fit.

Apparatus

The pilot usability study was conducted at the living room of the participant's house. I used my laptop to run the android emulator and our application. Michael So was my observer. During the tasks, the participant sat in the middle, I sat on the left, and Michael sat on the right. I used the stopwatch application on my cellphone to time each task. No recording was done, as both Michael and I took notes.

Tasks

Easy - Change the "From" language of translation

The "From" language of translation is set to Chinese by default, since it is the first one in alphabetical order. The user must change this to Korean for the purposes of translating the given picture, which is in Korean. To do this, first take the picture, entering Manage mode. From here, press Menu->Select Language. This will lead the the Select Language screen, where the user hits the right arrow 2 times to move the spinner to Korean. Press down twice, and then press the center button, to apply the changes. We especially looked for whether the user was able to figure out that the language could only be changed in Manage mode.

Moderate - Save result for future reference

The moderate task is to save a particular image into the favorites list. We provided the capability to be able to store images with their corresponding translations into the phone for when you want to look at them again at a later time. This may happen when you want to look for something, but you don't remember what the sign looks like. You can look for the image from your list based on the translation and look for the signs that look like the sign in the image. To do this, when on the screen with the image and its translations, the user would click the menu button and select the save option. Then the application would acknowledge the user that it has been saved to the favorites list. To bring up the favorites list, click the menu button at any time and select Favorites. We especially looked for whether the user could figure out what to do after the Saved dialog appears.

Hard - Selecting region of picture to translate

The hard task is to take a picture of a sign and then crop out unnecessary objects that may interfere with the text recognition program. This would be useful in situations where there are a lot of extra things in the background such as trees and buildings, or when you want to translate only a portion of the screen (as in our pilot test case). To perform this task, the user would take the picture and in the next screen before pressing the translate button, go to the menu options and select the "crop" option. A cursor would then appear on the screen and the user can move the cursor with the arrow buttons. When the middle button is selected the cropping begins, and the user can use the arrows to make the cropping rectangle. The center button is pressed again to complete the cropping. The user can then press the translate button to translate just the cropped area. In this task, we especially looked for whether the user knows what to do once in the Crop screen (this is especially tricky without a touchscreen).

Procedure

Michael and I entered the participant's house, and were led into the living room. There, we chatted with the participant while setting up the laptop and stopwatch. We had the participant read and sign our consent form. Thereafter, I demoed how one would take a picture and translate the entire screen. The details are in the Appendix. After the demo, I asked the participant if he had any questions. After that, I read off the description for the first task; right after the description was read, I started the stopwatch. While the participant attempted the first task, we made notes regarding what questions the participant asked, and what mistakes were made. The number of questions and mistakes were tallied up. The finishing time was also noted. The other two tasks were conducted in a similar fashion. After the tasks were completed, I asked for comments from the participant. This was particularly useful because the participant revealed what he was thinking as he tried to complete each task. After the participant commented, I asked for further suggestions for improvement. In the end, I thanked the participant for his approximately 50 minutes worth of time and effort.

Test Measures

Three things were measured for this experiment. First, the amount of time required for the user to complete each task was noted. As mentioned, the timing started right after the description of the task was read. We felt this was a good indicator because a short time likely indicated that the task was doable and the interface was usable for first-time users. We also measured the number of questions the user asked while performing the tasks, as well as the number of mistakes the user makes during the tasks. These provide a good measure of how intuitive the interface is--the more questions and the more mistakes, the less intuitive. That was our assumption.

Results

Total number of mistakes: 3

Total number of questions: 3

Task 1 (easy): Change the "From" language of translation to Korean

Time: 52.93 seconds

The user initially pressed the Menu button in Camera mode, as he did not know that he had to take a picture before being able to change the language. After that mistake, he correctly pressed the center button to enter Manage mode, and soon after, completed the task.

Task 2 (hard): Selecting a region (occupying one sign) to translate

Time: 46 seconds

On his first attempt, the user clicked the center button too many times, and was taken out of Crop mode. After this mistake, he got back to Crop mode and cropped the picture correctly.

Task 3 (moderate): Save result for future reference

Time: 47 seconds

The user had no problem saving the translation. However, he did not know how to exit the dialog box that indicated the translation was saved. After clicking the center button to try to get out, he pressed the Home button. After some initial confusion, he got back to the application, and finally pressed the Back button to get rid of the dialog. Retrieving the translation from the Favorites screen proved to be easy.

Discussion

I found out that our instructions for the first task (changing the "From" language) were probably unclear. The participant simply didn't know that changing the "From" language required taking a picture first (as the Select Language option was only available in Manage mode). After the experiment, the user said that it made sense that the language can be changed only in Manage mode, but our instructions should make that clearer. An alternative would be to start the user in Manage mode for that task. That would be a change we'd make for the "real" experiment.

There are also several changes that we may make to our prototype, based on the results of the user tests, and also on the user feedback at the end of the session. The element that most hampered the completion of the tasks was the getting rid of the Saved dialog box. The user said that it is very unintuitive and non-standard for the user to have to manually press the Back button to get rid of the alert. Instead, we could either make the alert disappear on its own, or make it disappear with a push of the center button (a more standard method). The user also thought that leaving the screen in translation mode (after the Save alert is gone) felt very unnatural, especially since there was no way to return to the home screen (Camera mode) without pressing the Back button multiple times. The user, after all, had viewed and saved the translation already. What we will probably end up doing is to have a Menu option to return to Camera mode, where the user could proceed to take another picture to translate. Another possibility is that after displaying the Saved alert, the application could return itself to Camera mode. If the user doesn't save the picture, he would still be able to press Menu to select the button to go back to Camera mode. The issue about the Cropping being non-intuitive the first time around was harder to address, however. Without a touchscreen, it is hard to improve the cropping interface short of having instructions telling the user what to do (probably a bad idea).

Besides that, there were some minor suggestions that the participant made while he played around with the application, after completing the three tasks. First, he thought that having the user select a "From" language may be unnecessary trouble. In fact, the user may not even know what language it is he wants to translate from. Instead, since we will (hopefully) be using a real translator eventually, it should be fairly easy to detect the incoming language by simply reading the first couple words. This way, the language could be auto-suggested to the user, and if the user knows it's wrong, he can still change it. Another suggestion is that in the Favorites list, we make it able to loop back to the top item after pressing Down, upon reaching the bottom item. This loopback feature is fairly standard across most phones, as having to press Up repeatedly after reaching the end of the list can be tedious. This should not be hard to implement, so we will almost certainly add this functionality for the final prototype. The user also commented at how there isn't a Delete feature in the Favorites list, since the user may want to delete entries that are either saved by accident or entries that become unnecessary. Specifically, we could make use of the "Delete (X)" button on the phone, to provide a Delete functionality in Favorites. There were other suggestions, that we most likely will not implement in the near future, but would serve as a useful guide in a far-future release of the product. These are in the Appendix.

Appendix

Materials

Demo Script

Note to self: Change language default to Korean, do demo of translating entire picture, then go back to code, change default to Chinese.

Say to user: "This is where our application starts, in camera mode. As you see, there is no real camera implemented at this point in our prototyping. Just imagine, however, that you are aiming your camera-phone at this wall of Korean signs that you want to translate. To take the picture, hit the center button. Now, this is manage mode. To translate these signs to English, hit the center button. Oh, look, there are two possible translations!"

Instructions for tasks (spoken to user)

1. The default "Translate from" language is Chinese. However, the signs are in Korean, so you must change the "Translate from" language to Korean.

2. We want you to pick any one sign to translate, by cropping. You're just interested in what one of these signs has to say.

3. Now, we want you to save the image [if there is more than one translation, pick the bottom one to save] to your Favorites. You believe this translation will be very useful for future reference. Now, pretend that two hours have passed, and you need the translation again: retrieve the saved translation and picture from Favorites.

Consent Form

INFORMED CONSENT FORM

You are invited to participate in a study of user interface design. We hope to learn how usable and effective our prototype is. You were selected as a possible participant in this study because you might be a potential user of our application

If you decide to participate, we will have you complete a series of tasks that the application will support, using our interactive prototype. The entire procedure should take less than 45 minutes. Physical discomforts may include eye strain because you will be staring at a computer screen. Emotional discomfort may include slight frustration from trying to complete the series of tasks. There are no direct benefits, but you may learn something new from this experiment, and if our product is ever commercialized, you will have known that you have helped us.

Any information that is obtained in connection with this study and that can be identified with you will remain confidential and will be disclosed only with your permission. Some anonymous information will be submitted to the professor, as this is a class project. Although we will try our best to keep your personal information confidential, we cannot make any guarantees, and in the event that your personal information is leaked, we will try to notify you as soon as possible.

Your decision whether or not to participate will not prejudice your future relation with the University of California at Berkeley. If you decide to participate, you are free to discontinue participation at any time without prejudice.

If you have any questions, please do not hesitate to contact us. If you have any additional questions later, please contact Henry Su at henrysu@berkeley.edu who will be happy to answer them.

You will be offered a copy of this form to keep.

_______________________________________________________________________

You are making a decision whether or not to participate. Your signature indicates that you have read the information provided above and have decided to participate. You may withdraw at any time without penalty or loss of benefits to which you may be entitled after signing this form should you choose to discontinue participation in this study.

_____________________________________ __________________________

Signature Date

Merged Critical Incident Logs

Questions asked during Demo

how to activate Camera mode?

Task 1

52.93 sec

mistake in trying to change language from Camera mode (hit menu button there)
- didn't know had to take a picture before can change language

Task 2

46 sec

mistake in first attempt: accidentally went out of Crop mode by clicking center button too many times

Task 3

47 sec

failed to get pop-up message away (user thinks: "Saved to Favorites: how to get out?")
hit center button, waited, and pressed Home (eventually found out that he needed to hit Back)
- not obvious to hit Back to get rid of alert dialog