PilotStudy-Group:Group3-MichaelSo
From Cs160-sp08
Contents |
Introduction
Our system uses the camera functionality of the mobile phone to take a picture of foreign words/text and then translates it into the user’s native language. The purpose for this application is for people who enter an area, whose language is foreign to them, to be able to interact with the surroundings comfortably. The process of translating a sign starts with the camera. The user takes a picture, manages aspects of the translation such as cropping an area of the picture or selecting the language to translate to, and obtains a translation with one push of a button.
The experiment I am conducting serves to help me discover the flaws in our design, as well as the components the user liked. I will to discover what elements were confusing for the user and how those elements could be made less confusing. The ultimate goal is to improve the usability of the application for the users.
Implementation and Improvements
- In our Camera mode, we are using a static image because it would be very difficult to actually simulate the process of taking an image on the emulator, which is a non-mobile device. Thus, the application starts with a single preloaded image, and pressing the center button to "take" the picture merely transitions the application to Manage mode; no picture is actually taken.
- In Manage mode, the user has access to the cropping function and the ability to change the “From:” and “To:” languages. The “From:” and “To:” languages displayed in Manage Mode determine what language to translate from and what language to translate to. The user can hit the center button on the D-pad to transition to Translation mode where the translation for the picture appears.
- In translation mode, there will be the picture displayed as well as translations for the picture. Translating the image could result in multiple translations, so the user is able to scroll in a list view to read each translation. In this mode, the user is also able to save the translation to Favorites.
- No actual translation goes on when the user presses the center (translate) button in Manage mode; we just hard-coded the translations. However, the improvement made since submitting the Interactive Prototype is that most of the signs have a translation instead of only one sign. Therefore the user has the ability to crop any of the signs and will likely get a translation for it.
- The From/To translation language selectors have been implemented. In Manage Mode, the user can access the language selector ability by selecting the “Select Language” option in the menu. The changes in “From:” and “To:” languages will be shown in Manage Mode.
- We provided a dialog box that informs the user that an image has been saved to Favorites when they select the option to save a particular translation. An improvement since submission of the Interactive prototype is that we implemented the actual picture-saving mechanism. So whatever picture the user cropped and got a translation for will actually be saved to Favorites.
- In Crop Mode, we actually implemented the cropping mechanism and the actual passing of the picture to Manage Mode and other modes. So the user can draw a box around whatever he or she wants and the result will show up in Manage Mode. To crop the part of the picture desired, the user’s first click of the center button sets the top boundary of the cropping box and the second click sets the bottom boundary of the cropping box. Therefore if you crop the image by having your first click setting the bottom side of the box and the second click setting the top side of the box, the picture in Manage Mode will show nothing cropped.
- Another improvement made since submitting the Interactive Prototype is that we removed the Options menu screen. We felt it unnecessary; the menu activated by the menu button is enough to display the options. So in Camera Mode, the menu displays the “Favorites” and “Manage Dictionaries” options. Selecting the “Favorites” will obviously take the user to the Favorites screen. In Manage Mode, the menu adds two more options: “Crop” and “Select Language”. And finally in Translation Mode, the menu replaces the “Crop” option with the “Save” option and removes the “Select Language” option.
Method
Participant:
Bob is a 20 year-old male in his 3rd year as a Chemical Biology Major at UC Berkeley. He likes TV, movies, and Apple products. He dislikes Microsoft products. Bob did not participate in the lo-fidelity prototype testing, so he has no experience with our mobile application. He was chosen because he fits our target user profile; he is someone who wants to be able to communicate in any foreign language area and who wants to learn foreign languages.
Apparatus:
I used one of the computers at the Windows lab located in 330 Soda Hall. I ran the Android emulator on that computer and our mobile application started up. The user sat in the seat in front of the computer to accomplish the series of tasks while I and another observer sat on both sides of the user taking down notes using our pens and notebooks.
Tasks:
Easy – Selecting the language to translate from (Task 1)
The user is able to change the language to translate from and the language to translate to. To be able to do this, the user would have to go through the normal process of taking a picture and enter Manage Mode. In Manage Mode, there is an option in the menus called “Select Language”. Selecting that option will bring the user to a screen where the selecting languages task is done. When the task was performed, I looked for how long it took the user to realize that in order to select languages, the user needs to get to Manage Mode. When the user got to the Select Language screen, I looked for how fast the user selected the correct languages and applied the changes.
Moderate - Save result for future reference (Task 3)
The moderate task is to save a particular image into the favorites list. We provided the capability to be able to store images with their corresponding translations into the phone for when you want to look at them again at a later time. To do this, when on the screen with the image and its translations, the user would click the menu button and select the save option. Then the application would acknowledge the user that it has been saved to the favorites list. To bring up the favorites list, click the menu button at any time and select the Favorites option. When the task was performed, I looked for how long it took for the user to figure out how to save the image and how to access the favorites list.
Hard - Selecting region of picture to translate (Task 2)
The user is able to crop the picture taken and to translate whatever the user cropped. To perform this task, the user would take the picture and in the next screen (which is Manage Mode), go to the menu options and select the "crop" option. A cursor would then appear on the screen and user can move the cursor with the arrow buttons. When the middle button is selected the cropping begins, and when pressed again completes the cropped area. The user can then press the translate button to translate just the cropped area. When the user entered Crop Mode, I looked to see how long it took the user to notice the cropping cursor. I also looked for how long it took the user to figure out how to crop the picture.
Procedure:
There were three people involved: myself, the user, and an observer. I started off by giving the user the consent form and a pen. The user was told to read the consent form and told that in order to participate in the usability study, he must sign the form. After the user signed the form, I booted up the Android emulator. While we waited, I told the user that once the Android emulator finishes booting up, I would do a demo to demonstrate the main features of our application. When our application was finally up and running, I started the demo. I introduced Camera Mode, telling him this is the first screen of the application. I also informed him that it was not a working camera; it was just a static image. I then showed how to move to Manage Mode. I informed him of what is seen in Manage Mode such as the picture that was taken from Camera Mode and the "From:" and "To:" language indicating what language the translation will be in and what language is being translated. Then I showed him how to get to Translate Mode. In Translate Mode, I emphasized that there could be multiple translations. I scrolled through them for him to see. After that, I told him it was his turn to try out the application. I restarted our Android application and got the user to sit in front of the computer. I placed myself to his left and the observer sat to his right. Both I and the observer had our notepads and pen out ready to take down notes. The observer also had his timer out to time how long it took the user to accomplish each task. Once our application was up and running, I informed the user of his first task, Task 1. Once I finished describing the task, I told the user to begin and the observer started the timer. When the user finished the task, the observer would stop the timer and record the time. While the observer recorded the time, I would assign the user the next task. The observer would then reset the timer and when I told the user to begin, the timer would start. During each task, I and the observer took down notes recording down critical incidents and steps the user took in completing the tasks. When the three tasks were completed, I asked the user questions for feedback and explanations of the actions he took in accomplishing tasks. When I was done asking questions and the user was done giving feedback and answers, I concluded the Pilot Usability Testing by thanking him for his participation.
Test Measures
Number of times user makes mistake
I recorded the quantity of mistakes made during the whole test. The number of errors a user makes shows how intuitive and easy to use the interface is. A lot of errors, such as 10 errors, suggests the interface is confusing to use and offers no or very little information in helping the user figure out how to accomplish the series of tasks. On the other hand, a small amount of errors, such as 2-3 errors, suggests the interface is simple to use and easy to learn.
Number of questions user asks
I recorded the quantity of questions the user asked during the whole test. The number of questions asked by the user reveals what is unclear about the interface. A lot of questions, such as 10 questions, would mean the user does not understand the interface at all. The user who asks a lot of questions is unable to figure out how to navigate through and interact with the interface. A couple or so questions would mean the user has a good understanding of the interface and is able to figure out how to accomplish the series of tasks using our interface.
Time it takes to complete task I recorded the times it took for the user to accomplish each tasks. The times reveal how long a user actually takes to accomplish one of our tasks. A relatively long time suggests that the steps needed to accomplish the task either need to be revised, or be made clearer. A relatively short time suggests that either the user found a new and shorter way to accomplish the task, or that the steps needed to accomplish the task is intuitive and simple to follow.
Results
Number of mistakes: 5
Task 1: 2
Task 2: 2
Task 3: 1
Number of questions: 1
Task 1: 1
Task 2: 0
Task 3: 0
Times to complete tasks
Task 1: 1 min 41 sec
Task 2: 1 min 14 sec
Task 3: 0 min 22 sec
Task 1
The first error occurred when the user hit the home button. He got himself out of the application and he was confused on how to get back. He asked, "How do I get back? I don't know what to do." That was when I decided to help the user get back to our mobile application. After a minute, the user got himself to Manage Mode. The second error occurred when the user, in Manage Mode, hit all the directional buttons on the D-pad. This was done after accessing the Menu in Camera Mode. Apparently, the user thought some other menu would pop up from hitting a directional button.
Task 2
The first error occurred when the user hit the back button, which brought himself back to Camera Mode. Apparently, he did not catch the "Crop" option from the menu in Manage Mode. The second error occurred in Crop Mode. The user did not see the cropping cursor and decided to hit the center button on the D-pad multiple times. This brought him back to Manage Mode with nothing cropped in the image. He went back to Crop Mode and finally noticed the cropping cursor.
Task 3
The first and only error occurred when the user was trying to get rid of the alert dialog box with the text: "Saved to Favorites". The user pressed the center button the D-pad, the Menu button, and the directional buttons on the key pad. Finally, the user pressed the back button and the alert dialog box went away.
Discussion
- The icon of the center button of the D-pad in Manage Mode should be removed. Its appearance seems to confuse the user because the icon looks like it can be pressed. I think the "Translate" text should remain above the D-pad. It can be questioned that if the "Translate" text is above the D-pad, will the user hit the center button, or the up-direction button on the D-pad? However, I think that most users will intuitively hit the center button though, so I will keep the center button as being the button to translate the image.
- The user took a while to see the cropping cursor, so the visibility of the cursor needs to be improved. One change would be to place the cursor about the center of the image instead of top left corner of the image. A second change to improve the visibility would be to make the cursor blink so as to attract the user's attention. The picture itself is static, so a blinking cursor would be sure to be noticeable.
- All modes that have a menu need something to indicate that there is a menu available. The user initially was hitting all the buttons on the D-pad instead of going for the menu button. Having a "Menu" text appear above the menu button will most likely indicate to the user that hitting the menu button will bring up a menu and also indicate that a menu is available in the mode he or she is in.
- The changing the "From:" and "To:" languages should be done in the Manage Mode screen instead of having a "Select Language" screen specifically for that purpose. In Manage Mode, the "From:" and "To:" text seems to indicate to the user that he or she can directly change the languages on the Manage Mode screen. Therefore, the selecting language function should be implemented in Manage Mode instead of a separate screen. This would make selecting languages a faster process because it eliminates intermediate steps, namely accessing the menu and selecting an option.
- The alert dialog box stating "Saved to Favorites" should either go away on its own, or go away by pressing any button. Apparently, hitting the back button is not a user's initial reaction to get rid of an alert dialog.
- In the case a user crops the image in a way that our application is unable to translate, the text "Please try cropping again" shows up in Translate Mode. In this special case, the user should be unable to select the "Save" option. As of this test, the user was able to save this message into Favorites.
- When assigning Task 1, the user should be told to take a picture first. It appears that having the user start in Camera Mode, the user assumes the "From:" and "To:" languages can be changed in Camera mode. This explains why it took the user a relatively long time to accomplish this easy task.
Appendices
Demo
1. Change language default to Korean, do demo of translating entire picture, then go back to code, change default to Chinese.
2. This is where our application starts, in camera mode. As you see, there is no real camera implemented at this point in our prototyping. Just imagine, however, that you are aiming your camera-phone at this wall of Korean signs that you want to translate. To take the picture, hit the center button. Now, this is manage mode. To translate these signs to English, hit the center button. Oh, look, there are two possible translations!
Instructions for tasks
1. The default "Translate from" language is Chinese. However, the signs are in Korean, so you must change the "Translate from" language to Korean.
2. We want you to pick any one sign to translate, by cropping. You're just interested in what one of these signs has to say.
3. Now, we want you to save the image [if there is more than one translation, pick the bottom one to save] to your Favorites. You believe this translation will be very useful for future reference. Now, pretend that two hours have passed, and you need the translation again: retrieve the saved translation and picture on Favorites.
Raw Data (Notes)
Task 1 Observations:
- Bob hit Menu button in Camera Mode.
- Bob hit Home button and got stuck. He did not know how to get back to our application. He asked, "How do I get back to the application? I don't know what to do." I therefore helped him get back to our application placing him back at Camera Mode.
- Bob hit the center button on the D-pad and got himself in Manage Mode. In Manage Mode, Bob clicked on the directional buttons on D-pad expecting something to happen. However, hitting those buttons causes nothing to happen. Eventually, Bob hit the Menu button and selected the "Select Language" option.
- Bob had no problem selecting the languages in the "Select Language" screen.
Task 2 Observations:
- Bob went back to Camera Mode.
- Bob got back to Manage Mode and found the "Crop" option in the menu.
- Bob did not see the cropping cursor on the screen. He clicked the center button on the D-pad too many times and got himself in Manage Mode with nothing cropped in the picture. I told him to crop again because he did not crop anything.
- The second time in Crop mode, he noticed the cropping cursor in the top left corner. He hit the appropriate buttons and successfully cropped a sign in the picture.
Task 3 Observations:
- In Manage Mode, Bob hit the center button on the D-pad and got himself in Translation Mode. Bob mentioned that icon of the center button looked like an button to push on the screen, but because of the demo, he figured out the icon was meant to tell the user to hit the center button on the D-pad.
- He hit the Menu button and clicked on the "Save" option. The alert dialog popped up. Bob hit the Menu button, the center button on the D-pad, and the directional buttons on the D-pad all in an attempt to get rid of the alert dialog pop-up. Bob eventually hit the back button and the alert dialog went away.
- Bob hit the Menu button and clicked on the "Favorites" option. In the "Favorites" option, he saw his saved translation and clicked on it.
After the tasks:
- Bob tried cropping a different sign. When he hit translate, he got the message "Please try cropping again" in the Translate Mode screen. He ignored the message and selected the "Save" option. Bob then checked the "Favorites" screen and saw that the message was saved there.
- Bob went back to cropping and cropped this time starting from the bottom and going up. He discovered the flaw in our program that our cropping only works if the user starts cropping from the top to the bottom. Refer to the Implementation and Improvements section for more detail about our cropping mechanism.
Consent Form
INFORMED CONSENT FORM
You are invited to participate in a study of user interface design. We hope to learn how usable and effective our prototype is. You were selected as a possible participant in this study because you might be a potential user of our application
If you decide to participate, we will have you complete a series of tasks that the application will support, using our paper prototype. The entire procedure should take less than 45 minutes. Physical discomforts may include paper-cuts, because you will be shifting around index cards. Emotional discomfort may include slight psychological uneasiness from the pictures on the index cards. There are no direct benefits, but you may learn something new from this experiment, and if our product is ever commercialized, you will have known that you have helped us.
Any information that is obtained in connection with this study and that can be identified with you
will remain confidential and will be disclosed only with your permission. Some anonymous information will be submitted to the professor, as this is a class project. Although we will try our best to keep your personal information confidential, we cannot make any guarantees, and in the event that your personal information is leaked, we will try to notify you as soon as possible.
Your decision whether or not to participate will not prejudice your future relation with the University of California at Berkeley. If you decide to participate, you are free to discontinue participation at any time without prejudice.
If you have any questions, please do not hesitate to contact us. If you have any additional questions later, please contact Michael So at michaelso@berkeley.edu who will be happy to answer them.
You will be offered a copy of this form to keep.
_______________________________________________________________________
You are making a decision whether or not to participate. Your signature indicates that you have read the information provided above and have decided to participate. You may withdraw at any time without penalty or loss of benefits to which you may be entitled after signing this form should you choose to discontinue participation in this study.
_____________________________________
Signature
__________________________
Date
