PilotStudy-Group:GeoPhoto
From CS160 User Interfaces Fa06
Contents |
Introduction (5 points)
Introduction to the System
Travellers wish to share their travels with their online friends. Our system streamlines the creation of an online map of your digital photos. The user prints out a paper map of their location. As they take photos, they circle their location on the map (and optionally mark tags that they wish to apply to that photo). When they get home, they use our application to correlate penstrokes and photos. Our application then uploads their photos to Flickr with tags and geolocation information, so that the user can see their photos on a Flickr map.
Purpose of the Experiment
Now that most of the functionality of our application is working, we wish to perform user testing to verify our design choices. We will use this information to make the final design changes before our prototype is finished, and to identify smaller issues that need polishing. Also, by collecting detailed timing information and error logs, we will verify that our system is fast and easy to use in spite of the small issues that still remain.
Implementation and Improvements(15 points)
Google Maps Integration
Once the user has entered a Google Maps URL into our program, the latitude, longitude, and zoom parameters are extracted from the URL. These are converted to Google Maps image tile numbers. Then, a grid of map tiles are downloaded from Google and combined into a large map image for use in the paper map.
Creation of Paper Map using Anoto Toolkit
The Swing application creates a new Sheet which is passed tag strings and a map image. It is then rendered to PDF, and the sheet is serialized and saved to a flat file for use in the next session.
Save-to-PDF Functionality
The user can choose to print their map or save it to PDF format for later or repeated printing. A file selection dialog box now allows the user to save their file to a location they specify on the hard drive. This option is now functional.
Flickr Authentication
Our application is officially registered with Flickr. To authenticate, we pop open a browser window so Flickr can verify that the user trusts our application. When this process is complete and the user has returned to our app, we double-check with Flickr before proceeding.
Image Selection Functionality
Previously, we had a hardcoded selection of example photos on our map preview page. Now, the user can select actual photos from their computer and see them on the map preview. We will generate thumbnails of them as needed. These images are then passed to Flickr for upload.
Additionally, a little more transparency was added to arrows indicated locations of images not selected on the preview pane. This was to visually de-emphasize the unselected image locations and cut down on visual clutter. We took into account Nielsen's practical suggestion to vary brightness/luminance values for objects that differ by color so as to help a potentially colorblind audience.
Flickr Upload
We are now able to hook into the Flickr API to upload photos to a (pre-authenticated) Flickr account. After the upload, the user will be given the option to see all the newly uploaded photos in his or her account.
Method (10 points)
Participants
Casper
Demographic Data
Age: 20
Sex: Male
Year In School: Junior
Major: Economics
Experience with Task and Application:
Casper is a very frequent Flickr user, he is the main photographer for his church, and take pictures very often. He understood very clearly what each part of the application refers to. He is very familar with the idea of tagging and uploading to Flickr and somewhat familiar with the idea of Geo-tagging the pictures (although he never used it). He was chosen because of his familarity with Flickr (which matches our user criteria) and his frequent picture-taking, which would necessitate an upload tool.
Brandon
Demographic Data
Age: 20
Sex: Male
Year in School: Senior
Major: Environmental Economics
Experience with Task and Application
Brandon is an avid photographer who began using Flickr approximately four or five months ago. He uses it to showcase his photos online for his friends to see (and to use as a burgeoning portfolio). While he has never used geotagging before (and the Flickr map feature was completely new to him), he appreciated the ability to portably mark his locations as he moved about. He is an intermediate-level computer user who demonstrated a familiarity with the web browser and file system. He was a friend of Andrew's and agreed to a request to participate in this experiment.
Mark
Demographic Data
Age: 20
Sex: Male
Year in School: Junior
Major: EECS
Experience with Task and Application
Mark is an advanced computer-user who is well familiar with Java programming, so he was already interested in our project when we requested him to participate. Though we realize this may contribute a bias in his observations (there is the possibility that he may evaluate our project based on the backend code rather than the front end user interface), he seemed to reserve himself to comments and judgments about the interface. We also did not discuss any specifics of back-end code with him, rather only revealing the front-end interface. Thus we can say that we did all we could to control for externalities in his trial. Mark is a periodic Flickr user but admits he doesn't tend to upload his photos very much. "If I took more photos," he told us, "I would definitely use this system."
Apparatus
First, we explained the master-apprentice model to the user and had them sign a consent form. Then, we read our demo script and gave them a handout with the tasks listed on it.
For tasks 1 and 3, we presented users with a laptop computer running our app. For task 2, we gave users a digital camera, sample paper map of Berkeley, and Anoto pen.
For all tasks, we tested the user while sitting at a table. Although it would be more ideal to test task 2 out in the real world while walking around Berkeley, it was raining and our users didn't have much time to spare. We decided to test them indoors and to tell them to pretend they were walking around Berkeley. We believe this is a good enough approximation because if the users can figure out how to mark the map while indoors, they can probably do the same outdoors.
Tasks
The following tasks were given to users to accomplish:
TASK 1: Using our application, create and print a map of Berkeley with the tags "campus", "telegraph", and "downtown".
TASK 2: Now you're outdoors, walking around and taking pictures. Take a picture of the Bart station, the Campanile, the Library, and Blonde's Pizza. At each place you take a picture, record your location and the appropriate tag(s) on the map.
TASK 3: Upload these photos with the corresponding pen data. Pretend you have already downloaded your photos to your computer--the four photos are located in My Documents\My Pictures\Sample Pictures. We will log in to a Flickr account for you, so you do not have to worry about using yours. At the map/photo visualization screen, deselect the photo of the sunrise. Upload the remaining three photos to Flickr.
Procedure
For each of our Pilot Testing, we have at least three people monitoring the testing procedure. One person will be the facilitator and other two people will observe and take notes while the testing is under progress.
First, the facilitator introduce the participants to the system. He describes to the participants what is the purpose behind our system. Since the participants are all flickr users, doing them will give them extra incentive to test the system and play with it to find flaw. Then the facilitator will biefly describes how it works, this is told on a very generel basis with all the detail eliminated. This is to ensure that the participant won't be completely directionless yet still do not know how everything works. We also encouraged the participants to speak their minds loudly and ensured them that it is totally our fault if they are thrown into a confucsion.
Then, the facilitator read our demo script to the participants, giving the scenario that participants may possibly want to use the system.Then we set up the laptop in front of our participants, with our program already started.
The participants start with Task 1, which is printing out a map of their choice along with the tags they want. The participants have to do everything themselves, including opening the brower to find google map and copy the link of a specific map to our application. We did not provide any help in the progress. We allow the participants to save their maps to pdf files and give them a demo map that has been printed earlier.
The participants proceed with task 2, which is taking pictures on the demo map we provided for them and tag it accordingly. We brought a digital camera with us and requested the participants to follow the exact same procedure, including actually taking a picture, mark the location of the picture with our pen and tag it accordingly.
For the last task, participants have to sync the pen with our application and actually upload to the flickr. Again, we started the application for the participants and provided no other help. Since everything is described in great detail in our instructions. The participants just have to follow our wizard-feel application and walk through the step of "Flickr-Authentication", which pop up a browser window to flickr, requestion participants to sign in, which we signed in with our account. After authentication is successful, we give instruction for user to close the browser window and return to our application. Then the user to prompt the put the pen in the dock and download the data accordingly. At last, we gave some instruction to the participants where to grab the pictures for upload. And participants uploaded the pictures to the flickr site.
After the experiement, we ask the participants to answer two survey questions. 1) How ease-of-use is our application and 2) How useful do you thinking something like this is. We also ask them to provide any other thought or additional comment they did not voice out during the experiment. At last, we thank them for their participation.
Test Measures (5 points)
We measured:
- The time to complete each task. The main goal of our project is to streamline the geotagging process, so it's important to collect data on how quickly users can use our system.
- The number of errors made. This allows us to see which tasks need the most attention when it comes time to redesign.
- The question "Is this system easy to use?" on a 1-10 scale. We want to know how comfortable the users felt with our system.
- The question "Would you use this system in your life?" on a 1-10 scale. We want to know if our application is useful to our users, or if they would rather use another tool or avoid geotagging completely.
Results (10 points)
Statistical Results
- Task 1 time to complete: average 2:48, std dev 26.6 seconds.
- Task 2 time to complete: average 1:42, std dev 43.0 seconds.
- Task 3 time to complete: average 3:20, std dev 110.0 seconds.
- Task 1 errors: average 1.66, std dev 0.577 errors.
- Task 2 errors: average 0.66, std dev 0.577 errors.
- Task 3 errors: average 0.33, std dev 0.577 errors.
- Is the system easy to use, on a scale of 1 to 10? average 7.33, std dev 0.577.
- Would you use this system, on a scale of 1 to 10? average 7.0, std dev 1.0.
Summary of Test Data
Overall, the tests went more smoothly than the last time, but we still have some minor kinks to work out.
Our first user had very little trouble with our system. Although there were situations in which he had to guess what to do, he always guessed correctly and did not feel confused.
In Task 1, he said he might have been confused when saving a map as a PDF. The save dialog box did not default to PDF format, so if he didn't know what he was doing he might have saved his map without a PDF file extention.
In Task 2, he wasn't sure how to mark a tag multiple times. Because of the large size of the boxes, he thought he might have to write numbers in them. Instead, he drew tally marks, which is a good solution. He asked us to change the size or shape of the tag boxes.
In Task 3, when our app asked him to close his browser and return to our app, he asked "close my browser entirely, or just the Flickr tab?" He then correctly proceeded to close just the Flickr tab.
Our second user was not very good at following directions, and he encountered more problems.
In Task 1, he browsed to a Google map of Berkeley. He read our instructions but could not find the "Link To This Page" link on the Google maps page. He decided to plunge ahead, so he skipped our instructions and went to the next screen. He then found that he couldn't go back to the instruction page, but he was able to click our help link above the URL field. This help link didn't mention the "Link To This Page" button, so he went ahead and copied the plain "maps.google.com" URL (with no location information) into our app, which we cannot use to generate a map. This is a very critical incident.
In Task 2, the user spent a while figuring out how to tag a photo. Eventually he read the instructions on the map and followed them correctly. Then, he tagged one of his photos with the wrong tag. "Can I go back and change my mistaken tag?" he asked, and the answer was no. Interestingly, this user used the pen like a stylus by tapping it on the paper, while most of our users prefer to draw small marks with it.
In Task 3, he had no problems. He said he would have preferred to have his thumbnails marked with checkmarks instead of "X"s when they are marked for upload.
Our third user had very few problems. In task 1, he repeated User 1's comment about making PDF the default file type for the save box. He also noted that it was annoying to go back and forth from our app to Flickr's page. He breezed through Tasks 2 and 3 with no critical incidents. Like our other users, though, he didn't read the paper map instructions until he wasn't sure what to do.
Discussion (15 points)
Interpretation of Statistical Results
Timing
Our goal is to make geotagging fast, and we've accomplished this. Printing a map takes only 2:48 on average, which is fast enough to do as you're running out the door on a trip. Marking photo locations takes little time. The entire uploading process takes only 3:20 on average, which is very fast. These times will be even shorter once users have become accustomed to our system.
Errors
A few error-prone areas still exist in our design, especially in the printing task. Getting a proper Google Maps URL has proven to be the most difficult part of our application for users to complete properly.
Survey Questions
Despite the few remaining errors, users rated our system 7.33 out of 10 in terms of usability. They also thought it filled a need in their lives, rating it 7 out of 10 in terms of their desire to use it themselves. We are proud of these numbers and plan to improve them with our final prototype.
What worked well
Correct Paper Map Tagging
Users understood the correct order to take a photo and give it metadata: first take a photo, then tap the map, then tap the label(s). This is attributed to the larger and clearly-worded instructions given to the paper map.
Understanding Flickr Authorization
Despite the fact our directions were slightly ambiguous, users generally were able to follow the somewhat-confusing process of Flickr API authentication. By nature of Flickr's API token authentication, there is no way around the fact that the user has to open a browser window to authenticate into their account, then close it again for the application.
Understanding Photo Thumbnail Checkboxes
Even though there's no text in our app to explain this (which we will remedy), users understood that the checkboxes next to each thumbnail in the photo-map visualization panel prior to upload indicated that a photo was in the upload queue (and that deselecting the checkbox next to the thumbnail indicated that the photo was not to be uploaded).
Multiple File Selection
Despite the fact that there is no text to explain this, all users understood that shift-clicking a range of files in the file selection box (the JFileChooser widget given to us by Swing) would allow users to select multiple images.
Improvements Needed
Improve Google Maps instructions
One of our users had a major problem getting the Google Maps url. He couldn't find the "Link To This Page" button the the maps site. We should include a screenshot of this part of the Google Maps interface in our instructions, to help the user find this button and to clarify that it's on the maps site and not in our application. We should also allow users to come back to our instruction page if they've gone ahead to the next page.
Write an error check for wrong map URL
In case a user enters a plain Google Maps URL ("maps.google.com/") instead of one with location data, we should catch this mistake and tell the user to return to Google Maps and click the "Link To This Page" button. We also need to include more error handling in our code in case a proper URL can't be fetched due to network outages.
Save dialog box should default to PDF format
All users were unsure what format to save in, and noted that if they were novices they would have saved the map with no extention at all. We should set the default file format to PDF in the file-save dialog box.
End the Swing print map subtask smoothly
By the time Mark had saved his map to PDF, the application should have informed him that the print task was now finished and allowed him to close the application and read the instructions on the paper map.
Improve tag interface on paper map
One of our users wasn't sure how to use the tag boxes on the paper map. The other two read the instructions and tapped them with the pen as if they were buttons, which is what we wanted. We're investigating three solutions to this problem:
- 1. Make the tag box small to indicate that nothing can be written in it and that it should be tapped. Since some users don't know that the pen can be tapped, this may confuse them.
- 2. Have a row of checkboxes next to each tag. This may confuse users who want to tap and don't know which of the many boxes to tap.
- 3. Leave the boxes alone, but include an image of a sample box with tally marks in it. We think this is the best solution because it gives users a clear model to follow.
Highlight instructions on paper map
Most users ignored the instructions until they didn't know what to do. We should visually highlight the instructions to make them stand out so users will notice them sooner.
Clarify Flicker Authentication instructions
Users were not sure whether they needed to close their immediate tabbed window or the entire browser process itself. We should change the text on the Swing interface from "Close your browser" to "You may now close Flickr", as this is more precise.
Include directions for selecting multiple photos
Although all our users knew how to use Shift to select multiple photos in the file open dialog, we cannot assume that they all understand this. We will add some text to the file chooser screen instructing them how to do this.
Include textual descriptors and instructions on the photo-map visualization panel
There is barely any text on the photo-map visualization screen, mainly because we did not find enough time to add it. We will add labels indicating that the panel on the right contains geo-location coordinates, and that the green arrow indicates the location of the currently-selected image. We should also add a label to the bottom-left panel named "list of photos to upload" and text (or a tooltip) explaining that if they do not want to upload that certain photo, they would un-check the selection box on the photo itself.
Change photo thumbnails to use checkmarks instead of "X" checkboxes
Since the checkbox indicated that the photo will be uploaded, a checkmark graphic indicating "accepted" is more appropriate than an "X" checkbox graphic, which could be interpreted to mean "rejected".
Workload breakdown (5 points)
Andrew Hao
Designed backend framework for interface between Swing application and map creation classes.
- Implemented Sheet object persistence between PrintMap/SynchMap sessions.
- Hooked into Flickr API using FlickrJ Flickr Java wrapper package.
- Stripped image EXIF data and parsed tags via MetadataExtractor libraries.
- Designed Photo and InkMark container classes to provide and interface with functionality to match photos to ink marks and tags.
- Wrote GMaps tile-stitching component to merge map tiles into a large image.
- Troubleshooted PaperToolkit bugs and errors, corresponded with Ron Yeh.
Obtained two users, Mark and Brandon for testing.
Wrote documentation: Participant Description, Implementation, Parts of Results & Discussion
David Wallace
Wrote Google Maps integration code.
Acted as note-taker in user tests, writing Critical Incident Logs.
Wrote Handout and Demo Script for user tests.
Wrote Introduction, Implementation and Improvements, Apparatus, Test Measure, Results, Discussion.
Yang Wang
Programmed batch eventhandler for the pen
wrote description for Casper
Wrote Experiment Procedure
Be an observer at the experiment.
Jack Yeh
Jack did not participate in this project.
Appendices (5 points)
Handout
TASK 1: Using our application, create and print a map of Berkeley with the tags "campus", "telegraph", and "downtown".
TASK 2: Now you're outdoors, walking around and taking pictures. Take a picture of the Bart station, the Campanile, the Library, and Blonde's Pizza. At each place you take a picture, record your location and the appropriate tag(s) on the map.
TASK 3: Upload these photos with the corresponding pen data. Pretend you have already downloaded your photos to your computer--the four photos are located in My Documents\My Pictures\Sample Pictures. We will log in to a Flickr account for you, so you do not have to worry about using yours. At the map/photo visualization screen, deselect the photo of the sunrise. Upload the remaining three photos to Flickr.
POST-INTERVIEW QUESTIONS Rate the truthfulness of these statements, with 1 indicating strong disagreement and 10 indicating strong agreement.
- This system is easy to use.
1 2 3 4 5 6 7 8 9 10
- If I were travelling, I would want to use this system.
1 2 3 4 5 6 7 8 9 10
Demo Script
Pretend you are travelling and you want to keep your online friends informed of your adventures.
Our system is designed to record the location of digital photos as they're taken. You will be marking your location on a paper map using a "smart pen" which digitally records your penstrokes. When you're done, our application will upload your photos to Flickr (a photo-sharing website) which will show you a map of your photos. In addition to recording the location of your photos, you can also apply "tags" (or categries) to each one to keep them organized.
There are three stages to your task:
- First, you will find a Google map of your city and bring the map into our application. You'll also type a list of tags you think you might use on your travels -- for example, the names of local tourist attractions. Then, you'll print out your custom-made map and take it with you. [Show user a sample map.]
- Second, you will go out on the town. Every time you take a picture, you'll circle your location on the map using the pen, and optionally check a box for any tags you want to apply to that photo. [Mime these actions on the sample map.]
- Lastly, when you get home, you'll download your digital photos the way you usually do. Then, you run our application which combines the location information with your photos. It will then send them to Flickr which will create a map with your photos in the correct places. [Show the user a sample map page on Flickr.]
Critical Incident Logs
User 1
User 1. Task1: printing 9:57:39: user starts app; reads directions. 9:58:00: user is getting a google map. 9:59:00: got a map. clicks "link to this page". copies URL. 9:59:20: pastes url, enters tags. user is confident. 10:00:00: clicks save map; gets impatient. we should add a progress bar here. ! 10:00:15: user says he might not know to save as a PDF since it's not automatically set as the default file type. 10:00:30: done. 1 incident, 2:51 elapsed. Task2: photo taking 10:02:30 takes bart station picture. marks map after taking photo. marks tag properly. 10:02:52 campanile. marks map, tag. 10:03:12 campus, ok. 10:03:30 blondie's pizza, ok. ! User didn't quite know how to mark a tag multiple times. He correctly made tally marks, but said he felt that he should write numbers because of the size of the box. 1 incident, 1:00 elapsed. Task3: uploading. 10:04:13. clicks Synch pix. 10:04:25. "authorize?" this is a novel concept, but he feels ok about it. reads instructions. 10:05:01: user signs in to Flickr. 10:05:30: user reads result page on flickr. wants to read about authentication list out of curiosity, so reads flickr help for a while. ! 10:06:16: user is satisfied, goes back to app. "close my browser entirely, or just close the flickr page?" 10:06:30: user inserts pen. 10:06:47: user finds photos and multiple-selects them properly using Shift. 10:07:26: "whoa, the window got bigger!" 10:07:40: "looks like I can cancel uploading photos with the checkboxes. that's nice." 10:08:19. Here I go. Uploading images. There's a typo on the upload button. 10:09:40. Success. user decides to see photos. "Hey, this is cool." 1 incident, 5:27 elapsed ease of use? 8 out of 10. would you use it? 6 out of 10.
User 2
User 2: Task 1: printing. 10:36:47: reads map instructions. 10:37:37: has gotten map of berkeley. 10:37:48: user can't find "link to this page" on google maps. ! 10:38:29: user skips instructions; can't go back. ! 10:38:40: user pastes plain url into our app. 10:39:06: user enters tags. 10:40:00: user prints map contrary to our instructions. 2 critical incidents, 3:13 elapsed Task 2: photo taking. 10:42:00: takes photo of blondies, taps map. 10:42:57: user doesn't know how to tag. eventually reads instructions, taps square. 10:43:15: campanile, ok. 10:43:30: library, ok. 10:43:42: tap, ok. ! "can I go back and change my mistaken tag?" 1 incident, 1:42 elapsed Task 3: uploading. 10:44:33: authentication. 10:45:00: reads auth page. ok. 10:45:10: ok, goes back to app. 10:45:25: pen, ok. 10:45:44: selects photos. uses shift. 10:46:00: user deselects with no problem. "check would be better than an x." 10:46:45: sees flickr page. No problems, 2:12 elapsed. ease of use? 7 out of 10. would you use it? 8 out of 10
User 3
User 3 Task 1: printing 11:36:10 reads instructions, opens browser 11:36:50 gets map 11:37:10 clicks "link to this page" 11:37:30 pastes url ! 11:37:36 "annoying to go between browser and your app" 11:38:00 enters tags. 11:38:10 saves map. should have a progress bar here. ! 11:38:30 "interface looks nice." saves map properly. expects save box to default to PDF format. "Normal people won't understand." 2 incidents, 2:20 elapsed Task 2: photo taking 11:40:10 user didn't see instructions; reads them now. 11:40:30 "'tap' on the map?" 11:41:00 ooh, the pen vibrates. 11:41:20 bart station photo, location, tag. ok. "is that it?" 11:41:50 library. location, tag, ok. 11:42:20 campanile, location, tag, ok. 11:42:36 Blonde's, location, tag, ok. No incidents, 2:26 elapsed Task 3: uploading 11:43:25 synch pics. 11:43:30 flicker auth. 11:44:00 auth finished. cool. 11:44:30 pen data 11:44:45 selecting photos. uses shift. 11:45:11 deselects sunset photos. 11:45:30 user plays with map preview page. "cool" 11:45:46 uploads, done. No incidents, 2:21 elapsed easy? 7 of 10 would you use? 7 of 10
Consent Form
Researchers: David Wallace, Andrew Hao, Jack Yeh, Yang Wang [Undergraduate CS 160 Students]
Description
You will be participating in human-computer interaction research involving the testing of a software interface. There will be between 3 and 4 other participants who will take part in this study. Your participation will help us iteratively test and design for potential hurdles in the software interface we are building.
Procedures
- We will explain and introduce you to the software, which is currently in a very early stage of development. You may ask questions at any time.
- We will then give you three tasks to accomplish via the interface, and ask you to think aloud as you go about performing these tasks. A research associate will be observing your actions throughout this process.
Risks/Discomforts
Reasonably speaking, there are virtually no risks involved with this experiment. If, at any time, you feel uncomfortable with the proceedings of the research task, you may alert a researcher and leave at any time.
Benefits
This is an unpaid experiment, and your participation demonstrates your boundless good will. We will express our gratitude in flattering comments.
Confidentiality
Your confidentiality will be maintained in all documents and findings after this experiment. Your name will not be referenced, nor anything that could belie your identity.
Signature
I have read and agree to the conditions outlined above.
______________________________ (Name) _______________ (Date)
