PilotStudy-Group:The Anototators
From CS160 User Interfaces Fa06
Contents |
Anoto Medical Imaging Annotator: Pilot Study
Introduction (5 points)
Introduce the system being evaluated
The system being evaluated is the AMIA (Anoto Medical Image Annotation) program developed by The Anototators. The purpose of this program is to provide medical professionals and students (primarily neurosurgeons and radiologists) the ability to gather collaborative work on medical images, all using the Anoto paper & pen system. AMIA provides a new way to write remarks on MRI image slices and other types of medical images that can be viewed on the computer. In addition, AMIA is uniquely built to also work well in a teaching environment, where students can comment on images and have their comments per slice be saved, available for review by the instructor.
The purpose and rationale of the experiment
The primary purpose of the AMIA, after receiving feedback from our target audience, has evolved into a teaching use in fields where students must learn to read and analyze any of several types of medical images. To be more precise, students are able to peruse images, print them out and write annotations and other markings on the slide. These markings can then be made available to other accounts that have the proper privileges (e.g. instructors), while remaining hidden from other student-level accounts. This individualizes the current teaching procedure, which consists of students pointing out features of interest with a laser pointer in a large group. With AMIA, each student would be able to find regions of interest in images independently, allowing more opportunities to learn.
Implementation and Improvements(15 points)
The first major improvement that was added to AMIA was a standard, file chooser dialog box for loading image sets. This replaced a text-box-based dialog window, which required the user to type in the image set he or she was interested in using. This change was prompted by a natural progression towards higher fidelity UI objects as well as feedback we received from the course administrators. Given that image sets are saved as numbered file names within directories, our file chooser dialog box allows users to select a directory or any image from the image set, to load the entire image set as thumbnails.
The second addition to AMIA was a working image thumbnail scrolling bar. The thumbnail scrolling bar provides the user with a film-strip browsing equivalent, allowing the user to quickly scan through a number of image slices and select one that is of interest. By connecting this with the file chooser dialog box widget, the thumbnail scrolling bar gets automatically populated whenever the user loads an image set. Bugs related to associating textual comments with images were also fixed.
Access control features were also integrated at this stage. By integrating access control, AMIA is a step closer to its ultimate goal of providing teachers (a type of user) with access to student comments (created by another type of user) and Anoto annotations.
The biggest addition to AMIA at this stage was stroke rendering using the Anoto system. When implemented for the pilot usability study, users were given the opportunity to manipulate (through annotations) printed copies of medical images using the Anoto pen and see their changes reflected on screen.
Method (10 points)
Participants: Demographics and Selection Process
1: A post-doc in the neuroscience department, her research focuses on aging, Alzheimer’s disease, and imaging. She uses functional MRI in particular to observe the effects of aging and related diseases. She participated in the low-fidelity prototype interviews, and was found through the spouse of a coworker in one of the Anototators' labs.
2: Professor of Radiology at UCSF and practicing radiologist. His research interests include the “development of new diagnostic techniques for the assessment of osteoporosis.” He uses MR imaging to investigate cartilage, bone tumors, and other musculoskeletal topics. He was found based on a recommendation from one of the contextual inquiry participants.
3: Student in Joint Graduate Group in Bioengineering at UCSF and UC Berkeley. Her research focuses on the use of novel MR image acquisition and analysis techniques to study intervertebral discs. She was found by one of the Anototators who is in the same graduate program.
Apparatus
1: Laptop computer (MacBook Pro running Windows XP and Eclipse 3.2) : This was used to host the AMIA application
2: Anoto pen : This was used to capture strokes and annotations created on printouts of medical images
3: Medical images : Images of the brain were printed on paper which had the Anoto pen pattern pre-printed
4: Timer : Used to time tasks
Tasks
In increasing order of difficulty,
1: Log into AMIA (using pre-assigned name and password) and load an image set. For this task, we were looking for users to quickly log into the system and load an image set with no difficulty.
2: Add text comments (via computer keyboard) and save them (paired with the associated image slice). Based off of the feedback we received at an earlier stage, we wanted to make sure that the users understood the "Save Comment" action and its association with the current image slice being viewed.
3: Print an image from AMIA and annotate it using the Anoto pen. As a feature that we never tested on users before, we were curious to see if the streaming of strokes from the pen and its emergence on screen would proceed without hitches (technically). We were also curious to see what kinds of annotations users were interested in making.
Procedure
Upon meeting our study subjects, we introduced ourselves and our project, providing them a brief overview of the system and its goals. We also highlighted the fact that we were testing our system and them, hoping to allay the fears and stresses related to testing. Once the background was set, we briefly ran through the tasks we asked them to perform and how they fit in to our applications goals as a whole. We then allowed the subjects to work with the system and perform the various tasks. At this stage, we answered usability questions and collected useful feedback on how we could redesign the interface in the next iteration. At the same time, we timed the tasks they performed and collected information that would form a basis for our quantitative analysis.
Test Measures (5 points)
For this pilot study we decided to measure the time taken to accomplish the three most common tasks on our application:
1. Logging into AMIA and loading an image set
2. Adding textual comments (through the keyboard)
3. Printing out an image and annotating it
Logging into the AMIA system involved typing in a pre-defined username and password and clicking an OK button. Loading an image set involved locating the Open button, navigating a file system structure through a standard file-open dialog box, choosing a directory and clicking Open.
Once an image set is loaded, the user can select an image and type in text comments (through a textbox) related to that image and save them (by clicking on a Save Comment button). These text comments are then linked to the image selected at the time.
Annotating the image was done by using the Anoto pen to write on a pre-printed medical image on paper with the Anoto dot pattern. As our printing function was not enabled at this stage, we used a WOZ technique to simulate the printing of the image.
As mentioned earlier, these tests are representative of the most frequently used tasks on the AMIA application. By measuring user performance against these tasks, we hope to gain insight into whether or not our UI allows its users to accomplish their goals easily. It also provides us much insight into the usability of our current interface as UI designers.
Results (10 points)
Interviewee 1
Task times:
1: 13 sec. 2: 16 sec. 3: 27 sec.
General Comments:
She felt that the red color of the printouts makes it difficult to differentiate between gray and white matter, and the anatomy of the brain in general. This shortcoming would especially affect the teaching setting, since it would make it more difficult for students to recognize features. A compromise would be to always display the black and white image on the screen, but printout the red version for use with the anoto dots.
During the second task, the requirement of the user to press the “save comment” button was not intuitive. She recommended linking the comments to the images so they automatically save after one enters text. Also, she mentioned that the anoto strokes must be saved to each slice, which was planned originally but not implemented in time for the interview.
One important comment we received was that the comment box would be better for all text, not just those added later by another person. In other words, the paper UI should be set up only to afford shape drawings and very brief comments. The user should be expected to enter most text on the computer to maintain legibility.
She appreciated the live streaming nature of our implementation. In her words, it “made it easy to get immediate feedback from the computer” to verify that one’s strokes are being registered in the correct location. In general, she felt batched mode would be less useful, unless one is traveling. She felt that the folder opening made sense, but likely only due to the tutorial we gave at the beginning of the interview. It would be nice to be able to choose individual slices for loading. Then one could load slices from multiple image sets.
Interviewee 2
Task times:
1: 22 sec. 2: 10 sec. 3: 16 sec.
General Comments:
As a radiologist, he stressed that HIPAA standards would prevent one from ever printing out patient data and bringing it to a café or on a plane. Thus, one would need to annotate images in one’s office or reading room.
He described how radiology students are currently taught. Fourth year medical students, usually in groups of 10, are lectured by a professor who uses a projector and laser pointer. The students also use laser pointers to indicate features of interest on medical images. Since the students are fourth years, they are well acquainted with healthy anatomy and can usually perceive pathological anomalies.
If one were to use the anoto system, the radiologist envisioned giving students handouts with anonymized patient data printed on anoto paper, and then instructing them to indicate pathologies with their anoto pens. However, the review would likely need to be one-on-one between the teacher and the student. This would be afforded by our various permission levels, which would only allow the teacher to view each student’s work. Otherwise, if students could see each other’s work, there would be a chance of humiliation if a student made a stupid mistake. The radiologist stressed that the system would not offer a big advantage over current systems. However, he also emphasized that the “gimmick” nature of the device would be a strong selling point. That is, the novel idea of a digital pen for student interaction would like grab their attention and promote participation in class discussions.
As the interview concluded, the interviewee made sure to point out that very stringent requirements are in place for monitors that are used for medical image-based diagnostics. These include contrast, brightness, and resolution criteria that could not be provided by a paper printout. Therefore, the anoto-based system would not be acceptable as a diagnostic tool. However, it would be very handy for teaching purposes.
Interviewee 3
Task times:
1: 20 sec. 2: 8 sec. 3: 13 sec.
General Comments:
The interviewee was visibly impressed by the technology. She thought the implementation was intuitive and easy to follow. However, she did note that the “save” and “save comments” buttons were confusing. She recommended that we get rid of the latter button and make comment storage automatic. She also thought the program should be able to load two images from different sets and display them side-by-side. However, she then noted that this could possibly be done by running two instances of the program and loading different sets or individual slic
Discussion (15 points)
Given that the primary use, or at least, the primary intended use of the AMIA is in a teaching setting, it would be best for us to do as much as possible to facilitate the use of the program in such a setting. Thus, the remark that the first interviewee made about the color of the image and how it would detract from the usability of the program in a teaching setting was very important. The choice to tint the image in red was originally made to facilitate the pen's ability to sense the dot field printed over the image. One workaround that we can use to overcome both obstacles, that is to say, allow the pen to still read the printed pattern well and also allow the image to be in its native color or grayscale, would be, as written in the notes, keep the printout in red, but display the image on the computer in the normal color. This workaround leverages, and relies on, a fact that was also mentioned in the same interview, which is the ability of the pen in streaming mode to give the user instant feedback about where exactly on the image the user is marking at any given time. Thus the user can refer to the screen in order to find the appropriate regions of interest on the image, then, using that as a reference, draw on the printout.
On a recommendation from an interviewee, we will get rid of the save comments button and make comment storage automatic. This should serve to deobfuscate the commenting procedure, and lead to users losing their comments less often due to forgetting to save. Of course, this is assuming the problem that people would have most is forgetting to save their comments. If, instead, the problem ended up being that people tended to accidentally mess up the comment or comments already entered into the box, then an autosave feature would actually be less effective and detract from usability, in which case the save comments button is entirely justified.
Another possible change is to change image set loading from a normal open dialog to a directory tree, which is intended to make figuring out how to open image sets trivial. This change would happen if a test, conducted over a much larger user base, indicated that the dialog box was confusing and that a tree directory structure would be better. If it proves that a tree directory structure ends up just as bad, if not worse, other alternatives will have to be found.
The Experiment
If we are to conduct the "real" experiment, we would only use radiologists and radiology students, further classified by the amount of their experience with current technologies. In that case, we would also use real medical images featuring real pathologies.
Workload breakdown (5 points)
Robert Held
>Code (22% - paper user interface, directory loading functionality )
>Interview #1,2,3
>25% of write-up
Edward Karuna
>Code (22% - login/password management, image exporting)
>Interview #1
>25% of write-up
Anirudh Vemprala
>Code (22% - image resizing/scaling, dynamic thumbnail selector)
>Interview #1
>25% of write-up
Gene Zhang:
>Code (33% - integrating all members' contributions, stroke rendering)
>Interview #1
>25% of write-up
Appendix
Consent Form:
Media:AnototatorsPilotConsent.pdf
Study Script:
Raw interview notes:
Media:AnototatorsPilotInt1.jpg
Media:AnototatorsPilotInt2.jpg
Media:AnototatorsPilotInt3.jpg
