LoFi-Group:The Anototators
From CS160 User Interfaces Fa06
Contents |
Team
Introduction and Mission Statement
The system being evaluated is the AMIA (Anoto Medical Image Annotator) system first proposed by the Anototators. The purpose of this system is to provide medical professionals (primarily neurosurgeons and radiologists) the ability to select regions of interest (ROIs) as well as gather collaborative work on medical images, all using the Anoto paper & pen system.
Mission: To provide medical professionals a collaborative system for medical images, that enhances their existing workflows by leveraging the benefits of the Anoto platform.
The experiment conducted was an evaluation of a low fidelity prototype of AMIA amongst medical professionals to gather appropriate feedback on the proposed design.
For this assignment, the team divided the work amongst ourselves as follows:
Anirudh: Built lo-fi prototype; wrote wiki report
Edward: Built lo-fi prototype; conducted user testing
Gene: Built lo-fi prototype; conducted user testing
Robert: Built lo-fi prototype; conducted user testing
Prototype
Figure 1: Initial Sketches
To build the low fidelity prototype needed for user evaluation, our team started with a sketch of the UI that corresponded to the features we felt were essential to completing the necessary tasks on the medical images. We quickly identified the need for having a large image viewing space, a sliding preview pane to navigate through the slices that constitute a given image (seen at top) and a collapsable hierarchy panel to illustrate the layers of annotations or ROIs (seen at left).
Figure 2: Completed low fidelity UI
The prototypes seen here focuses on the main application screen. The main screen provides a large viewing area for the medical image, clearly labeled buttons (right hand side) corresponding to possible actions on the image, a text box for typed text input, drop down panels (left hand side) corresponding to different annotation layers.
Prototype Interactions
Figure 3: Login screen
Figure 4: Print screen
The user (more specifically, a resident medical student) begins his interaction with AMIA by logging into the system with a username and password. Upon successful login, the user is directed to the application's main screen from which he or she can select images for examination. Upon selecting the appropriate image, the user can print out an image slice on Anoto paper for annotations or ROI selection.
Figure 5: Print out of image on Anoto paper
In this case, the user has chosen to print out the medical image for annotations. After completing the annotations, the user checks off the completed check box.
Figure 6: Imported annotations
When the user has completed annotating the image, he can dock his pen and have the annotations transferred from the hard copy onto the digital image as seen in the above figure.
Figure 7: Instructor view
Through a similar login process, an instructor can log into the application and view the annotations of all the residents using the system. The instructor (usually a chief radiologist) can then bring collaborative discussion on the related medical issue and analyze the specific input of each resident.
Method
Interviewees:
Three participants were selected. Two were part of the neuroscience program at UC Berkeley and one was a graduate student in the Joint Graduate Group in Bioengineering at UCSF/UC Berkeley. The neuroscience participants were found through the spouse of one of Robert's labmates. The bioengineering student was found through one of Robert's classmates. Although one of the group members is also in the Joint Group, he had no prior contact with the selected interviewee.
Interviewee 1: A post-doc in the neuroscience department, her research focuses on aging, Alzheimer’s disease, and imaging. She uses functional MRI in particular to observe the effects of aging and related diseases. She also performs ROI analysis, chiefly using a program called MRIcro. She uses SPM and Excel to perform numerical analysis.
Interviewee 2: Graduate student in neuroscience department. She received a BS from UCLA and works on image processing in MRI. Her research focuses chiefly on neurodevelopmental disorders in children, including ADHD and childhood-onset schizophrenia. She ROI editing and viewing tools routinely.
Interviewee 3: Graduate student in bioengineering department (UCSF/UCB). His work deals chiefly with MRI and CT bone scans. He has written custom software for ROI selection and management and deals regularly with radiologists who review medical images obtained by his lab.
Interview Roles
Interview 1:
Ed: Recorder
Gene: Computer
Robert: Facilitator
Interview 2:
Ed: Computer/Recorder
Robert: Facilitator/Recorder
Interview 3:
Gene: Computer/Recorder
Robert: Facilitator/Recorder
Testing Environment and Equipment Setup
For each interview, a large conference table was employed. The Anototator group members all sat on one side, opposite to the interviewee. Initially, the log on screen mock ups were the only the only things shown to the user. Then, once he/she logged in, the main interface board was shown. All the other parts, such as menus, alternate medical images, and dialog boxes, were kept behind the group members on the floor or on their laps. Care was taken to hide items not currently in use on the interface to avoid confusing the user. Three tasks were required of the interviewees. In the hardest one, they had to perform ROI analysis by drawing ROI's on a medical image printout and choosing either to run histogram analysis or to calculate the area of the ROI. The easiest task required them to printout a medical image, add general comments, upload them back into the program, and save the changes. The medium-difficulty task had the interviewees logging into the program as a radiology teacher, loading Anoto-based comments made by several students on one medical image, and adding his or her own feedback using on-screen controls. A dry erase marker was provided to the interviewee for use on a transparency to simulate Anoto pen usage. All members present helped at some point to find items that needed to be displayed, but the person playing the computer was chiefly in charge for displaying things to the interviewee. Specific comments were collected on the interface as the task proceeded. At the end of the tasks, the interviewees were asked to provide general feedback on the entire display. A few suggestions were immediately implemented by changing menu names and dialog box functionality. Please see the discussion section for more details.
Results
Interview Summaries:
Interviewee 1:
At the beginning of the first task, she mentioned that she would much rather be able to select folders than be confined to a pre-determined directory. This would allow one to access other people’s scans on the network. In her line of work, it is also necessary to view sagittal, coronal, and transverse planes from a patient scan. She struggles a bit with initiating the ROI analysis process, mostly because she wasn’t sure where to click to begin. Once the process was initiated, she noted that multiple slices should be printable with a single command. Also, ROI’s tend to be made of several separate regions, so it would be good to group many shapes into one ROI. For analysis, mean, median, max, min, and standard deviation values of pixel intensities are important. She said that it is also crucial to be able to manually define exported filenames when one is saving Excel and .tiff files.
The second task involved commenting on layers. The interviewee noted that it would be idea to be able to overlay Anoto-created layers on many different images. That way, one could compare comments made on MRI images to CT scans of the same patient. Also, autosave is not good. A save prompt when quitting reminds the user to name the session appropriately for later use. Similarly, being able to manually define the layer names is crucial for easy reference later.
During the third task, it was clear that the user had become well acquainted with the printing process. She began the printing process without hesitation. While she was commenting an image from the student point of view, she noted that the teacher should be able to share with all students. She also thought the “add” button for appending text comments was not intuitive. In particular, it wasn’t clear whether or not she had to press it after entering text. She did appreciate the auto upload feature of the Anoto pen, and thought it was nice how effortlessly the pen operated. She suggested that the “Layers” label for the pull-down box of multiple user comments should be better named. We subsequently changed it to “Comments” and renamed the “ROI’s” tab to “ROI sets.” At the end of the interview, she emphasized that the classroom setting would probably be the most applicable and open to adoption. Using it in an educational setting would get students used to the software, and then they would likely want to use it in their professional careers.
Interviewee 2:
At the outset of the first task, she stated that the program needed to provide access to multiple folders, not just one preset directory. She also thought it would be good to not grey out the “ROI analysis” menu. It would be better to let the user try to press it and be prompted to print out and add ROI’s first. After the prompt, however, the “Print” button seemed intuitive. Like the first interviewee, she thought that multiple slices per print command would be useful. Also, one should be able to rename the ROI’s after capture so they are more descriptive than just “ROI 1” and “ROI 2.” At the same time, ideally one could open or double click on of the ROI names to open the accompanying analysis. The interviewee mentioned several times that she would expect documentation and a tutorial with the program. When she starts using a new program, she looks over those types of materials first to get acquainted with the interface. She also told us that we should include the ability to export .tif files of histograms, as well as the capacity to specify the directory for saving and exporting.
The second task was completed quickly and with no negative comments. The interviewee felt that the process was simple and straightforward. During the third task, the “add” button was not intuitive for adding text. However, the save button was clearly labeled and easy to find. She told us that the to saved file name should be displayed in a dialog box, not constantly displayed in the main interface.
In general, the interviewee thought the interface was “really good.” However, she made it clear that the student/teacher exchange was the best part. As an afterthought, she gave light praise for the ROI functionality, but only as a teaching tool. She mentioned that she knew of no other system that could so quickly and intuitively share hand drawn medical imaging commentary. Then she repeated that it was critical to improve the file management system.
Discussion
Running the low-fidelity interface by the users was remarkably useful. Some of the major points we from the experiments that will shape the redesign of our interface include:
ROI feature removal
Upon examination of our users' interaction with this feature, we learnt that not only were they having some difficulty achieving the required tasks, but they also found the use of paper-based medium for selection regions of interest rather unhelpful. Given that there are existing tools that can do ROI selection in an automated fashion with more sophistication than AMIA, it was decided to focus in on the collaborative/instructive aspect of the program as there are no tools that provide such functionality in the market today.
Change in commenting
AMIA's collaborative/instructive functionality allows for typed text input to also be submitted for review by an instructor. However, users found the "Add" button (meant to represent Add Comment) rather confusing. This will probably be renamed to "Save Comment".
File saving and opening
Some users expressed concerns with the way file I/O was conducted through the UI. The prototype had a persistent "save file" pane which was confusing to users. Some suggested moving this into a more traditional File->Save or Save As action. Users also recommended not launching the application with a "Open File" dialog, but rather allowing them to manually opening files. To that end, features like thumbnail features in the file dialog boxes would be helpful as our application deals with images.
Importance of orientation
Though the current prototype provides limited information regarding orientation of the image being examined, user requests for sagittal, coronal, and transverse views will have to be addressed in the AMIA.
Print multiple slices
The current prototype of AMIA required the user to print out slices of an image one at at time. Based on the prototype test, we learnt that it would be easier for the user to select multiple slices and print them at once on Anoto paper.
One that not revealed in this experiment was how our system would work alongside existing systems used by our target users.
Appendix
Critical incidents
Task 1:
Participant 1:
- Would much rather be able to select folders than be confined to a pre-determined directory. This would allow one to access other people’s scans on the network. (3)
- Need to be able to view sagittal, coronal, and transverse planes. (4)
- ROI analysis startup needs to be more intuitive. (3)
- When printing, one should be able to select multiple slices at a time. (4)
- Some ROI’s have separate regions. One needs to be able to group multiple shapes into one ROI. (4)
- Mean, median, max, min, and standard deviation values of pixel intensities important. (3)
- Need to be able to manually define exported filenames. (3)
Participant 2:
- Need access to other folders. (3)
- Don’t grey out “ROI analysis.” Let the user try to press it and be prompted to print out and add ROI’s first. (2)
- Multiple slices per print command would be useful. (3)
- Naming ROI’s in collapsible box on left hand side would be good. (3)
- Ideally one could open or double click on of the ROI names to open the accompanying analysis. (3)
- Documentation would be critical. Willing to read it over before using the software. (5?)
- Ability to export .tif files of histograms would be great. (3)
- Export options need to be expanded and improved. (3)
- Need to be able to specify the direction for saving and exporting. (2)
Task 2:
Participant 1:
- Teacher should be able to share with all students. (3)
- “Add” button confusing. She wasn’t sure if it was necessary to press it after typing in a comment. (3)
- “Layers” term confusing. Should rename them “comments.” Also rename ROI’s as “ROI sets.” (2)
- Then offer the ability to manually select individual ROI’s within one person’s set.(3)
- Reiterated importance of sagittal, coronal, and transverse views.(4)
Participant 2:
- Thought it was straightforward. No complaints about the interface.
Task 3:
Participant 1:
- Autosave not good. Provide prompt when quitting to save. (2)
- Need to be able to overlay layers on other images. This could be done by saving and opening layers separately from the image sets. (3)
- Need to be able to manually name the Anoto layers (2)
Participant 2:
- “Add” button not intuitive for adding text. (3)
- Save file name should be displayed in a dialog box, not constantly displayed in the main interface.(2)
Contents of Consent Form
The below was read and signed by all three participants in our user testing:
CONSENT FORM THE ANOTOTATORS INTRODUCTION The purpose of this consent form is 1) to inform you of our agenda and intentions, and 2) to minimize the possibility of coercion from anyone. YOU You are asked to participate in our study because we believe you have experience pertaining to a field that could help us design a better user interface for you and your peers in the field. WE We are four University of California, Berkeley students doing a class project. The goal of our project is to create a user interface that will aid practitioners in the field of medical imaging and research. TODAY The purpose of today's particular study/interview is to provide us with usage data regarding the interface we are designing. DESCRIPTION OF PROCEDURE You will be presented with a crude prototype of our user interface. You will then be asked to perform a series of tasks using said interface. We will encourage you to share your thought process and perceptions as you perform the tasks we request. The entire procedure will take no more than an hour. You may withdraw yourself from the procedure at any point in time. HARM/BENEFITS FROM THIS PROCEDURE For better or worse, you may gain some insight about the designs of user interfaces used in your practice. Other than that, there should be no harm to you in a physical, psychological, or social sense. Everything we will do today is common practice and no part of the procedure bears any risk of negative effects. Thus, we are confident that there will be no unexpected or undesirable side effects. CONFIDENTIALITY No one will acquire your contact info beyond those of us who already have your email address (We, the four students and Google, because we use GMail). Professionally, The Anototators Robert Held Gene Zhang Edward Karuna Anirudh "Ani" Vemprala Sign below to acknowledge that you have read the above: __________________________________ Date:_____________
Gene - Oct 11, 2006 12:58:26 pm
AWESOME WORK, GUYS!!!
Gene - Oct 11, 2006 01:01:24 pm
OK I'M PRINTING IT
