PilotStudy-Group:ET
From CS160 User Interfaces Fa06
Introduction (5 points)
Introduce the system being evaluated (1 paragraph)
At Electronic Test (ET) we have built a system to assist high school teachers in completing a critical function in their daily jobs: test administation. The ET system takes any existing test file a teacher has already created and turns it into a time saving tool. Through the affordances provided by the Anoto system a teacher can create an Answer Key to a test in much the same way they currently do and never manually grade a student's test again. The ET system compares the Answer Key created by the teacher and compares it against the student test to determine if the student made the correct marks, auto enters the grades into a grade book, and then provides powerful statistical analysis right at the teacher's fingertips. With the ET system teachers have more time for teaching, one-on-one student interaction, and their lives outside of class.
State the purpose and rationale of the experiment (1 paragraph)
Now that we have a working prototype of our ET system, we would like to get user feedback on the dynamics of the user interface. How long does a user sit at each screen before understanding what to do? Are there times when the user does not understand what to do? Are there times when the user makes mistakes? Does the user interface allow them to correct them? By observing the habits of a user as they navigate through our user interface we can get a better idea of places where more thought needs to be put in and where features are getting in the way of the user. Actually having the user interface in front of users instead of the paper prototype gives us more insight into the actual use habits of the user and their interaction with our system.
Implementation and Improvements(15 points)
General Look and Feel
- The overall look and feel has been enhanced with nice uniform image icons for buttons and text. This provides a feeling that each section of the application is unified and complete.
Import Test
- Major improvements have been made to this portion of the application. The user starts off by selecting any pdf file off of their hard drive to start the importing process. The next step is the answer key generation step. This launches an interface containing a scaled image of the pdf chosen by the user along with a status message telling the user the next step to take: draw a box around an answer region for question #1. For every mark the user makes on the test with the anoto pen the mark is reflected in the interface. Also after every mark the status message is updated with the next task for the user to preform. After the user is done creating the answer key the interface confirms with the user the number of questions and answers created in the previous step. Then the user saves the data as the last step in this process. This creates test file with an associated anoto pattern overlay to be used in for student tests along with an XML file containing the answer key strokes information later used to compare with student marks. All files are saved in the directory where the original file was picked from. This may need to be changed in the future to put all files imported and associated into our own application directory for easier tracking.
- Fixed minor bug: Whenever a user left the import test section of the interface and returned later, the interface would be how they last left it and not started over from the beginning. This has been changed so that an exit from the import test section causes the interface to be reset.
Class Setup
- This module allows the user to manipulate class rosters.
- A list of classes is presented along with the option to create a new class. For a new class, we require that the name is not already used. The user can also delete existing classes.
- The user selects a class and the details of the class roster are shown in a table form on the same page. The table lists student ids and student names for all the students in the class. The user can edit all the information and also add more students to the class. If they decide to save the changes they made, the data is written to disk and is available later. If the user attempts to leave while the current roster has modified data, an appropriate popup message appears to verify the action.
Browse
- The browse module gives the user the ability to go through the existing tests he or she has created. This will be presented in the form of a list, with several sorting options included.
Print Test
- The module for printing a test has been improved from the previous prototype. The user first selects a file and then clicks on "Print". This opens up a print dialog to the computer's default printer and all the usual printing options.
- Fixed minor bug: Whenever a user left the print test section of the interface and returned later, the interface would be how they last left it and not started over from the beginning. This has been changed so that an accidental exit from the print test section causes the interface to be reset.
Method (10 points)
Participants (who -- demographics -- and how were they selected) (1/2 page)
- Customer #1
- Jane participated in our Contextual Inquiry. Jane is a high school chemistry teacher with 6 classes of standard chemistry. She is a 30 year veteran with a varied background teaching in many school districts and teaching many different subjects. She currently uses a computer based program to track students and their grades in her classes. We thought she could provide good feedback for our application since she is familiar with computer based "gradebooks" and already understands what works best for her and what she has come to expect from new systems.
- Customer #2
- Jack is a beginning teacher at the high school level. This is his 3rd year teaching science and math subjects. He is rather computer savy but is far from a programmer, we can say he is computer literate. We chose Jack because A) He was available and agreed to our study and B) He is probably most representative of the future teaching workforce. Jack's feedback could potentially help us to see where this technology should be in a few years.
- Customer #3
- A college student outside of the CS department. This customer was primarily available and willing to do the study. This person is computer literate but not a self described expert so we thought this person could also help point places where a novice user could run into problems.
Apparatus (describe the equipment you used and where) (1 paragraph)
A laptop was taken to each interview with our application installed and ready to go. Since we were having problems getting good print outs of user tests (especially when at a user site) we opted to go with the Meade notebook as the pattern used on each imported test. This meant that the user could write on the notebook and see the marks scaled and represented in the user interface but it was difficult for them to accurately define regions and answers because of this non one-to-one correspondence. In a real environment we expect the user to have a printer of high caliber.
Tasks (1/2 page) [you should have this already from previous assignments, but you may wish to revise it] describe each task and what you looked for when those tasks were performed
- Easy Task
The objective is to print a prepared test for administration.
The user will select the "Print" button from the main menu. She will then be prompted to select the test she wishes to print. Assuming this is a valid prepared test, the test will be printed for as many copies as the user requests on Anoto paper.
- Medium Task - Import Test
The objective is to import a new test and create the answer key for it.
The user will click the "Import Test" button from the main menu. She will then be presented with a window containing 4 steps she needs to perform. The user will have the ability to move forward or backward between the steps. The first step is to select a file that is either the original test document. The second step is to create an answer key for the test. A hard copy will be printed on Anoto Paper and also a pdf file of the test will be displayed. The user will be instructed to mark a question area, answer area and optionally correct answer using an Anoto pen for all the questions. Once she is done she will proceed to step 3, which is to review and verify the answer key the system created. Finally, step 4 will save the test data and return to the main menu.
- Hard Task - Manual Grading
The objective is to complete the grading of a test that has been administered.
The user will click the "Manual Grading" button from the main menu and will be prompted to select the test she wishes to grade. Assuming this is a valid test and all the necessary data for it can be read, we will then display a table for the grades. In the table there will be a row for each student that took the test (with his name) and a column for each question. An individual table cell will represent the grade for a specific student and a specific question. There will also be a column (not editable) for the total test grade of each student. Questions that do not have a grade associated with them yet will be represented as empty table cells.
The user will be asked to complete the grading of the test (in any order she prefers), save the grades and return to the main menu. We believe that the layout of the window makes this task conceptually easy, so the user is unlikely to be confused. For that rare occasion, the user can click the help button to view instructions on grading.
This task would be monumentally difficult if not for a picture under the table. Initially the picture is a line saying "<select a specific question (table cell) to view the respective student response>". As soon as the user selects a table cell that represents a (student,question) tuple, the picture changes and the student's response to the particular question is displayed. So the user does not need to move back and forth between the hard copy of the tests and the computer to decide what grade to award.
Procedure (1 page) describe what you did and how
A laptop with our application was brought to each user site along with the pen and a Meade notebook. The user was then described what our application was and what is will be used for. We then described this particular procedure and had them read and sign the Consent Form. Once the environment was setup for the user we described the first task they were required to complete: Import a test and return to the main menu for further instructions. Second we had them walk through the Print Test task and return to the main menu. Lastly, we had them manually grade some scores, save their progress and return back to the main menu. We then had each user run through the tasks again in any order they wanted to see what learning had been achieved from the first set of task completions. Finally, we had them do it again.
We tried to provide as minimal help as possible to the user while preforming each task. If a user was really stuck or getting frustrated we would then provide some assistance to get them back on track.
The Answer Key creation step did require some hand holding as we elected to use the Meade notebook as the simulated print out of the answer key itself. We had to describe the paradigm to them and tell them how it would work in a real environment where an adequate printer was available to print out Anoto patterns. All of our users didn't seem to mind working on the Meade notebook but would have rather had the real test with pattern.
Test Measures (5 points)
Describe what you measured and why (1/2 page)
Now that we have a working prototype of our ET system, we would like to get user feedback on the dynamics of the user interface. How long does a user sit at each screen before understanding what to do? Are there times when the user does not understand what to do? Are there times when the user makes mistakes? Does the user interface allow them to correct them? By observing the habits of a user as they navigate through our user interface we can get a better idea of places where more thought needs to be put in and where features are getting in the way of the user. Actually having the user interface in front of users instead of the paper prototype gives us more insight into the actual use habits of the user and their interaction with our system.
- We paid attention and took notes about the Character of the user Talking Out Loud.
- Comments that were favorable like, "Oh I like that."
- Comments that were unfavorable like, "This is awkward."
- Comments that were confused like, "What do I do now?"
- Time spent completing each task
- Some inaccurate measurements were taken on the time taken per task. A watch was used and times recorded. These times were then compared against each other from different users.
- Asked questions by the interviewers
- Did you understand the tasks given to you?
- How difficult you you rate the experience?
- Would you use this application on a daily basis?
- What was the most unclear task given to you?
- Rate this application on a scale of 1-5.
Results (10 points)
Results of the tests (1 page)
General comments
- This and other areas of the system use a wizard style interface to walk a user through the necessary steps needed to complete the task of importing a test. The way this wizard has been implemented a user is presented with a highlighted first step and grayed out future steps. The current step is an actual button the user must press in order to actually start that step's function. This is a little counter intuitive and breaks the normal wizard interface. Users were having problems understanding if there even was something they were supposed to do or if something was going to happen. They then didn't understand if they were to press the button or press the next button to continue. While we have the next and back buttons available for the user to navigate through the steps, it isn't clear they add any real functionality. Maybe replacing these buttons with a cancel button would be better. This would allow a user to cancel if they felt there was a problem while clearing up some confusion stated above. This also helps to reduce the mental load on the user trying to understand what every function/button does on the screen before being able to continue.
Import Test
- The main issue here is in the answer key creation phase. The current system alternates between asking the user to draw a box around an answer region and then mark the correct answer within that region. While this works for multiple choice type tests, it does not allow for the following: answer regions with multiple correct answers, answer regions without correct answers (essay for example), incorrectly marked answer regions, incorrectly marked correct answers. These features will need to be understood and added for future releases.
Print Test
- The user needs less options when choosing a test to print. The user does not know which file they should choose and from where they should choose it on the file system. Possible solution: limit the directories viewable by the user and restrict the types of files that can be seen to only imported tests.
Class Setup
- Users did not like needing to add student ID's. While this is necessary information to track with student records we should put in an auto incrementing field for uniqueness purposes and add a student ID as an optional field for each student added. Users would have also liked the ability to sort their created classes instead of them being in order of entry.
- Creating the roster for each class is tedious since many classes have the same students. It would be nice to be able to copy/paste students between classes and also import rosters from an xml file.
Manual Grading
- This area was the least flushed out since the interactive prototype. We are still struggling with how to show a student stroke information over the actual image of the test taken. In order to get this area completely working an extensive backend database will need to be in place to support it. We may scratch some of these features and put into place basic editing of already auto-graded questions. However, when fully described to customers the reaction was quite positive. This feature will be the icing once all other portions of the interface are implemented.
Discussion (15 points)
What We Learned From The Pilot Run
From the three pilot experiment sessions, we learned that even though we thought most part of our interface was relatively easy to intuitively understand, there were still many places that could be improved or revised. The most common complaints from the participants focused on the wizard style interface. Our wizard interface was a little different than the normal wizard interface. Participants needed to click on the icons in order to perform the job, but they did not know what to do because intuitively, wizard mode should pop up things by itself. In addition, each participant's comments reflected on their habits and tastes, as one said printing was too complicated while another thought printing could have more functionality. We learned that different users have difference preferences. It would be rather hard to satisfy all users’ need, but it should be practicable to satisfy most users’ needs.
What We Might Change For The "Real" Experiment
For real experiment, we would have a much larger testing group. In this case, we would make these changes to our experiment:
- Based on the difficulties our participants had with the wizard mode, a notice or special instruction could be added.
- Better directions would be provided to the user regarding create active region on the paper UI with streaming.
- We might want to have xml import function so that our big participants pool won’t waste time on entering ID numbers and names for students.
What We Might Change In The Interface From These Results Alone
- Change the cursor when it is moved over icons so users know they are clickable.
- Add more instructions/help buttons if users don’t know what to do.
- Make creation of rosters faster, such as the ability of copying/pasting or importing xml files.
- Add instructions on pen gestures.
- Eliminate options for users when choosing test to print. We could limit either the filetype or folders users could view.
- Optimize the database for manual grading.
- Some clearer direction needs to be implemented for importing.
Workload breakdown (5 points)
- Percentage and nature of contribution from each group member (1 page)
- Joe
- Added pen functionality to import test answer key creation
- Added ability for pen input to be on Meade notebook
- Portions of report
- Interview with high school teacher in front of working prototype
- Antonis
- Implemented Class Setup module
- Class data files are dynamically created, deleted and modified on disk
- Portions of report
- Interviews
- Qingyun
- Implemented Print module, Browse module
- Icons Modification
- Portions of report
- Interviews
- Anthony
- Parts of class setup
Appendices (5 points)
Consent Form
Problem Description
- An overview of the problem our design is set out to solve was discussed with the users. This was done in the form of a free discussion and our purpose was to convince the users and make them think of how the issue of time and tests applies to them.
Proposed Solution
- Once the problem was established we gave an overview of our proposed solution. These are the main points we covered:
- The use of existing computerized tests in our system.
- Autograding multiple choice questions
- Easy storage of test data (tests, student responses, scores)
- Easy grading of tests without the need to switch between the hard copies and the computer
- Automatic generation of statistics
- Class rosters
Quick Demo
- We then went over a really quick demo of the three tasks we wanted the user to complete. On the most part, we tried not to perform the "difficult" parts of the task, but instead briefly explained how they could be done. We then handed these basic instructions to the user.
- Import Test
- From the Main Menu, click on Import Test.
- Locate a pdf file that represents one of your existing tests. If you do not have one, select exam_chem.pdf under the data/ directory.
- Create the answer key by following the on screen instructions.
- Review and save the test.
- Print Test
- Print 30 copies of the test you just imported.
- Grading
- Complete the grading of a test by selecting "Manual Grading"
- After you select the test file "test2GradeData.txt", review and complete the grades for all questions and all the students
- Save the grades and return to the main menu
- Import Test
Raw data - Notes taken during the usability study.
- Step by step wizard needs better instructions on the high level
- Need feedback that each step was successfully completed
- Answer key creation needs to allow "Undo" actions. Also need to improve functionality for non-multiple choice questions.
- When printing a test, user was not sure which files were valid tests
- For class setup, requiring student ids is not wanted
- Better sorting for roster
- Alphabetically sort classes
- Would be nice to have the ability to copy/paste set of students and also to import a roster from an xml file
Main Screen. |
Import Screen. |
Class Screen. |
