ContextualInquiry-Group:ET
From CS160 User Interfaces Fa06
Contents |
Contributors
Anthony Franco
- Interview 3 preparation and analysis.
- Interface Design
Antonis Mannaris
- Interview with customer 3
- Overview
- Contextual Inquiry
- Interface Design
Joe Hart
- Setup group wiki page
- Interviewed Customer 1
- Problem and Solution Overview
- Portions of Contextual Inquiry
- Analysis of tasks
- Task analysis questions
Qingyun Tang
- Interviewed Customer 2
- Problem and Solution Overview
- Contextual Inquiry
- Analysis of Tasks
- Analysis of Approach
Target Users
Describe the rationale behind your choice of target users. For each of the three (3) customers, give some details of their background, their likes/dislikes and priorities. Avoid information that may reveal their identity.
Customer 1: High School Chemistry Teacher
Customer 1, who we will call Jane, is a 23 year veteran high school chemistry teacher. She was born in the United States with English as a first language. Her personal education was completed though college and she has obtained her masters degree in plant genetics. Jane main value is connecting with her students and engaging them in the learning process. Important in this process is the student / teacher feedback loop on assignments, testing, and day to day interacting. Parent involvement is also greatly important to a student's success. Jane is often complaining about how time demanding teaching is and wishes she had more tools to lessen the this burden. Although she is computer literate she would not claim mastery and sometimes struggles to learn new programs and processes. Once a program/process is understood she is quite comfortable using the computer as an effective tool.
Customer 2: College Student
Customer 2, who we will call Joe, is a 19 year old college Student. He was born in China with Mandarin as his first language. He finished his high school in San Diego and he is attending UC Berkeley at the moment. His major is Industrial Engineering and Operations Research. Joe doesn't like exams, just like most students. However, Joe does feel that it is important to have tests back faster to review the mistakes made on the tests. Although Joe is not very much into computers, he feels comfortable using them. He mainly uses his computer to do school work, never done any type of programming. He doesn't have any difficulty to learn to use new software as long as the interface is not too hard to operate with.
Customer 3: College GSI
Customer 3, Theo, is a GSI/TA for a MicroEconomics class at UC Berkeley. He has been in higher education for almost 10 years and has significant experience as a GSI. He is currently finishing up his PhD in Economics. Theo, through his long academic experience, is comfortable with computers and he uses a variety of software tools. He is quite able to quickly adapt to new software and he strongly believes that computers are increasingly becoming an essential tool in Academia. As a GSI now and in the past, he has been responsible mainly for fascilitating and grading exams and homeworks. On occasions, he also took part in the preparation process of tests.
Problem and Solution Overview
This overview should be a concise statement of the problem you are tackling and a brief synopsis of your proposed solution.
The demands on the modern intructor (teacher, professor etc) are many and difficult. A wikipedia article, current issues in teaching, paints a picture of the varied demands for a teacher's time. A typical teacher has 5 classes a day with ~30 students in each class totaling ~150 students to: administer testing to; evaluate performance; keep records of a students performance. Our project is aimed at lessening the amount of time a teacher spends doing the "grunt" work of grading and recording grades. Through the use of an Anoto pen/paper interface we can help by streamlining the grading and digital recording of grades allowing teachers to focus more on other areas. Our design will also allow instructors to quickly "digitize" their tests, in particular store an answer sheet, distribute and give feedback on the tests, as well as easily and quickly compute useful statistics on the results.
The demands on college students vary depending on the classes they take. A typical full time college student takes four to five classes every semester. Each class has an average of one to two midterms and one final. Our project tries to observe how students write on the tests in order to develop digitalized tests that are easy for grading and reduce the amount of time teachers spend to grade and record grades, thus return to students at a faster speed.
Contextual Inquiry - Interview Descriptions
Describe the process you followed when conducting the interviews, and environment where you observed their work. Identify tasks and themes that the customers shared in common in their work practices. Then, note anything unique about each interview and comment on the rationale behind these events.
We used the apprentice / master paradigm for our contextual inquiries. We focused on learning the test cycle to gain a deeper understanding of what all goes into designing, administering, and grading a test.
Main most common elements of the test cycle:
- 1. Test creation
- Create test using Microsoft Word, Excel, or similar editing software
- Use tests from the previous year
- Use tests provided by text book manufacturer
- Test design more an art than science
- Create test before teaching section then teach to the test
- 2. Test administering
- 2a. Cheating detection/avoidence
- Teachers make multiple versions of a test
- Compare completed tests for similarities
- Physical teacher presence in room
- Physically separate students
- Timer
- Identifying students / attendance
- 3. Grading environment setup
- Environment
- Large desk at home
- Large desk at school
- Commuter train
- Use of an answer key
- Compiling different parts of a test to compute the whole grade
- 3a. Tools
- Red pen
- Grading key
- Printed verion of gradebook
- Calculator
- Computer version of gradebook
- Use of technology to ease the grading process (scantron)
- 4. Feedback to students and/or parents
- School website
- Written comments on test
- Email home to parents
- 5. Records
- What type of records are kept (per semester, class etc)
- How the results are stored
- Type of statistics on the results
Customer 1: Jane
I visited this customer at their home over a weekend to observe them grading a test given the previous week. When I arrived Jane was about mid-way through the grading of a multiple choice test which also contained some short answer and math portions. She had developed an interesting approach to grading her multiple choice exam. Each student bubbled in an answer on an answer sheet similar to the more well known scantron. She then constructed the answer key by punching holes in one of the same answer sheets used by the students. When the key is placed over the student answer sheet it is clear if the student has marked any of the correct answers by looking through the holes in the key. A pictoral of this process is here. This allows her to grade multiple choice tests quickly and efficiently. This also allows her to easily hand off grading to a TA with minimal instruction. She has also noticed that this method generates virtually no mistakes. The only problem she has is the generation of the key is a labor intensive process taking up to ~90min.
Some specific tools this teacher used:
- Gradebook tasks
- Statistics
- Seating charts
Customer 2: Joe
I visited this customer at his home to observe him doing a test. I made the test that it contains multiple choice questions, short answer questions, math questions, and essay questions. These are the common types of questions instructors will have on their exams. So my goal was to observe how the customer uses a pen to work on the exam instead of pencil we usually use. First, for multiple choice questions, Joe circled the entire answer, including the choice letter and the content of the choice. He found out that his answer was incorrect later during checking, so he crossed out the circle, and circle the other answer he thought was correct. I asked him what if that answer was wrong but his first choice was correct. Joe crossed out the new circle, and put the correct choice letter on the side of the problem. For short answer questions and essay questions, whenever Joe made a mistake, he had to cross out the incorrect part and rewrite the rest. This was not as convenient as pencil. In addition, this process would require much more spaces than a penciled test. In the math questions part, I noticed that Joe overwrote the number he wrote earlier to change it to another number. For example, in a matrix problem, Joe had 1 in a place, which later changed to -2, with the existing 1 still there under the new -2. It happened several times, so this is a noticable problem as well.
Customer 3: Theo
We met with Theo at a coffee shop where he often grades exams for his class. Even though he didn't have any grading to do currently, he was kind enough to bring past papers and walk me through the process. For a typical Economics class, including the one he is teaching this semester, an exam is comprised of a multiple choice part and an essay part. The latter is graded by hand and naturally takes up most of the time. For the former, the class will use a scantron system to automatically grade each paper. Naturally, I wanted to know more about this system. Theo described this as an "old technology" since it has been used for quite a while. All the answer sheets from the exam are fed one by one into a scantron machine which holds an answer key in its memory. The latter is the only memory the machine has. The output is the number of correct answers, which is then recorded manually into a computer. This presents two main problems. One is that it is time consuming to carry the papers to the scnatron machine and feed them in one bye one, recording the result for the particular student each time (the class has ~160 students). The second problem is that the manual tasks involved are prone to human error. I also asked Theo about the day of the exam. He told me that because of the large number of students, it is not practical to verify their identity with a student ID. It is also very problematic if a student claims his exam was lost, since there is no proof either way. So the stuff relies on Honor code and face recognition. During the exam the 2 GSI's and the professor for the class are present in the room. They supervise to prevent cheating and to make sure the time limit is honored. Naturally, due to the large number of students, the latter is challenging and not perfect. Finally, we talked about feedback. The exams are returned to the students during discussion section and the stuff keeps records of results for the whole semester. Simple statistics are computed (average, s.d) and the students are informed. For most classes Theo has taught, there is no online access of individual results.
Task Analysis Questions (5 pts)
Answer the 11 task analysis questions. Use examples from your interviews when applicable.
- 1. Who is going to use system?
- High school teachers
- High school students
- Possibly higher education instructors and students as well
- 2. What tasks do they now preform.
- Test creation
- Creating the test to be taken by students
- Creating the Answer Key for grading the test
- Test administering
- Grading
- Grade entry
- Student feedback
- Students take test
- Test creation
- 3. What tasks are desired?
- Old
- Hand processed grading
- Grade recorded on student answer sheet
- Grade recorded on printed grade sheet
- Grade recorded in Computer
- New
- Partially autograded
- Grade recorded on student answer sheet
- No printed grade sheet required
- Grade auto entered into computer
- Old
- 4. How are the tasks learned?
- Preknowledge: User needs to know how a general grading system works
- Training Needed for our system
- Manual with complete walk through of system for the most novel of tasks
- Manual or cheat sheet describing gestures etc. used on regular basis
- Use website to have all of this information for download or display.
- 5. Where are the tasks performed?
- School environment, classroom
- Home environment
- Commuter train, carpool
- 6. What is the relationship between user and data?
- Private data
- test answer key marks
- grade books
- test information
- student grade information associated with name
- student marks during and after test
- Public data
- grade stats with no identification of individuals
- Private data
- 7. What other tools does the user have?
- Laptop
- pen
- paper
- scantron machine
- copier
- hole punch
- computer printer
- cell phone
- calculator
- blackboard
- digital timer
- Jumpdrive for gradebook tracking and for test documents... etc... data storage and portability.
- 8. How do users communicate with each other?
- Marks on paper/ comments connect student and teacher
- School website connects students, parents, and teachers
- 9. How often are the tasks performed?
- Min once per week... max 5 per week a test is given.
- 10. What are the time constraints on the task?
- Students want feedback immeadiately
- Closer feedback to assesment, the more valuable the feedback
- Test = 1-2 day acceptable
- 11. What happens when things go wrong?
- If tests are lost, then the test must be readministered.
- If the Anoto auto-grading goes completely wrong or does not work the the teacher can fail back to the manual grading of tests.
- If the Anoto auto-grading incorrectly grades random student tests the error is harder to detect. This can only be detected through student regrade requests on a individual to individual type basis. The paper copy with the student pen marks should be referenced for verification of Anoto faults.
Analysis of Tasks (10 pts)
Choose 6 tasks (2 easy, 2 moderate, 2 difficult tasks) and describe them. These should be real world tasks that have details (e.g., note-taking in a lecture).
- 2 Easy Tasks
- Printing out test on Anoto paper. A teacher simply prints out existing tests on Anoto paper for students to take test on. Since Anoto paper doesn't cost much more expensive than regular paper, and teacher can write out the test using Anoto pen, this task should be fairly simple.
- Student taking test and marking answers. The students mark their answers on the test sheet using Anoto pens following the guidelines specified by the teacher. The only concern here is that the students must follow the guideline, otherwise the auto-grade parts will not be graded correctly.
- 2 Moderate Tasks
- Matching answer keys with tests taken. The teacher must select a marked answer key and correctly match it with test given in order to take advantage of the auto-grading features of the system. It is more scantron typed, with a few exceptions that some answers will be OCRed to check the correctness.
- Viewing statistic results from graded tests. Once the tests have been graded the teacher can lookup statistical information about the student results. The scores of the tests will be OCRed, thus enables the teacher to view the distribution of the scores clearly and decide what the curve is quickly.
- 2 Difficult Tasks
- Creating answer key for test. The teacher must mark a test printed on Anoto paper using predefined gestures the system has predefined. This becomes the answer key that will then be compared against the student tests. The answer key must fit all situations that students will mark on the tests to avoid wrongly graded tests.
- Managing grading problems. If there is a problem with the auto-grading of the student tests, the teacher must first detect there has been a problem and then either manually grade tests or fix a problem in the answer key by remaking it.
Interface Design (20 pts)
Give a rationale for your design ideas. This section should clearly indicate the functionality of our artifact and what the user interface will be like (described and sketched -- references the figures in your text). You should then describe three scenarios of how someone would use it to accomplish three of the tasks above. Scenarios include the steps customers will go through to accomplish the task. You should include "storyboards" of the sequences described in your three scenarios. The solution section should be the bulk of the write-up and take several pages.
Test Creation
The main computer interface of our proposed system will be the test creation. To be more accurate, tests will not be created from scratch but rather teachers will have the ability to import their existing tests and set up a version of those tests using our system. We feel that this will make the transition from the current test creation habit more flexible and much more desirable from the teacher's prespective. As a first step the user will be presented with a "New Test" window (Figure 1). Here we will provide general instructions that will hopefully inform the user (with no ambiguity) of what her next step should be. If the user has a computer document of the test they can indicate the file here. This will provide us with the ability to extract more data for the test like individual questions. Alternatively they can choose not to include the document, in which case only the answer key and other general information about the test will be saved. (NOTE: Depending on implementation details, the former option may not be present in our final product). For multiple page tests, we need to distinguish between the pages. To make the new system as error-proof as possible for the students, we decided to include the requirement that each page be printed on a unique pattern of Anoto paper. This is prefered to the alternative of the student having to somehow indicate which page she is currently working on for two reasons. One is that it will be easy for a student to forget or misunderstand the instructions on how to do this, leading to a big mess when the answer strokes are collected. A second, less important reason is that we believe the majority of uses for the system will be for multiple choice questions that have a single answer sheet, so requiring unique patterns will not be too expensive. The user will then be asked to provide the general details for the test (Figure 2). Out of these, the required fields are the time and date that the test will be taken and the duration of the test. These will be the primary means by which we identify Anoto pen input with a particular exam. Also required will be the number of pages of the test. One other useful piece of input here will be a class with which a roster of students will be associated. The association is a separate part of the system. Other details may include subject, grading options etc. Then the user will be guided through a series of steps which will map out the test and the answer key (see scenario solution below). This will complete the creation of a test. The test data will be stored for future usage and the test prepared for the day of the exam. The user will be instructed to print the test on ANOTO paper and make sure that the pattern on the paper matches the pattern on the paper she just used.
Using the pen annotations made by the user, we will be able to divide the test into questions. Some of those questions will also have a correct answer associated which we can use to autograde the test. For this we will need to process the location of the annotations to some detail.
Taking the Exam
At the time of the test creation, an attendance sheet may be printed out for the day of the exam. Alternatively, a permanent attendance sheet may be used. In either case, before the exam starts, each student will grab an Anoto Pen and put a check next to their name on this sheet. Using this we will associate a pen id to each student, that will enable the system to associate a test submission to each student. The advantage of this is that there is no need for a student to buy a pen. Instead a set of pens owned by the school may be used from all the students. Note that our method for test creation imposes no limitations on the outline or format of the questions. In particular, it allows the use of a single multiple choice answer sheet which is the norm for most multiple choice exams. By marking the width and height of a question (in essence we will have a rectangle that will be the "question space") we can easily determine which answer goes to which question. As far as the students are concerned, only two details will be different as to how they take an exam. Our instructions must make it clear that only their LAST SUBMISSION will be graded for multiple choice questions, and no pen erasers! Even though all their markings will be collected, the last submission principle will solve any ambiguity when more than one answer is marked. As long as this last submission covers any of the correct answer area, the question will be marked correct. For other question types that need to be manually graded we will simply collect all the strokes the student has made.
Completing the grading
The teacher will then collect the papers and complete (if needed) the grading. Since at this point there is no automatic way to detect which student an exam belongs to (unless we use unique paper patterns for each copy), the teacher will need to indicate to the system which student paper is being graded. We realize that this may not always be desirable. For that reason, we will provide the alternative to simply manually input an overall grade (Figure 4). This is something all instructors have to do today so there will be no additional burden imposed. As the figure shows, the screen will include the grade of autogradable questions. The teacher will then be able to add the manual grading part or simply the overall grade directly.
Test Statistics and records
Computing the average and standard deviation of a set of scores is not a simple problem to instructors. Assuming the tests scores are already in the computer, she would have to learn to use a new program, such as Excel, to get the desired information. If the test scores are not already in an electronic form, she will have to input them manually which is both time-consuming and error-prone. Our system will simplify this step in the grading process by presenting to the user options for viewing the various statistics. We will also be able to present data on various other pivot points that otherwise would be too complicated or time-consuming to compute. For example, selecting a student could display a graph showing the student's performance against the mean throughout the semester.
Task Scenarios
Below is a step by step solution to three of the scenarios mentioned above.
- Student taking test and marking answers
A test has both multiple choice questions and short answer questions. The former is autograded so there are slightly stricter guidelines as to how they should be answered, as we described above. As far as the student is concerned, they just need to make sure that they follow guidelines similar to those followed on any multiple choice test. They also need to know that once the time is up, anything they write after that will be ignored. Other than that, test taking will not change in any way.
Storyboard
- Viewing statistic results from graded tests
Suppose the teacher wishes to see a statistics on some of the tests completed so far. The system will allow him to choose which tests to include, and we will list the classes that have taken all the tests selected. The teacher will be able to select all or some of those classes (Figure 5). The figure outlines the interface. To compute the results, our system will need to gather the appropriate data from the database.
Storyboard
- Creating answer key for test
The answer key will most likely be created at the time of the test creation. This will enable the system to process the students' submissions over a much longer period than if the answer key was entered at a later stage. We now pick up the test creation process after the general details of a test have been input. For each page, the user will be asked to first indicate the question space. Namely, she will use the Anoto pen (the test needs to be printed on Anoto paper) to draw two corners around each question and the answer space for that question (Figure 3). There is no real need for a "corner" shape here, but it is more natural to the user than a simple mark. Our system will take into account the leftmost, rightmost, top and bottom markings to create a virtual rectangle around the question. Note that this requires a streaming input from the pen to distinguish between the questions. The alternative of analyzing the mark locations and perhaps shapes is much more difficult to implement, especially for the tight spaces of a multiple choice answer sheet. Before moving on to the next page, the user will be asked to fully shade the correct answers of any multiple choice questions. (NOTE: If time permits, we may also allow the user to write the correct answers to one word or one number questions, and analyze those with OCR). The question space answer key will then be stored into the database and the necessary associations made. Once the user indicates that she is done, we will display a summary of number of questions and which questions are part of the answer sheet. If the data is correct, the user will confirm and the test will be finalized.
Storyboard
Solution Sketch
(As presented in initial proposal/brainstorm)
Description
- 1. The teacher is able to print out existing tests on special Anoto paper. Using their Anoto pen they can mark answer regions and correct answer selections by using special predefined gestures.
- 2. The teacher wiill print out the student tests on special Anoto paper. The students take the tests with Anoto pens, marking correct answers and scratching out incorrect answers. The backend will keep track of latest mark info and analyze marks to ensure the students intended answer. If there is a question the hardcopy can be pulled for verification.
- 3. The student tests are compared against the teacher's answer key to determine a grade.
- 4. Student's grades are autoentered into a gradebook and statistics can be made available such as:
- Mean
- Std Dev
- Curve Information
- Estimated time spent on problem
- Difficulty of problems (number of students getting same problem wrong)
- etc...
Analysis of Approach (5 pts)
Explain how your application takes advantage of the affordances of the Anoto system. If there are other technologies such as TabletPCs, PDAs, SmartBoards, etc. that could be used to perform the tasks your application addresses, you should explain why using the Anoto system is a better approach. List the pros and cons of your approach.
We believe that the proposed system makes use of two major advantages of the ANOTO technology. The first is the obvious answer that schools use pens! Unlike a fully computerized system, Anoto allows the users to use an interface (pen/paper) which are very well trained in and used to (and it is cheaper). In our case, this mainly applies to students taking a test, and teachers grading it.
The second major advantage is that the system provides a smooth transition into a more efficient way to prepare, take, grade and store exams. In particular, teachers will be able to use old exams - even hand written exams - to create electronic versions of those exams.
Also, Anoto will enable the autograding of certain questions, while at the same time allowing hand grading (also with an Anoto Pen) and merging the results as needed. We also feel that the use of the Anoto pen leaves great room for improvement. As OCR improves we could add features like signature recognition, conversion of written text to computer text for spelling and automatic grading, etc.
Probably TabletPC could perform the same task to allow users to work directly on the material. However, TabletPCs don't have the advantage of using paper, the familiar interface we have, to perform the task. Furthermore, TabletPCs are more expensive than Anoto Pens, which could be a bigger financial problem. Anoto system uses a more user-friendly interface than TabletPCs.
In general, the use of the Anoto System for exams will (to some extend) remove the manual labor involved in grading and processing test results (while at the same time provide many of the advantages of a fully electronic system) without the need to change the current way an exam is prepared and taken.
Pros:
- Weight and size are more affordance than TabletPCs
- Greater durability than tabletPCs
- Easy to use, more friendly user interface
- Reduce teachers' work and save their time
- Greater education tools
Cons:
- Requires foresight. Teachers need to find all possibilities how students will mark the answers
- Students need to pay attentions to guidelines on the tests besides paying attention just to questions
- More works are created if tests are wrongly auto-graded
