ContextualInquiry-Group:BigBrother
From CS160 User Interfaces Fa06
Member Contributions
This group met several times both online and in person to collectively discuss the tasks and requirements of this application. Once we had worked all content, we split up the work of writing up each section and providing the appropriate diagrams.
Randy Hilarbo
Randy provided descriptions of professor and GSI target users, conducted interviews with Professor 1 and GSI 1, wrote up the Analysis of Tasks section, and provided the interface description and diagram for the professor interface.
Charles Leung
Charles contributed to the GSI user profile, interviewed GSI 2 and did a writeup for it, and answered the 11 Task Analysis questions.
Leo Chen
Leo wrote the Problem Description and Solution Overview, and provided the images and written description for the GSI/TA user interface.
Jonathan Chang
Jonathan wrote the Student user profile, interviewed two students and provided writeups, provided the student interface writeup and diagram, wrote the Analysis of Approach, and organized everyone's work on the wiki.
Target Users
While this application is meant to serve the university as a whole, users can be divided into three groups by the tasks they would use the application to accomplish.
Johnny Deng
Johnny is an undergraduate student in the College of Engineering, currently in his second year at Cal. He is taking an average courseload of 16 units, and considers himself reasonably proficient with computers and other technology. Two of his classes, Math 53 and Chemistry 1A, are the type of large classes this application is meant to address. He can recall several times during past exams when the size of the class prevented GSIs from addressing questions in a timely fashion. He has once observed cheating during an exam, though he has never done it himself.
Jerry Sun
Jerry is a graduate student instructing two discussion sections of Professor Yu's large class. He is originally from Europe and came to UC Berkeley to do his Ph.D. This is his third semester teaching as a GSI. He has about 50 students. He is also taking a total of 8 units of graduate course works. He does not consider himself an expert computer user but he does use spreadsheets and word in his teaching. There's really nothing that he likes about giving out exams. Exam days are the busiest for him because it usually takes him 10 hours to grade the exams. He also gets bored while monitoring a 3-hour exam. His priority is his studies. When it comes to giving out exams, his priorities include consistently grading the exams. He is only willing to adopt new technology if he believes that it will make his job easier.
David Yu
Prof. Yu is a professor here at UC Berkeley. He is teaching two classes this semester; one is a lower division course consisting of about 270 students. He likes giving exams because it gives him an idea of how the class is doing. He dislikes seeing low scores on exams and monitoring a large class. His priorities include preparing clear questions for the exams, making sure that the students have enough time during the test, and being able to attend all the students' questions about the exam.
Problem and Solution Overview
The interviews conducted revealed a disturbing fact. TA’s were almost apathetic about cheating in classes. Human nature makes people care less about things that are harder to stop. Thus, our system strives to serve universities in making tests easier. Our system’s goals are to assist students, professors and TA’s; all the people who are effected by testing.
Large classes make life difficult for test administrators. There are a lot of tests to grade, regrades are a hassle, not to mention the test itself. It’s hard to proctor over a large gym full of students crammed in. As a student, life isn’t easy either. You’re sitting in a crowded room trying to get the attention of a GSI by frantically waving your hand in the air.
Our solution to this is a system that involves the Anoto pen. By establishing a wireless network we will find a way to link all the participants of the test together. Students take a test on specialized paper. To get the attention of a GSI, all they need to do is check a box. GSI’s get a realtime display of the students’ status; the system calculates the probability that a student is cheating based on their location in the room, and the questions other students around them are working on. The system has the ability to alert the GSI about student’s with questions or who could be cheating. Professors get a statistics package that facilitates analysis.
Contextual Inquiry - Interview Descriptions
Professor 1/Exam 1
One of the group's main objectives is to identify problems in the current system of monitoring an ongoing examination. A little research led us to a class with 270 students undergoing an hour long midterm exam. Due to the needed silence in the exam setting, I decided to perform the interview in two parts. First, I sat in through the entire exam and logged every actions (by professor, Graduate Student Instructors and students) observed into our notes. The second part was the actual interview with the professor guided by the notes that we have taken during our observation of the examination. Basically, the chronologically ordered notes taken of the exam guided the flow of information from the professor. We believe that this process is equivalent to interviewing the subject while he is monitoring the exam. Aside from the professor, four other GSIs are monitoring the examination. The instructor started the exam by addressing the students of keeping the exam face down as it is being passed out. While the GSIs are passing out the exam, the professor gave instructions for the exam. He talked about the format of the exam, how the student has to write their names on each page of the answer sheets, and other instructions, which are unclear from the back of the lecture hall. The professor ended the initial remarks with a warning that "cheating is not allowed." When asked later of what ways does he ensure that cheating is prevented, he answered that "there's really no way [to ensure this]" and that he "just got to trust the students not to cheat." Twelve minutes after the hour is when the exam was officially started even though I saw that some students already started before that. Late students took the midterm way later than that. Sitting at the very back of the auditorium, I had a clear view of the entire environment. I could see every movement of the GSIs and the professor as well as the students. The midterm was a five-paged closed-book exam, each page containing one or two questions. Two students already asked questions right away attended by the two GSIs. One GSI went to talk to the other GSIs and the professor, resulting to the professor's interruption of the class to give one more instruction about the exam. Two GSIs started to roam up and down the room. Then, one of them joined the other three GSIs in one corner of the room, only looking up to the students once in a while. The professor stayed on the blackboard, constantly advancing the clock hand he drew on the board. I asked the professor whether there were a set of instructions or roles given to the GSIs in monitoring the exam. Aside from passing out and collecting the exams, he told the GSIs to make sure they "attend students' questions." Throughout the exams, both the GSIs and the professor attended at least 11 questions. What I noticed about this is that the questions usually came from those who are near a GSI or a professor. One student from the back was ignored when he raised his hand because no one saw his hand. Also, the professors would interrupt the class and clarify a question outloud once in a while. When I asked the professor about what kind of questions did the students asked, he mentioned that the wording of the problems seem to confused some students. Sitting near me are two students with extra sheets of paper; they could have been using these as scratch but they were not supposed to have these. The professor spotted these and spent about a minute eyeing them. I think that the professor hesitated to ask about these sheets because of the pressure of drawing attention to the students as well as interrupting the exam. Throughout the exam, only thirty percent of the time did anyone came up to monitor the back of the class. From my perspective, anyone sitting on the back could easily cheat on this exam if they want to. When the instructor told everyone that the time is up, students crowded in the front of the room to turn in their exams to their GSI. The professor mentioned that having the students turned in their exams into their respective GSI would help them in grading them. Even though the time was already up for the exam, a lot of students are still working on their tests. I even saw one student on the back showing his answer to the person beside him. I thanked the professor and sent him a thank-you note for the interview.
GSI 1
I have also interviewed a GSI of the observed exam. The interview occurred in his office for about an hour. I let him interrupt the interview once in a while as he checks his email or talk to someone on the phone. First, I asked a little about his background. I also asked his opinion about monitoring an ongoing examination. He said that monitoring an exam is a little bit "boring." He also admitted that he was not really alert all the time during the exam. At the time, he was finishing up grading the exams. He told me that they would usually grade the exam in groups, where each GSI grades part of an exam. However, in grading the recent midterm, each GSI would be grading the whole exams of students in their respective sections individually. So, I asked him first of how they usually grade the exams in groups. Using blank sheets of paper as mocked exams, I asked him to demonstrate the usual grading process as if we were both GSIs grading them. Then, I asked him if I could just watch him grade an actual exam. He pulled out the next midterm to be graded. I decided to described out loud what he was doing and he would correct me once in a while. I would ask him about the marks that he was making, his criteria of awarding points and those that troubles him in grading the midterm. The next part of the interview consists of him demonstrating to me his way of entering the scores on the spreadsheets. It was supposed to be a quick process but he ran into trouble while showing it to me. The application that he was using just disappeared from the screen. I let him figure out for a while what happened to the application before I offered my help. Unfortunately, I was also unable to bring the application back to the screen. He had no choice but to open a new one. The interview ended with a description of how the information I got from him would affect our design. He just nodded his head. I thanked him and offered if he needed any help. He had me done one errand. I still have to send him a thank you note.
GSI 2
We met GSI 2 at Café Milano when she had a break from class. At the café, we talked about how she currently proctors tests and what kinds of things she does while she is giving a test. In addition, we looked at her grade a few papers and saw how she recorded scores. Although she was very comfortable with new technology, she seemed very pleased with the convenience that the traditional paper and pen process was when grading papers. We also observed her administering a quiz to her students and realized that she in fact did not spent too much effort trying to catch potential cheaters. When asked about why she read during her exam she noted that she in large part trusted her students and that her schedule was very tiring and enjoyed being able to relax.
Students 1 and 2
Students 1 and 2 were interviewed at their apartment where both of them lived. This was the best alternative to interviewing them during an actual exam, which was unconditionally not allowed, since they did much of their school work at their apartment. During a testing situation, both agreed that the only way to get someone's attention to ask a question was to raise your hand and hope to make eye contact. One student said when he has a question, he raises his hand and moves on to another problem, since he knows it will take awhile for someone to get around to answering him. He agrees that this breaks his chain of thought, but accepts it as a necessary evil of the system. Both agree that being able to "text message" questions to TAs or GSIs would be useful, at least in eliminating hand-raising. Only one student had observed cheating during an exam before, but had not reported it since it was someone he knew, and he did not want to create an awkward situation. He said that if he had had an inconspicuous and anonymous way of implicating the cheater, he would have done so. Both also agreed that "time-stretching," on the other hand, is much more prevalent among students and much more annoying because of it. They both showed hearty approval for a system that would strictly limit answering questions to the time constraints given by test proctors.
Shared Tasks and Common Themes
Shared Tasks and Common Themes
1. Those who facilitated the exams all share the same tasks of scanning the classroom for raised hands and attending students' questions. All admitted that monitoring can be boring. In longer exams, both the GSIs try to find something else to do (read a book) while monitoring an exam. All are also familiar with the same process of grading exams (each grader grades a part of the exam). They also use applications such as spreadsheets to record exam grades.
2. At the beginning of the exam, all subjects look over exam questions; the students answer them while the GSIs and the professor look for questions that might have been unclearly stated.
Unique Events
Since our project mainly focuses on the whole exam process, roles span among the students, the GSIs and the professors. In each interview, we focus on unique roles that each subject takes in an examination. First, the interviewed students were asked of how the current exam process is from a perspective of someone who is taking it. This identifies problems that we considered in designing an interface for students. Next, interview questions for the professor were mainly focus on the problems encountered during an acual monitoring of an exam. We identify the professors priorities when it comes to giving an exam so we can gear our design towards these. Lastly, we focus on the grading process when interviewing the GSIs. The different information the we acquired from the subjects about their different roles in an exam setting gave us a wide background in designing the different interfaces that hopefully address the problems we identified.
Task Analysis Questions
Who is going to use the system?
- Universities
- Professors
- Students
- GSIs
What tasks do they now perform?
- Administer tests
- Take tests
- Accumulate and store records and statistics
What tasks are desired?
- They want to accomplish the same tasks as before, but be able to do it in a more efficient and convenient manner.
- New Features
- Automatically create records and statistics
- More convenient test administering
How are the tasks learned?
- The users need to be trained in order to understand the new test-taking system. Over time, the users should be able to become comfortable through repitition.
Where are the tasks performed?
- The tasks are performed on campus in classrooms, and occasionally large open areas like Hearst Gym and Haas Pavillion. Other aspects like grading and record keeping are done in offices and café’s around campus.
What’s the relationship between user & data?
- Students and GSIs make the data.
- Professors, students and GSIs need to interpret the data.
What other tools does the user have?
- laptop
- paper
- pen
How do users communicate with each other?
- Published tests statistics
- Returned tests
- Class website
How often are the tasks performed?
- Every class has a few tests every semester
- Professors and students look up or modify records almost daily
What are the time constraints on the tasks?
- Sometimes professors need to finalize grades by a certain deadline
What happens when things go wrong?
- The actual physical tests can be archived because the students can access the digital records of the test, so a loss in the digital data will not lead to disaster
- If there is a problem with the Anoto pens during test-time, the GSIs can fall back upon the paper records of the exam
Analysis of Tasks
Easy
Students taking an exam
The students are given a sheet of questions along with an answer sheet. Sometimes, the exam questions are in the answer sheet. The students only need to write their answers on the answer sheet. Problems only occur when there is no more room on the answer sheet for students to write on. Regardless, this is an easy task.
Students asking questions during the exam
Our observations show that more questions are coming from the students seated near a GSI/professor than those who are farther from where the exam facilitators are. The student seating in the middle row at the center of the class is less likely to ask a question than those who are seated in the front of the room. Also, there are students who are just shy and don't want to bother raising their hands to ask a question. The student's raised hand also needs to be noticed by a GSI or professor. Given that the students are working under the pressure of time, some who raised their unnoticed hands would just put their hands down. Despite this, we still classify this as an easy task since a student only need to raise his/her hand in order to ask a question.
Moderate
GSIs/professor proctoring the exam
The only required action needed in this is the GSIs full attention of the examination in progress. However, we classify this as a moderate task because according to our interview, the GSIs always find it hard to be alert during the entire examination. Also, as the ratio of the size of the class to the number of facilitators gets bigger, it gets harder to observe an ongoing examination; more questions to attend to and harder to enforce exam rules.
Detecting/preventing cheating
We classify this as a moderate task because this is both easy and difficult in different aspects. The presence of GSIs during an examination already discouraged cheating (easy). However, our observation of an ongoing exam point out a lot of opportunities to get away from cheating (hard). If a student knows that the facilitators don't always have eyes on the entire large class all the time, s/he could cheat without being detected. Also, the time to start and stop the exam is poorly enforced; some students start to write even before the exam is officially started and continued to write even after the time is up for the exam. Also, graded exams handed back to the students offer another opportunity to cheat through changing their answers and asking for a regrade.
Difficult
Grading an exam
We consider this task difficult based on the results of our interviews. The GSIs complain of the amount of work that they have to do in grading an exam. If they are grading the exams in groups, there is usually a process that they must adhere to. Also, they must be consistent in the way they grade the exams. Also, when they are grading a lot of exams, there is also this problem of miscalculating scores. After grading, they need to sort out the exams. This process can get really hard especially when students do not put enough information (section number or GSI's name) on their answer sheets. Regrades also posed another difficulty since the GSIs need to make sure that no changes has been added to a student's answers after the exams are passed back. One GSI commented that "desperate students" would try to do anything (argue their answers or possibly make changes on them) just to get more points on their exams.
Gathering statistics from the exam
We classify this as a hard task because of two reasons. First, tallying scores can be a time-consuming process. From our interview, we found that the GSIs or the professor usually use a desktop application such as spreadsheets to store scores so that these can be subject to statistics calculation. One interviewee complained of the time it takes to manually enter scores. The second reason is the limit on what statistics can be computed. Current exam statistics only include the mean, median and the standard deviation. Other kind of statistics (average time spent on each question) might be useful in evaluating the performance of the class but these are just hard to calculate.
Interface Design
Student Interface
The exam would be composed of a question booklet, an answer booklet, and an attention page. Separating questions and workspaces allows students to work on problems in any order they want. To answer a question, the student would turn to an empty answer page, mark the question number at the top of the page, and work out the problem on the rest of the page. If they reach a solution on this page, they would write their answer in the defined solution box. There are also boxes for the grader to mark the student's score, but the student is not allowed to mark in this area.
If the student has a question or would like to report suspicious behavior, he or she would turn to the attention page. The student provides all relevant information in an empty input area, then checks the Send box to send the question the screens monitored by the exam proctors. The student can then return to work, secure in the knowledge that each proctor in the room is aware of the question, and the one most able to answer it will be addressing it shortly.
GSI/TA Interface
Each circle represents a student taking the exam. Arrows allow TAs to scroll around the room if necessary. Circles light up different colors according to the students' action. Potential cheaters are highlighted in red, deeper shades equate to high probability of cheating. Questions flash blue. If a TA clicks on a flashing blue student, the question asked will be displayed in a pop up window.
Professor Interface
The above figure illustrates an interface for the professor (and GSIs) in viewing statistics from an exam. The program is a desktop application designed to closely resemble the interface familiar to the users. Our users are familiar with Windows interface, which influences the above design. Aside from viewing exam statistics, the above interface also offers a way for the user to compare a student's answer sheet to the answer key. It also provides a mechanism for the user to send a digital copy of an answer sheet to a student as well a way to manually change scores stored in the database (useful for handling grading errors, regrades, etc). Other features such as easy sorting of data is also provided. (1) The main page allows the user to select which exam statistics to display. Selecting which course and section to display the data for does this. On the left pane is a list of students with their scores and section number. On the right pane, on the other hand, exam statistics for the selected course is displayed. The statistics include the mean, median, standard deviation and the average time spent per question for the exam. (2) If the user clicks on one of the names in the left pane, a digital copy of an answer sheet for that selected student is displayed in the right pane. (3) Clicking the "More" button pops up a new page which displays a student's answer sheet next to the answer key. This allows easy visual comparison between the two sheets. The arrows on top of this window provide a familiar way for users to browse sheets page after page. Clicking on the "Email" button would email two sheets to the student. (4) Clicking on the "Update Grade" button allows the user to manually change the scores for a particular student. At each stage of the interface, a cancel button is provided, allowing a user to undo his/her actions.
Analysis of Approach
Anoto is an ideal solution for this particular application because it means making as little change as possible to the existing interface. Especially with a testing application, it is extremely desirable for users to have a high comfort level so that critical functions are not in any way affected. For students, it would be unfair for a testing application to lower their grade, or otherwise negatively affect their ability to display their knowledge. For GSIs or TAs, grading can often be a time-critical process, so any application should, if anything, make grading easier and more efficient rather than more cumbersome.
The only real competition with an Anoto-enabled pen is the Tablet PC. Other options like PDAs can be eliminated outright because they are not feasible test-taking options.
Anoto
Pros
Using an Anoto-enabled pen is no different from using a normal pen; this means that an application developed with Anoto capabilities will already be intuitively familiar to users. Extensive learning in order to use and benefit from the application will be almost completely unnecessary. Also, while there is a significant initial overhead cost in purchasing pens for the application, paper costs almost nothing to reproduce in mass quantities. Even the Anoto dot matrix can be reproduced with a sufficiently high-resolution printer. Another important benefit is the robust nature of this interface; while a tablet can crash, paper will always preserve a hard copy of all work done so that it can be reconciled in any necessity.
Cons
Making the application so intuitive for the users means hiding the technology on the developer side. This will mean a more difficult implementation of the application; the penstrokes of a user will have to be either interpreted so that the computer knows what they mean, or they will have to be transmitted and presented in a way that other users know what they mean. There will also be some difficulty in increasing the range of each pen's bluetooth range into a network capable of spanning a typical examination room.
Tablet PC
Pros
Tablets have been heavily developed in recent years, and now boast many features like handwriting recognition that are not packaged with an Anoto-enabled pen. Tablets also have the advantage of much more computing power available where pen meets screen, making possible several interactive features that a piece of paper cannot duplicate. Finally, nearly all tablets come with wireless capability that make setting up an exam room network easy.
Cons
Writing on a tablet is not as intuitive as writing with a pen; doing anything requires some configuration, and the same interactive features that could help a user may also hinder a student's test-taking ability. Also, there is an undeniable comfort to the familiar feel of physically writing on paper, and this psychological edge could mean a difference in grades. There is also the possibility of a tablet crashing, destroying all progress on a timed and critical examination.
