InteractivePrototype-Group:BigBrother
From CS160 User Interfaces Fa06
Contents |
Prototype
Image:BB Proctoring.zip
README
Web Interface
Presentation
Team Members
- Jon Chang – worked on SwingUI class, listeners, debugging, slides, report.
- Randy Hilarbo – worked on PaperUI classes, website, debugging.
- Charles Leung – worked on SwingUI class, listeners, debugging, report.
Problem and Solution Overview
While the main function of university exams is not to entertain and delight the student, we of Team BigBrother believe that there are aspects of the process that can be improved, resulting in improved academic performance and more efficient distribution of resources post-exam.
What's Wrong?
For a test-minded student, getting an exam-related question answered in the middle of a large undergraduate midterm proctored by a single GSI can be a daunting process. Such an environment is also rife with "gray area" cheating like starting ahead of time, or having an extra five minutes to work from sitting in the back of the testing room.
After the exam, students often do not have all the resources necessary directly at hand in order to make an informed decision of whether or not to request a regrade. This can lead to either frivolous requests, which waste the graders' time and frustrates students who mistakenly believe they can earn more points, or a decision not to submit a valid regrade request.
How Do We Fix It?
We deploy a testing application that records everything written on an exam, and processes it to solve the stated problems. Students can reliably ask questions with a paper messaging system, and timestamps allow the system to flag pens that are working before the test begins or after it ends.
The same application deploys what it recorded to a web application where students on their own time can view their graded tests, consisting of everything the wrote and everything the grader wrote, and based on a perceptually close grading rubric submit regrade requests.
Tasks
Easy: Asking Questions
In the current exam setting, students only have to raise their hands in order to let the monitoring figures know of their inquiries about the exam. However, this current method does not always guarantee that a student’s question is immediately attended if it is attended at all. The reasons why this is included, based on the results of our contextual inquiries, the inattentiveness of those who are monitoring the exam, the invisibility of the raised hands to the monitoring figures due to the large size of the class as well as the distant position of the GSIs/professor relative to the student with the raised hand.
Our interactive prototype provides a better alternative to raising hands guarantees that a student’s inquiry will be attended by immediately sending notification to the test proctor of what the question is. In addition, if the proctor sees that many students have similar questions, he can simply make an announcement to the whole class and save himself, and his students time.
Medium: Taking the Test
Taking the Anoto-enabled pen is categorized as a medium task because of some expressed discomfort during the Lo-Fi prototyping with taking the test in our altered format. While we did spend some time attempting to address this issue by brainstorming ways of making the format more approachable, we eventually decided that not only was the dissenting voice in the monitor, but each designer could remember a similarly designed exam they'd taken before. Therefore, we decided to keep our previous format of a sheet of questions, and blank Anoto-enabled answer sheets for the students work and answer on. With respect to the potential discomfort, we kept the rating of this task of Medium, and chose exam implementations that already used the separate question and answer format.
Hard: Reviewing and Requesting Regrade
In some classes, the GSI only gives students a limited time to decide whether to ask for a re-grade. These comprise the reasons of why we identify this as a hard task in the current exam setting. The students are required to follow certain methods (i.e. prove necessity of a re-grade through writing, etc.) just to challenge the consistency of the graders in marking exams. In terms of this assignment, our interactive prototype for performing this task may require the most effort from our users. Our users need to interact with a computer in order to complete these tasks by visiting a website that we built for students to lookup their tests and request re-grades.
Revised Interface Design
Changes from Lo-Fi Testing
As discussed earlier, we were unable to find a better way to design the physical test students write on. However, we did make a minor addition to the Proctor Interface and some major design changes to the Web Interface.
The Proctor Interface
Most feedback to this interface was positive, and there weren't many things people wanted changed. A request we implemented was the desire of proctors to view old messages and questions, in case they wanted to compare to previous questions or gather information about how the test is going in general. To accomplish this, we added a history queue to the Proctor Interface in addition to our existing message queue, and had messages jump from the message queue to the history queue when first activated. Messages in either queue can be selected and viewed at any time.
The Web Interface
Based on Lo-Fi testing, we learned that students would definitely want to use the benefits of the online review and regrade system. However, they did have some problems with our original mock-up of how the website would be navigated and the information displayed; even the rudimentary paper navigation showed how difficult it was to perform their desired tasks with our current design. Using this information, we sat down and made several design choices that have been reflected in this iteration on the interface, now in actual webpage format rather than paper cutouts.
Test Selection
Knowing that students would have more than one class with tests to record, we took the concept of respectful plagiarism to heart and adopted the design of a currently implemented design for retrieving test files. The lists and clicks involved are intuitive, and similar to many other context navigations used in selecting particular car models, or books at the bookstore website.
Viewing
We realized after the Lo-Fi Prototyping that our original plan of having multiple panes visible at once for comparing graded and commented exams to score rubrics or other documents of note would leave too little screen real estate to efficiently view anything. Looking at any normally sized document would require both horizontal and vertical scrolling, in amounts distasteful to our prototyping subjects. We changed this to a few buttons to overlay what the student and grader wrote over the original exam, view the questions, or view the grading rubric within a single viewing pane.
Regrade Submission
We noticed the prototyping subjects wanted to correlate their regrade request comments with actual test material, and navigated back and forth between their the viewing page and the regrade page several times to get the message right. To remedy this, we made the regrade page pop up as a new, small window containing only a text field and a submission button. While this change requires about the same number of clicks to correlate information, it makes the user feel like there is less distance between their tests and their note to the graders.
Unimplemented Sections
At this point, we have left out the business logic between displaying an image of the students' and grader's inks on the webpage and collecting that ink during testing and grading. This would involve creating an HSQL database and using already implemented Java classes to store the ink collected on this database. The ink would then be retrieved from the database and painted onto a visual surface of the webpage, using the Paper Toolkit with JavaServer Pages.
Task Storyboards
Asking Questions
Taking the Test
Reviewing and Requesting Regrade
Prototype Overview
Implemented UI
Message UI
We provide three opportunities during our prototype test to ask a question using our novel messaging technology (see Message Sheet screenshot below). Each section on the message page contains buttons to select which question the message pertains to, a large space to capture the message, and a send button. The question and send buttons may actually be converted to checkboxes or bubbles, due to difficulties with recognition during our own preliminary testing; we've already had to enlarge the question buttons because the camera wasn't registering their existence at a normal writing hand angle. Each space is designed to be used only once, so it is assumed that student's can ask at most three questions for the duration of this exam.
Test UI
The question sheet of this exam is identical to the conventional exam it is taken from. The answer sheets however, have been Anoto-enabled over the entire writing space (see Answer Sheet screenshot below). Students will take their test with these sheets as normal, and the entire collection of ink is sent to a central database, which is accessed by requests from the Web UI to view the exams. While students will have to take their exam in pen, rather than in pencil, the interface is otherwise identical to a normal testing environment to produce as little degradation in score performance as possible. At any rate, the pen follows the theme of recording all things written, and with so much real estate to work with on the answer sheets it is a simple matter to learn to cross out errors, rather than spend valuable time erasing them.
Graders will also be asked to grade on the Anoto-enabled areas as well, so that their marks and comments can be recorded to the same place as the student's ink. Switching this part of the application to grading mode changes the color of the grader's recorded ink and allows the system to differentiate between student work and grader marks even when the same pen is used. This was a conscious design decision; while it is generally advisable to avoid modes, we decided that it was still equivalent to only one mode per person and could be justified.
Proctor UI
This interface is coded in Swing, and is split into two halves horizontally (see Proctor Interface screenshots below). The upper half is a desktop where messages from students or system warnings can be expanded into internal frames and viewed as small windows. The lower half is a control panel containing the message and history queues, buttons for starting and stopping the test, and instructions to the proctor for using the interface. The business logic behind this interface is keeping track of whether the test is in session or not, and will flag students that do work on their test outside of the session. New messages are placed in the message queue, but moved over to the history queue when accessed for the first time; they are never lost from the history queue unless the application terminates.
Web UI
This is a website where students can login with their SIDs and gain access to all Anoto-enabled tests they have taken that semester (see Reviewing and Requesting Regrade storyboard for screenshots). Logging in takes you to a set of dynamically-filtered lists that allow the user to select by class and exam exactly which test they want to view. After selecting an exam, clicking the "View Exam" button navigates to a new page with the graded exam taking up most of the room on the page. There are controls for viewing your ink, the grader's ink, or the grading rubric, and a "Request Regrade" button pops up a small window with an empty text box, a submit button, and instructions for requesting a regrade. If professors find they are receiving frivolous regrade requests, they can limit the number of regrades allowed per student, and this number can be integrated into the system.
Left Out
Data Model
We left out the data model for the Web Interface, consisting of a database for storing ink and a second database for holding student logins, exams, test statistics, and other information. This kind of backend is already in common usage, and contains neither user interface nor usage of the Anoto pen; thus, we decided that it didn't serve the goals of this class to spend time on a part of our application that didn't pertain to our UI education.
Cheat Detection
Readers will note that flagging cheaters has, except for people that start before and work after the testing session, been completely eliminated from our project. This is not only because of the loss of an integral member of Team BigBrother, the driving force behind the detection idea, but also because of the difficulties of designing an appropriately sensitive algorithm and administering it fairly. Throughout the process we tried to answer the myriad of questions surrounding this idea: What qualifies as cheating? What is observable to the pen as cheating? How can we avoid false positives? How can we avoid false positives and still detect positives? As time went by and we were still unable to answer these questions, we finally decided to scrap the idea altogether. Students can still report cheating with the messaging system, and students will be held to working within the prescribed time limits, but this is all that survives the removal of cheat detection as a goal of this project.
Wizard of Oz Requirements
The only part of this application requiring magical assistance is the transportation of the graded exam from the Proctor's application to the Web Interface. We've accomplished this by creating our own graded exams using a Tablet PC and integrating them with our Web Interface to give the appearance of retrieving them from a database. All other portions of this application function as advertised.
Prototype Screenshots
Proctor Interface
Early Start Flag
Question Asked
Late End Flag
Student Interface
Answer Sheet
Message Sheet
