PilotStudy-Group:BigBrother
From CS160 User Interfaces Fa06
Contents |
Introduction
While the main function of university exams is not to entertain and delight the student, we of Team BigBrother believe that there are aspects of the process that can be improved, resulting in improved academic performance and more efficient distribution of resources post-exam.
The System
We deploy a testing application that records everything written on an exam, and processes it to solve the stated problems. Students can reliably ask questions with a paper messaging system, and timestamps allow the system to flag pens that are working before the test begins or after it ends. The same application deploys what it recorded to a web application where students on their own time can view their graded tests, consisting of everything the wrote and everything the grader wrote, and based on a perceptually close grading rubric submit regrade requests.
Purpose and Rationale
Since the function of an exam is to test the knowledge of the exam-taker, it should be the goal of proctors to eliminate as completely as possible all other factors that may interfere with performance. Preliminary surveys established that addressing questions and enforcing time limits in a large testing environment, like many lower-division undergraduate classes at Cal, detract from the exam's ability to fairly evaluate knowledge. Our system minimizes these factors in an intuitive and user-friendly way, using Anoto-enabled pen technology. The pen also allows us to offer the opportunity for students to easily evaluate their performances on exams, another heavily-requested feature in our surveys.
Implementation and Improvements
AskQuestions Paper UI
One flaw that we noticed in our AskQuestions Paper Interface from our Interactive Prototype is the lack of feedback to the students. We were faced with the problem of what happens if the student's tap on a question or the send box did not get register. This will prevent the student's question to appear in the proctor interface. We remedied this by adding tiny boxes within the active regions in this paper interface and require the students to bubble this boxes when using the paper interface. This will make sure that the students will take more time on these actuve regions and thus, increasing the probability that their actions will be registered. Additionally, we changed the instructions in this paper UI to accomodate the above change.
View Exam and Regrade Request Interface
Our third application provides an interface for students to check their graded exam from their computer and allows them to send a regrade request to their instructor. From the previous assignment, our Interactive Prototype for this interface is an online website. Here, the student can log in online using their SID and browse through all the exams they have taken using our digital exam system. In this part of the project, we decided to convert this web application into a desktop application. We re-implemented our third interface; this time, we used the JaVA Swing. The reason leading to this decision is our desire to remove the "wizard of oz" implementation of manually transferring the student's exam to our website server's database. That is, this interface automatically display the student's graded exam as it made available by our proctor interface through a single click. The basis of this application is entirely our previous web interface. Each page on this interface is the JAVA Swing equivalent of the corresponding page in our previous web interface; that is, the layout of this interface is guided by the website's layout. Aside from the improved functionality, we made this interface more user-friendly; we added more confirmation/error popups to check on our users' attention on important tasks they are currently working under. This interface provides an important part in our entire system in that it makes our digital exam application more complete than before. It also added an additional consistent feel to our system as a whole: our interfaces are now consist of desktop and paper user interfaces.
[A storyboard/screenshots of this interface is provided in the appendix]
Method
Participants
We selected two undergraduates and one teaching assistant for an undergraduate class for this study.
The two undergraduates were chosen to provide a varying level of technical comfort and familiarity. One of the participants is a recent Berkeley Molecular Cell Biology graduate, and is now a graduate student in the Optometry School. He is 21 years old and is very familiar with taking tests in a college environment. In addition, he is fairly comfortable with new technology and enjoys trying new devices that make his life easier. He was selected based on availability and the fact that he would be a target user for our product. For example, MCB classes are known to be very large and have very wordy questions, as a result we believe that the BigBrother testing interface would be well suited to Biology classes at Cal. The other is a junior majoring in Electrical Engineering and Computer Sciences, with recent experiences with large lower-division classes and difficult testing environments. He also, as a result of both his major and his hobbies, has extensive experience with both computers and unfamiliar technology.
The teaching assistant we selected also has an expert level of familiarity with computers and other related technology. As a senior undergraduate student, he has experience with both taking and administering large undergraduate exams, and so his feedback was doubly valuable.
Apparatus
We used an HP Compaq tc1100 tablet PC to run the TA interface for experiments outside the lab, while we used the workstations in 330 Soda when participants were willing to go to lab with us. In both cases, we used the Nokia SU-1B digital pen and a Zoom USB Bluetooth device to provide connectivity to the machine running our application. The sample test was a randomly chosen biology midterm, and the students were given a sheet of the questions, our answer sheet, and our question sheet stapled in a packet in that order.
Tasks
Easy: Asking Questions
In the current exam setting, students only have to raise their hands in order to let the monitoring figures know of their inquiries about the exam. However, this current method does not always guarantee that a student’s question is immediately attended if it is attended at all. The reasons why this is included, based on the results of our contextual inquiries, the inattentiveness of those who are monitoring the exam, the invisibility of the raised hands to the monitoring figures due to the large size of the class as well as the distant position of the GSIs/professor relative to the student with the raised hand.
Our interactive prototype provides a better alternative to raising hands guarantees that a student’s inquiry will be attended by immediately sending notification to the test proctor of what the question is. In addition, if the proctor sees that many students have similar questions, he can simply make an announcement to the whole class and save himself, and his students time.
Medium: Taking the Test
Taking the Anoto-enabled pen is categorized as a medium task because of some expressed discomfort during the Lo-Fi prototyping with taking the test in our altered format. While we did spend some time attempting to address this issue by brainstorming ways of making the format more approachable, we eventually decided that not only was the dissenting voice in the monitor, but each designer could remember a similarly designed exam they'd taken before. Therefore, we decided to keep our previous format of a sheet of questions, and blank Anoto-enabled answer sheets for the students work and answer on. With respect to the potential discomfort, we kept the rating of this task of Medium, and chose exam implementations that already used the separate question and answer format.
Hard: Reviewing and Requesting Regrade
In some classes, the GSI only gives students a limited time to decide whether to ask for a re-grade. These comprise the reasons of why we identify this as a hard task in the current exam setting. The students are required to follow certain methods (i.e. prove necessity of a re-grade through writing, etc.) just to challenge the consistency of the graders in marking exams. In terms of this assignment, our interactive prototype for performing this task may require the most effort from our users. Our users need to interact with a computer in order to complete these tasks by visiting a website that we built for students to lookup their tests and request re-grades.
Procedure
Preliminaries
After the participant has been greeted and seated at the appropriate workstation, an experimenter briefly explains the purpose of the experiment, and the task the participant will be performing (script provided in the appendix). Then, experimenters set up the application for the experiment while the participant reads and signs the consent form (provided in the appendix).
Experiment
Taking the Exam
The same test was given to all participants for purposes of consistency, but we made it clear to each participant the accuracy of the exam was not as important as the experience of taking it. Participants were given half an hour to simulate the test-taking experience, and instructed to answer each problem and ask at least on clarifying question of the "proctor" using our interface. The experimenter posing as the test proctor was carefully consistent throughout the exam in their behavior as a TA; he did not look up from the computer screen displaying our application unless he received a message notification, and treated the participant as impersonally as a student. While none of our participants were tempted to cheat by starting too early or ending too late, the experimenter explained this aspect of the application to the participant after the exam.
Proctoring the Exam
If the participant was the proctor, an experimenter served as a student taking the exam. The experimenter started his exam early in order to demonstrate this aspect of the application, and the proctor was allowed to react as his experience defined. The participant also asked two questions throughout the exam in response to different questions, and the participant was instructed beforehand to also act on these queues as they best saw fit. After the exam, the experimenter handed the participant his completed exam, and the participant was instructed to use our application to grade the exam.
Viewing Exam and Regrade Request
We asked each participant to use two versions of our interfaces. First, we had them interact with our previous web interface and asked them to perform the tasks described above. We asked them to think out loud while they are doing the experiment. During the process, we placed special attention to the errors that they were making as well as to their reaction toward the interface. Then, we introduced to them the desktop equivalent of the web interface and asked them to perform similar tasks to it. We kept note of the same measures we were keeping track of during their interaction with the website. We had them perform the tasks on each interface three times while keeping a timer of how fast they perform each interface. After the experiment, we asked them to give us their opinion on each interface they encountered. We primarily inquired on the easiness of using each interface as well as which interface they prefer better. We also looked for their insights on the colors we were using, the design layout as well as the functionality offered by the interfaces. We ended the experiment on this task by taking their suggestions of how we might improve our interface.
Exit Survey
After the experiment was completed, participants were given a questionnaire to determine quantitatively their reaction to the application (see the appendix). After completing this questionnaire, the participant was given an extremely small stipend for participating, the size of which was determined by our nonexistent experimental budget.
Test Measures
The following are measures we were looking for in this pilot usability study:
Average number of errors performed on each interface
We measure this to get a good opinion on how user-friendly our application is.
Average time it takes to perform the tasks associated with each interface
Our system is something that our users have not encountered before and this measure provides a good estimate of how fast our users can adapt to this new application.
Users' personal opinion
Primarily measured in our third interface, this measure influences our decision of whether to stick with our previous web interface or to officially switch to our JAVA swing interface.
Colors, layout, design preference
We personally wanted the participants opinion on the color, layout and design choice we made on our interface. This is the area we are not completely confident about and we are looking forward to this pilot usability study offering us good suggestions on how we maybe able to better these parts of our interfaces.
Implementation errors
Though we tried to rid as much bugs as possible from our interfaces before deploying it for testing, we are still hoping that this study would identify bugs that we missed.
Results
Taking the Exam
As can be seen from our raw data (see Appendix), there were some problems in the students ability to adapt to the new style of exam taking. Participants also seemed to suffer slightly from the lack of feedback from the ostensibly computer-like messaging system, though it is impossible for a piece of paper to provide such feedback. Before our Final Presentation, we will be analyzing methods of designing the messaging interface to eliminate the need for feedback.
Proctoring the Exam
This part of our application met with the most favorable response; the participant expressed very positive feedback for the innovation and had little to no difficult manipulating the interface, even for the first time. He credited this to its similarity to applications he'd used before, such as instant messaging clients or other timing mechanisms he has used to set time limits on exams.
Viewing Exam and Regrade Request
Our results placed us in a rather difficult position in deciding which third interface to officially incorporate as part of our digital exam system. The following is a summarized result for evaluating our two interfaces which perform similar functions:
- The average time of performing all the tasks (viewing exam and requesting a regrade) during the first iteration for the web interface is less than that for the desktop application. But as we do the second and the third iterations of the study, the discrepancy between the times it took for the participants to use the two interfaces became less; though the web interface won this category.
- There are more errors done by the participants while using the desktop application. These errors obviously slowed the users in performing the intended tasks. Some errors include visibility errors where the user can not find immediately the buttons in the page. Another kind of error is the recognition error where the participants can not identify readily the provided functionality that a page in the interface offered. For example, there is a part in our desktop experiment where we rendered an html page on a JPanel and one of the participants was trying to interact with the html by clicking on it even though it provides no functionality. These errors, however, decrease significantly through the following iterations. Errors done using the web interface include clicking the wrong link and making a wrong selection.
- The participants' personal opinions on each version of this interface varies accordingly. Two of the participants complained about the time it takes to wait until their exams get loaded in the web interface. The participants also comment on the color choice we made in implementing the web interface. The most important comment that they made, we believe, is that the web interface looks too simple and ordinary. In using the desktop interface, the participants prefer its interface: the buttons are more recognizable and the rendering of the pages is just "nice." Overall, they like the desktop application better.
- Our experiment also identified some bugs in both interfaces. There were some spelling errors as well as some grammar errors. In the desktop application, one participant discovered a small bug that we missed.
Discussion
Viewing Exam and Regrade Request Interface
Based on our results, we made the decision to abandon the web interface and incorporate the newly-implemented JAVA swing GUI as the third interface of our system. Though the web interface provides a better recognition property and less error rate, our participants (and the design team) like the feel of interacting with the desktop application better. Though, the learning rate is considerable, we understand that the desktop interface is a new application to the users and we somewhat expected that the users will require time to get used to it. However, the results of the pilot study provide us with information that should help us lessen this required time to learn this new interface. Before deploying this application into a larger audience, we would make the following changes:
- Increase visibility - We would make the buttons more recognizable and arrange them in a way so that they can easily be identified. We will also increase the contrast in our pages to differentiate the intended functionalities of the different sections of the pages. For example, in the exam page, we will increase the contrast between the page background and the course/exam lists so that the users will easily recognized that they can select an item from the lists.
- Add more functionality - One common errors we noticed in this interface is the participant's attempt to interact with the html page rendered in one of the sections of our interface. To avoid this error and also increase the features provided by the desktop interface, we are thinking of making this html page active so that the users can actually interact with it.
- Consistency - We are discussing of changing the color choices for this interface so that they are consistent with each sections of the interface as well as with the proctor interface.
- Defaults - We want to work on figuring out a good default size for our application as well as an acceptable default size for the pop-up windows and error panes in the application.
Workload breakdown
Randy "Superman" Hilarbo: 37.5%
Randy implemented the entirely new Swing version of the View/Regrade interface, to give participants the option of giving feedback on which they preferred. He contributed explanations of this new interface to the Wiki, and also procurred a new bluetooth dongle, after our original was removed from our possession by some scalliwag or ragamuffin after our presentation of the Interactive Prototype.
Charles "the Monk" Leung: 32.5%
Charles implemented the new features of the Proctoring interface, including the new look-and-feel, the timer, error messages, and other usability details. He also performed the experiment with one of the students and contributed the results and other aspects of our final report to the Wiki.
Jonathan "the Assassin" Chang: 30%
Since Jonathan had to be away for much of this past week, he made most of his contributions in writing up the report and participating in post-experimental discussion. He also made minor contributions to the Proctoring interface and performed the experiments with one of the students and the TA.
Appendices
Materials
Forms
Image:Consent form.pdf
Image:Questionnaire.pdf
Introduction Script
Student Participant
Your goal is to take a test. Unlike normal tests you have taken at UCB, you are required to use a special Anoto pen that will send data in real-time to a computer, which is monitored by a GSI. For answering questions, all you have to do is write in the grayed out area on the answer sheet. As long as you write your answers in the answer sheet, your answers will be recorded, if not, the computer will not record your answers.
Unlike normal test-taking systems, our testing interface has a new innovative way to ask GSI questions. If you want to ask the GSI a clarifying question, or need to request something from the GSI (ex: permission to go to the restroom), then you should check the question box that pertains to your question, then write down your question in the large grayed out area, and then finally TAP the send button to send your message to the GSI. If you are unsure whether you sent your question to the GSI, feel free to tap the SEND button more than once.
TA Participant
Your goal is to proctor a test. You will act in your role as a TA to monitor the room for questions from students, to prevent students from gaining unfair advantage, and to grade the exam afterwards. Throughout the exam, you will be sitting at this workstation and monitoring the students from our application.
Questions or notifications or early-starts or late-ends will show as messages in this queue. Clicking the message will bring up a window with the message, similar to a chat client window, and closing the message window will place the message in the history. Messages can be reviewed from the history at any time by clicking them in the history. Students may send questions multiple times; please disregard this as an application bug that we will be working out in the next revision.
Grading will occur with the application in grading mode by simply writing on the grayed out areas of the answer sheet with the digital pen. Please grade as you would normally, and rely on the system to record what your are writing.
Raw data
- Some questions from one participant did not register correctly. This is due to the low quality printout of the Questions UI.
- The participants were not immediately able to follow the instructions in using the Questions UI.
- One participant forgets to tap the send button once in a while when using the Questions UI.
- The participants seem to be focus on their exams - the Answer Sheet UI does not seem to bother them.
- The following are errors done by the teaching assistant in using the proctor interface:
- Clicking the wrong button
- Forgetting to enter a time for the exam - (happens several times)
- Confusion between the message list and the history list in retrieving question messages
- Not being able to see error popups and confirmation messages immediately
- Not closing the popups before interacting with the main panel
- Sequence of actions that are required to be done not mastered immediately
- The following are errors done by the undergraduate participants in using the "View Exam | Request Regrade Interface":
- Attempt to go to the Exam page before logging in
- Pressing enter when meant to press a button (after entering some text - username/password and regrade petition explanation)
- Attempt to interact with the Link/FAQ page which are inactive.
- Clicking on the wrong buttons
- Confusion between the exam and course lists (not immediately able to identify that they are supposed to select an entry in each list)
- The following are suggestions from the participants:
- Make the choice of color consistent throughout the application
- Larger font size
- Provide controls (Ok/cancel) buttons in all popups
- Increase the dimension of the "View Exam | Request Regrade Interface"
- Have a larger default size for popup windows
- Better instructions on the Questions UI
- Increase the size of the 'send' button in the Questions UI
Miscellaneous
- The following is the newly-implemented interface that lets the user View his/her exam as well as request a regrade:
