LoFi-Group:BigBrother
From CS160 User Interfaces Fa06
Contents |
Group
Randy Hilarbo : Method, Consent form,Script Reader/Observation,Prototype
Charles Leung: Discussion, TA/PC,Prototype
Leo Chen: Draw, Prototype, Results(1)
Jon Chang: Results(2,3), demo script, Prototype
Introduction and Mission Statement
This system is a modification of the existing process of university examinations that adds useful and efficient features for the benefit of all parties. We will be focusing our design on student interface, since we are most personally concerned with their needs. Also, the student interface makes the most use of Anoto digital pens, the theme of this class. In short, we will design paper tests that use the Anoto pen to digitally record everything written on them, then use this information to facilitate both taking and grading exams. Our idea is that only the material on the test should be difficult, the actual process of taking a test should be as pain free as possible. As such, we intend to streamline the entire testing process, from the actual test taking to anything else that follows.
Our common purpose and goal for this project is to achieve a functional, small-scale proof of concept of this application that showcases the vast potential of Anoto-enabled digital pen technology. While we have many good ideas, we understand that we are limited by the time and resources allotted by this one class in this one semester, and so will strive to refine only one of our original three user groups, applying the concept of this class to design an optimal user interface.
Prototype Description
The following demo script was read aloud while presenting the 3 different tasks to the target user group: the test taker.
Our project is unique in the fact that the Anoto acts as a interface between two humans. So, our demoer also doubled as a virtual TA in certain parts of this.
Low-Fi Prototype
Our focus in this particular session was to test 3 parts of our system, taking the test, asking a question during the test and students getting regrades. So, in order to meet try out these three aspects of our system, we created a mock test for a student to take as well as a mock web page (on a piece of paper) that they could use to check their score and request a regrade.
For this particular test, we had one person act as the TA or computer and one act as the reader of the demo script. If a student checked a the question box, the person acting as the TA would wait 30 seconds, then come over and respond to the question written inside the question box question area. We purposely "smudged" a part of our test to try to force the student to use the question box.
Once a student was done, the person acting as a TA would become the computer. He would hold up the "Web Page" with the student's finger acting as a mouse on the screen. The student would then observe the fact that the professor or GSI had obviously misgraded a question on the exam, and fill in the regrade request. The person acting as the computer would beep when the wrong password was entered on a paper keyboard, and would flip the page he was holding up to the correct page when the student would click on a button.
Overview of the Entire System
Test
As noted above, we purposely smudged a question in order to force the student to use the question page of the test.
Questions Page
When a student would check the question box, the person posing as a TA would wait 30 seconds, come over and answer the question written in the box. Students thought it was funny to ask where babies came from.
Grading Website
One of the pages of the website was the student's actual graded test. So we would actually mark up the test they just took as a representation of this. Everytime we would grade the test, we would purposely mark a question wrong.
Login Page
A successful login would make the PC flip to the Tests Page
Tests Page
This displays all the available tests the students can view.
View would take the user to the copy of his test all marked up, stats would show the stats for the exam and regrade would take the user to the form to request a regrade.
View
This would be a marked up version of his exam with one of the questions obviously marked incorrectly. The difference would be at the bottom of the exam page we would have taped on Regrade, Stats, and Main buttons. Also at the top of the page we would have a box displaying the student's score.
Regrade would redirect them to the regrade page. Stats would redirect them to the Stats page and Main would redirect them to the Tests page.
Stats
Regrade
Demo Script
We're designing a new testing system that integrates digital pen technology into existing tests to better serve students. We’re hoping to give you some features you didn’t have before that will be useful and increase testing and teaching effectiveness. With these goals in mind, please take a look at the test before you. As we talk, feel free to say anything that comes to mind. We especially appreciate negative comments that give us guidance for improving our ideas and interface.
Answering a problem
So the first thing you’ll notice is that the questions are separate from the spaces provided for answers. It’s important for the application that you fill out the question and solution boxes in the solution spaces, otherwise the data won’t be organized properly. How does this seem to you so far?
Asking a question
If you have a question during the test, we’ve provided a page in the back of the test packet for you to ask questions on. All you need to do is write your question in the space and check the box, and the question will be sent to your TA or GSI. They’ll either come around to answer your question personally, or if they receive enough of the same question they’ll just announce the answer generally to the class. You can also report cheating or other sensitive comments in this way. Either way, you can trust that when you’ve checked the box, whatever you’ve written in the area will be read by at least one of the test proctors. Does this interface make sense? Would you use it? Would it interfere with your test-taking?
Student takes mock test
With the amount of explanation I’ve given you so far, would you be comfortable taking this test? Please take the test now, and use the question and answer pages as if this were actual testing conditions.
Checking Score
Another aspect of our application that relates to you as a student is how you check your tests for regrades. In our system, tests are also graded with a digital pen and the strokes recorded with your own into digital records. After grading is completed, students would be able to go online and retrieve their tests in an interface that allows them to compare an answer key to what they wrote. If they still feel that they would like a regrade, then there is a mechanism in the website to submit a request with a reason for the regrade. This would also prevent people from submitting false regrades, since we’ve recorded everything they wrote at the time of the test.
Your password right now is your last name, but of course in the real system you could change it to anything you would want.
Student interacts with "Web Page"
Does this sound like something you would use? Does the interface we’ve provided work for you, or do you have suggestions?
Methods
Participants
In this assignment, we decided to limit the tasks we are addressing to the tasks that the students assumed during an examination. As a result, we omitted the GSIs and the professors from our original target users and decided to focus on students. Three students have been asked to participate in the low-fidelity prototype testing of our interfaces. One of the participants interviewed (subject 1) from the previous assignment was retained to take part in this process. To remind us, he is a second year undergraduate student in the College of Engineering here at Cal. Part of his schedule this semester includes two large-size classes. This participant has a good computer background. From the previous interview, he complained of GSIs not being able to address exam inquiries in a timely fashion. Subject 2 and 3 are also undergraduate students of UC Berkeley. Similar to Subject 1, each one is also taking an impacted lower division course. Similar to subject 1, we also consider subject 2 as an expert computer user, averaging 4-6 hours in front of a computer everyday. On the other hand, subject 3 does not consider himself as an expert computer user. While subject 2 admits that he hesitates to ask a question during an exam, subject 2 complains of not having his hand (raised for inquiry) immediately noticed by the teaching assistants or the professor monitoring the exam. All subjects also complain of the limited opportunity of asking for exam re-grades. Subject 1 told us that his Math GSI only give them ten minutes to look at their graded exam and ask for a re-grade. On the other hand, subject 2 hesitates to ask for an exam re-grade because of the possibility of also losing points for having his exam re-graded; that is, his professor discourages re-grade petitions by having the graders carefully re-grading the exams looking for all kinds of grading errors. In the selection of the participants for this assignment, we hope that the differing computer backgrounds of the participants should adjust the usability of our interfaces in terms of being easy to use for those who does not have enough experience with computers while not being too easy for those who are expert computer users. We also hope that each subject's experience and opinion about taking and asking for re-grading of exams would yield different opinions on our interfaces and will therefore help improve our initial designs. Overall, we feel that we have a good choice of participants for this assignment.
Environment
Our first low-fidelity prototype test was held in Subject 1's apartment. The room was quite small and five people crowded the room. However, we managed to setup the prototype while giving more room to the subject. We placed a table in the middle of the room and pulled up three chairs for the subject and two of the testers. The prototype is then laid out on the table where the current interface is facing the subject while other parts of the prototype are stacked on the sides. 1 rough sketch of the setup is illustrated in figure 1. Since subjects 2 and 3 know each other, both requested to be tested in the same meeting. We allowed this since we thought that this might help each other feel comfortable of the procedure. However, we made sure that subject 3 remains clueless of the prototype until subject 2 finished the process. The testing occurred in a classroom which has a bigger space than subject 1's testing environment. The setup of the prototype is almost similar to the previous setup. Figure 2 illustrates this environment.
Tasks
Easy: Asking Questions
In the current exam setting, students only have to raise their hands in order to let the monitoring figures know of their inquiries about the exam. However, this current method does not always guarantee that a student’s question is immediately attended if it is attended at all. The reasons why this is includes, based on the results of our contextual inquiries, the inattentiveness of those who are monitoring the exam, the invisibility of the raised hands to the monitoring figures due to the large size of the class as well as the distant position of the GSIs/professor relative to the student with the raised hand.
Our low-fidelity prototype provides a better alternative to raising hands and still raised the guarantee that a student’s inquiry will be attended. In performing the task of asking questions using our prototype, we asked the participants to each perform the following:
- Distinguish the prototype used for asking questions from the answer-sheet prototype
- Use the prototype without us telling them how to use it
- Use the prototype after telling them what it is for
- Ask questions using the prototype
Medium: Taking the Test
This task has been previously categorized as easy. However, after a little evaluation, we raised the difficulty of this task to medium. Aside from refocusing our tasks to a specific target user as a reason for this decision, we also realized that adopting a new technology to perform a task that has been natural to everyone can be a challenging transition. In the current exam system, students are provided with a sheet of questions along with an answer sheet and a pen or a pencil. Our prototype requires our users to use the Anoto digital pen along with a customized answer sheet. Moreover, this task also requires our users to feel comfortable with the answer-sheet prototype. To determine how comparable our prototype with the existing tools of performing this task, we asked the participants to perform the following tasks:
- Distinguish the prototype used for taking exams from the answer-sheet prototype
- Answer sample test questions (both easy and hard questions)
- Take a timed quiz
Hard: Reviewing the Test/Asking for a Re-grade
From our interviews, we found out that the current way of performing this task involves students submitting their exams subject for re-grades along with an explanation of the re-grade petition. In some classes, the GSI only gives students a limited time to decide whether to ask for a re-grade. These comprise the reasons of why we identify this as a hard task in the current exam setting. The students are required to follow certain methods (i.e. prove necessity of a re-grade through writing, etc.) just to challenge the consistency of the graders in marking exams. In terms of this assignment, our low-fidelity prototype for performing this task may require the most effort from our users. Our users need to interact with a computer in order to complete these tasks. To thoroughly test the usability of our prototype, we asked the participants to complete the following tasks:
- Interact with the prototype without us telling them what it is for
- Interact with the prototype after telling them what it is for
- Check an exam score
- Submit a re-grade petition
Procedure
1. Roles and Preparation
The roles played for the low-fidelity prototype experiments are assigned as follows:
- Jonathan – facilitator/greeter
- Charles – computer/observer/greeter
- Leo – computer/observer/greeter
- Randy – observer/greeter
To prepare for the experiments, each of us made sure that we know our assumed roles. We did a quick round of a mock low-fidelity prototype testing in order to realize our roles. We then discussed the kind of information (i.e. test measures) that we wanted to acquire during the testing.
2. Setup
In the first experiment, we were welcomed by subject 1 into his apartment. After some refreshments, the facilitator explained to the user an overview of our project without giving descriptions of our prototypes. We then presented a quick demonstration of how would the experiment go. After making sure that the participant understands the procedure, we then assumed our positions based on our roles.
With the prototypes laid out on a table, the computer sat across facing the participant while the facilitator stayed beside the subject. The observers stayed on the sides as to not distract the ongoing experiment while having a clear view of what was going on. The facilitator reminded the subject to voice out his thoughts before the actual testing began.
In the second session, we met up with subjects 2 and 3 in a classroom. The same setup was done. This time, Charles and Leo alternated to play the computer role. Also, there is only one observer at this time since the other person makes sure that subject 3 will remain clueless of the prototypes until subject 2 finished.
At the beginning of each session, the facilitator presented and explained the informed consent form. Each participant is then asked to sign it.
3. Testing
During the actual experiment, only two persons remained talking: the participant expressing his thoughts about the prototypes and the facilitators constantly encouraging the user to talk out loud. The facilitator and the computer avoided guiding the participant through the process as much as possible despite the subject’s constant inquiries of how to do what. The facilitator also decided when to transition from one prototype to another. Before going through the next prototype, the facilitator also made sure that the experiments cover everything that we wanted to be evaluated about the prototypes. In this process, the facilitator now guided the participant by having him performed the methods (described above) that were not yet covered. Throughout the procedure, the computer tried to react as quickly as possible to match the user’s actions. The observers stayed quiet on the side while timing each task performed and counting the errors made. Two rounds of low-fidelity prototype testing are performed for each participant.
4. Debriefing
At the end of the experiments, each participant is then given a questionnaire to fill out answer regarding the prototypes they just encountered. We also asked them questions that were brought up during the experiment. After the session, the team discussed the information gathered from the experiments
Test Measures
Before starting the experiment, we outlined several measures to look for during this low-fidelity prototype testing. The acquired results for the following provided us with a good evaluation of our interfaces: 1. Usability of the interfaces – We evaluated how complete each performed task is and if each can be performed under a desired time. 2. Comparison of prototypes to the current tools in performing the addressed tasks above – We determined this by observing each participant’s reaction to the first method of each task described above. 3. Confusion – We measured this in terms of the amount of errors produced during the experiment. 4. Speed in learning to use the prototypes – For this, we kept count of the number of iterations performed by each participant in mastering the tasks. 5. Personal opinions – We took note of each participant’s comments during a quick interview upon finishing the procedure.
Results
Reactions to the application were generally positive, even enthusiastic about the possibility of having it actually implemented for their exams. While there were some hesitations about the actual test, they seemed to feel that the benefits of the application outweigh their issues and that they would be willing to use it if implemented.
Easy Task: Asking Questions
With all three subjects interviewed, there was unanimous consent that they liked this feature, found it easy and intuitive to use, and would use it if their own exams included such an option. They also did not feel that this detracted from taking the exam; indeed, they felt instead that this was a better option than the conventional methods of raising their hands, or getting up to go ask TAs or GSIs in person. While none of them had ever observed cheating in their classes at Cal before, they felt they would be willing to report it using this method.
Moderate Task: Taking the Test
There was some diversity of opinion on the execution of this task. While two subjects found the low-fi exam to be similar to exams they had taken in other technical classes, like lower-division physics and math, one subject (1) was visibly put off by the appearance of the exam. Our group had decided it would make more sense for the implementation to have a sheet of questions, then a packet of identical answer pages that could be associated with a question. Subject 1, however, preferred to have the questions, even the individual parts of the question, more closely associated with his answer spaces. He and another subject (2) were also a little uncomfortable with the requirement of using a pen, since both were used to testing with a pencil and being able to erase their work. Our group had considered eraser capability and even conceived of a potential implementation, but eventually decided the following through was probably beyond our abilities in the timeframe of this class. Both agreed, however, that they could eventually get used to such a system if it was implemented. Subject 2 also inquired as to how much a pen costs, and whether they would be provided by the school or not. All three expressed a desire not to pay for the pens as a requirement of the class.
Hard Task: Reviewing the Test
This task was defined as hard because there is no precedent for an interface with this goal in mind. There didn’t seem to be a solid process for resubmitting exams for a regrade, allowing room for students to take advantage of the system, inefficiency in the process, or just things slipping through the cracks. All subjects responded positively to the interface for independently comparing what they did on an exam with an answer key through an online application. Subject 3 commented that this was a good way to deal with the type of people that attempt to take advantage of the situation by either writing in or changing answers, are just arguing at length with graders in person until they give in. Having an electronic system for both letting students examine tests on their own time and submit formal requests for regrades will cut down on staff work and reduce trivial regrade requests.
Discussion
From our interviews we discovered that there was in fact strong demand for some of our features, especially the messaging, anti-cheating, and regarding ones. Although some of the students were apprehensive at first, once we informed the interviewees of the new features available with our new BigBrother test format, they quickly warmed up to learning the new test format. It also seems like all the students believed that they would be able to adapt to this new test-taking format if it were widely accepted.
After our interviews we learned that while all our group members easily understood our UI, the interviewees were intimidated and unsure of how the messaging system worked. Although they quickly learned how to use the UI system after some simple instruction, we realized that personally walking through every student individually would not be feasible in a large classroom setting with time constraints. We decided that in our future designs, we should include a short written description of what exactly the messaging system is expected to be used for and how to use it (reporting cheating, asking questions, etc). The messaging system is in fact rather basic, but since the students were not used to communicating in this way with their GSIs, there needed to be clear instructions on what exactly they should do. If there are clear written instructions given to the user, then the user can always fall back on the written instructions if they get confused or forget what to do.
In addition to the messaging system, our interviewees also displayed some apprehension over whether the records of their tests would be lost by our new system. Although the students calmed down after we reassured them that there would be both digital and physical copies of their tests, we also realized that this could become a significant problem if our system is newly adopted and a large number of students were to use it in a real test setting. In our next UI design, we may add more written instructions to reassure and clarify what exactly our new system will do in order to reduce confusion.
Our experiment helped us learn new things about our UI, but our low-fi tests could not reveal how easy our system actually worked. For example, we assumed that our messaging system works exactly how we defined it, but we still cannot be sure how user-friendly our end product will be (i.e. how careful does the user have to be when checking the box?, is the box too small to be easily recognized by our software?).
Appendix
Critical Incidents
Subject 1
- Asking questions is easy
- Exam looks weird, hard lines bad? : 3
- Wants questions integrated with exam : 3 (hard to implement)
- Wants pencil : 2 Anoto doesn't make pencils
- Webpage seems simple
Subject 2
- Asking questions is easy
- Prefers pencil over pen : 2 Anoto only uses pen
- Curious to see how much pen actually costs : 1 We're being forced to use Anoto
Subject 3
- Asking questions is easy
- Good idea on the regrades interface
- Web page kind of simple : 2 It's a mockup
Survey
Test
Aesthetical view of test:
With the amount of explanation I’ve given you so far, would you be comfortable taking this test?
Question Asking
Does this interface make sense for question asking?
Would you use it?
Would it interfere with your test-taking?
Web Page
Does this sound like something you would use?
Does the interface we’ve provided work for you, or do you have suggestions?
Consent Form
Lo-Fi Prototyping: Consent to Participate in Research
Purpose
Team BigBrother is seeking to apply the Anoto digital pen technology in examination. In order to develop a useful and user-friendly interface, the team needs to conduct a real-world test of the design at its early stage. The subject’s interaction with the design will be examined in this study.
Invitation to Participate
The person whom this consent is written for is invited to participate in the study because s/he is part of the target users that the team’s interface is aimed for.
Procedure
• Four people will be present during the study to monitor it. The study will be held in a place that is convenient for the subject. The subject in study may be given a questionnaire to answer at the beginning of the session. The subject will then be shown a demonstration of how the procedure will go. The person in test will be presented with a design created in paper. That is, s/he will be required to interact with a paper interface. S/he is asked to pretend that s/he is interacting with a real interface. One person from the tester’s side will act as a computer to carry out the actions triggered by the subject’s interaction with the interface (i.e. if the subject click on a button, the tester would perform the action on which that interaction with the button is supposed to perform). The designers may make changes on the design immediately following the test. If this is the case, the subject will again be ask to undergo another round of similar procedure. The procedure will end with a series of questions from the testers about the subject’s experience with the design.
• Again, the purpose of this study is to test the interface and not the subject in testing. In order to attain the desired results of the study, the subject is asked to pretend as if s/he is interacting with a real interface rather than the actual paper one. The subject is also expected to comment on the design as well as on every interaction that s/he has with the interface if possible.
• The procedure is estimated not to take more than an hour.
Risks/Discomforts The interface on which the subject will be interacting with is in paper form. Therfore, there is a minor risk of acquiring paper cuts to the subject. The testers will make sure to observe measures in order to minimize this risk.
Benefits
There is no direct benefit to the individual subject is anticipated.
Alternatives
The development of the interface resulting from the study provides a useful benefit for the subject in testing as well as the group of users that the subject belongs to.
Confidentiality
The team is mainly interested on the results of the procedure. The subject is assured not worry about having his personal data disclosed to any third parties. When presenting the results of this study to the third parties, the team will anonymize the subject’s data. The team will take all possible measures to protect the identity of the subject in study.
Discontinuing Study Participation
If a subject, for any reason, decided to withdraw his/her participation to the study at any time, s/he will be allowed to do so.
Compensation/Payment of Subjects
The team will not be gaining profits in any way as a result of this study. Therefore, there will not be any compensation that will be offered to the subject.
Questions
If the subject has further inquiries regarding this consent form, please send an email to any of the following:
- Randy Hilarbo
rhilarbo (at) berkeley (dot) edu
- Jonathan Chang
jiajih (at) berkeley (dot) edu
- Charles Leung
charlesleung (at) berkeley (dot) edu
- Leo Chen
leochen (at) berkeley (dot) edu
Voluntary Nature of Research Participation
Participation in this study is strictly voluntary. The subject is free to refuse participation in any part of the study anytime.
Consent/Signature
The subject will be given a copy of this consent form.
By signing below, the subject verifies that s/he reads and understands the information presented in this consent form and that she agrees to participate in the study.
Name (please print): ____________________
Signature: _________________________________ Date: ______________________
