From CS160 User Interfaces Fa06

Contents 1 Final Report (60 points) 1.1 Team Members and Roles 1.2 Problem and Solution Overview 1.2.1 Problem 1.2.2 Solution 1.2.3 Solution Sketch 1.3 Target user group 1.4 Tasks 1.4.1 Easy Task 1.4.2 Medium Task - Import Test 1.4.3 Hard Task - Manual Grading 1.5 Design Evolution 1.6 Final Interface 1.7 Oral Presentation (20 points) 1.8 Poster

Final Report (60 points)

Team Members and Roles

• Qingyun Tang
  • Coding prototype
  • Proof-reading wiki
  • Wiki write-up
• Anthony Franco
  • Coding prototype
  • Proof-reading wiki
  • Wiki write-up
• Antonis Mannaris
  • Coding prototype
  • Wiki write-up
• Joe Hart
  • Coding prototype
  • Wiki write-up
  • Design Evolution
  • Final Interface
  • Powerpoint presentation

Problem and Solution Overview

Problem

Modern instructors have increasingly time consuming jobs. More students, more classes, more demanding curriculum translates into the simple fact that teachers don't have enough time. A rough calculation can show that an average high school teacher, for example, needs to spend as many as 30 hours per week for test grading and evaluation reviews only! We feel that this is unnecessarily high, and results not only in the teachers' distresses, but also it will reduce the quality of education the teachers are able to offer.

Solution

We propose a tool that will significantly reduce the time spent on test grading and overall test administration for teachers. Built based on the Anoto Pen and Paper technology, our system will provide teachers with many of the advantages of computerized testing (automatic grading, automatic generation of statistics, storing, compiling results over many tests, etc) and without one of their main disadvantages.(the need for students to have their own computer during test taking) Teachers will use their existing tests (or create new ones) to easily make computerized versions of the tests. Once students take a test,(using Anoto pens) their responses will be gathered, compiled, and auto-graded where possible (multiple choice questions). The teacher will then be able to view their responses on a computer, and at the same time complete grading of the rest parts of the test. Once this is done, statistics will be readily available, and all data will be stored for future reference.

Solution Sketch

Target user group

Our target user group was very well defined from the beginning of the project: High School Teachers. From the contextual inquiry all the way to the final prototype, high school teachers provided feedback and assistance in designing a useful user interface. There were some obstacles in using this target user group. We found that getting time with teachers was difficult and could only really be done over weekends when they do not have classes and have more time. Also, we could not meet them at their place of business because of the availability of the proper facilities, equipment, etc. So, meeting this target group proved to be time consuming and difficult due to meeting each separately in separate places (usually their homes) only on weekends. However, for the teachers we met, they responded that they usually prepare and grade tests at home, so we had it right location-wise. Overall this user group provided consistent excellent feedbacks, and even suggested great new ideas for us to improve our system.

Tasks

Easy Task

The goal of this task is to print a prepared test for administration. We choose this as our easy task because we need to print tests after adding active regions and converting into digital format. We expect our users to perform this basic task intuitively.

1. The user will select the "Print" button from the main menu
2. The user will then be prompted to select an imported or old test he/she wishes to print
3. Assuming this is a valid prepared test, the test will be printed for as many copies as the user requests on Anoto paper
4. The program goes back to main menu

Medium Task - Import Test

The goal of this task is to import a new test and create the answer key for it. We choose this as our medium task because it is definitely harder than printing out the processed test, and create answer key for a test is very important. We need to create answer key in order to achieve autograding.

1. The user will click the "Import Test" button from the main menu
2. The user will then be presented with a window containing 4 steps he/she needs to perform
3. The user will have the ability to move forward or backward between the steps
4. The first step in importing is to select a file that is the original test document
5. The second step is to create an answer key for the test. A hard copy will be printed on Anoto Paper and also a PDF file of the test will be displayed
6. The user will be instructed to mark a question area, answer area, and optionally correct answer using an Anoto pen for all questions
7. Once the user is done, he/she will proceed to step 3, which is to review and verify the answer key the system created
8. Finally, step 4 will save the test data, and automatically return to the main menu

Hard Task - Manual Grading

The goal of this task is to complete the grading of a test that has been administered. We choose this as our hard task because it is a very important part in our program. Most tests include not only multiple choice questions, but also short answer questions, essay questions, etc. Teachers still need to manually go over these questions because they cannot be autograded. However, we make it easier for teachers to do so by taking the questions that need to be manually graded out side by side with the students' names.

1. The user will click the "Manual Grading" button from the main menu
2. Program will prompt to select the test that user wishes to grade
3. Assuming this is a valid test and all the necessary data for it can be read, we will then display a table for the grades
4. In the table there will be a row for each student that took the test (with his name) and a column for each question
5. An individual table cell will represent the grade for a specific question of a specific student. There will also be a column (not editable) for the total test grade of each student. Questions that do not have a grade associated with them yet will be represented as empty table cells.
6. The user will be asked to complete the grading of the test (in any order he/she prefers), save the grades and return to the main menu. We believe that the layout of the window makes this task conceptually easy, so the user is very unlikely to be confused. For that rare occasion, the user can click on the help button to view instructions on grading.
7. This task would be monumentally difficult if not for a picture under the table. Initially the picture is a line saying "<select a specific question (table cell) to view the respective student response>"
8. As soon as the user selects a table cell that represents a (student,question) tuple, the picture changes and the student's response to the particular question is displayed. So the user does not need to move back and forth between the hard copy of the tests and the computer to decide what grade to award.

Design Evolution

• How did your UI change from initial sketches, low-fi testing and pilot usability test?

Our initial design had almost nothing to do with our final prototype. We were unclear about how the interface would work, what problems should be solved, how the user should interact with the pen interface, etc. The most major milestone in solidifying our design came from the lo-fi prototype exercise. When faced with actually showing a prototype to a user that actually has to interact with your design buttons start developing, menuing systems start to become clearer, and defining which functions are necessary are critical to the success of the exercise. For example, before the lo-fi prototype, we had not thought that our interface would be built around six-button interface. After finding the six most major functions a target user might want to accomplish, defining a menu system built around the functions became clearer. Thus the drill down button interface was born. After building the interface we thought it would be nice to have nice looking icons for a unified look and feel along with pictorial representations of the functions (see below for more discussion).

Another major leap in our design came after the HelloWorld program. We saw how we could achieve a close interaction between the user creating an answer key and our program in realtime. This realization lead us to design an interface that has the real printed test in the hands of the user along with a digital version rendered on screen. As the user makes marks on the real test the screen is updated with corresponding marks. This allows the user to see how their marks are being interpreted by the program in realtime allowing the user to correct mistakes and help prevent errors and thus reducing the gap of evaluation. Our lo-fi prototype did not have this part of the interface because we had not conceptualized it yet. By our interactive prototype we had this part of the interface partially working but ran into troubles with the toolkit and rendering PDFs as JPGs. Now in this final prototype this portion of the interface is fully working with some user friendly enhancements such as undo and interactive text.

• Show what the major changes were and why they were made

• Answer Key Creation Evolution

This portion of the interface changed the most from conception to final prototype. The core concept behind this function had remained constant throughout the design cycle but the implementation of the interface changed drastically. The concept rests upon the user printing out existing tests they already have created with an Anoto pattern overlay. The user would then create marks to indicate questions and answers to the test. Initially, this mark schema borrowed from the ButterflyNet (ButterflyNet: A Mobile Capture and Access System for Field Biology Research) project created by Ron Yeh at Stanford's HCI Group. By marking "L" brackets around each question and circles around each correct answer our system could create an answer key that could be later used to compare to student tests taken using the same system.

When we moved into the lo-fi prototype portion of the design cycle we "Wizard of Oz"d the Answer Key Creation portion of the interface and didn't give much thought to how the user would understand how to complete the actual creation of a Answer Key. We put a "Help" button in the interface and thought this would solve all issues. However, when actually face to face with users we discovered we had to explain what steps to complete in what order verbally and they were not getting the necessary information from the UI. Users did not know they had to draw L's followed by circles, etc. and were generally lost in this function.

In the interactive prototype we learned some new tricks from the HelloWorld program and decided that the ability to show realtime stroke information on an image in a computer could solve our interface issues. By having the strokes appear on screen superimposed over an image of the test they are currently working on, the user could be prompted for what they needed to do next. After each box drawn around a question a prompt would appear telling the user to now draw a circle around the correct answer, then back to the prompt for drawing the next box and so on.

Feedback from users was good initially with this enhancement but they pointed out that mistakes were costly. If a user made a miscreant mark the interface would interpret this as a question or correct answer mark and move on without the user. They needed some type of Undo operation to sync the user's intentions with the interface. This ability was incorporated into the final design. Also included in this revision was the feature to not provide a correct answer mark for every question. This allows for the user to indicate that a particular question is not auto-gradable.

Initial Design.

Lo-fi Prototype.

Interactive Prototype.

Final Prototype.

• Wizard Style Interface Decisions

After getting feedback from our interactive prototype we discovered the wizard style buttons provided to the user in "Import Test" and "Print Test" were redundant and confusing to users. Originally we had created "Forward" and "Back" buttons for users to navigate from one necessary step to the next. We had this ability in the interface because of an initial suggestion from a contextual interviewee. They had asked us that they might want to skip steps in the interface so they could just do one step at a time on a whole bunch of different tests. For example, a user might want to import a test into our system and not create an answer key for it at that time, followed by importing another test, and so on. In response to this we added the ability for the user to step through the various steps we had created so to circumvent certain unwanted steps.

However, after providing this feature in the interactive prototype we found it did more harm than good. First, our interface requires the user press a button on the current step they are on in order to complete that step. If we provided a next button the user could also press it became confusing whether they should be pressing the step button or the next button to complete the task. Second, by providing the ability for users to skip tasks the system gets very confusing to use. Now the burden of remembering if an answer key had been created for which tests was tossed onto the user. The state of the system could be forgotten from one use to the next and mistakes would be made, confusions would arise, and the user would stop using the system.

Simply by removing the next and back buttons in the interface users are protected from increased cognitive problems and they have clearer tasks to complete on each screen. In the latest version users are ushered through necessary tasks by completing the previous one. This ensures correctness, ease of use, and decreased user cognitive load.

Wizard Style Interface

Latest interface: Without wizard

• Evolution of Icons

By choosing a unified look to our icons we achieve two goals. 1) The same look to icons gives separate functions of our program the feel like they act as one unit. This helps the user by creating a predictable interface that can be relied upon. 2) Provides a pictorial representation of tasks for ease of recognition. After repeated use the user will grow accustom to the pictures for functions rather than needing to read text based descriptions.

Old Icon.

New Icon.

Old Icon.

New Icon.

• Manual Grading Evolution

Our original contextual inquiry didn't contain a manual grading aspect. Even now, if there were a way to rid this function from the system we would do it. The whole point of this project was to eliminate the need for a teacher manually grading anything. So why did we put in a function called "Manual Grading"? It arose as a necessary evil. We realized there would be portions of a test that were not "auto-gradable" by nature.

Our system is built upon the ability to grade simple multiple choice questions. But we found the following questions could not be graded with the system we were using

• Essay questions: Even with character recognition we would have to be able to determine if the answer was written in the matter the teacher wanted. We considered doing keyword search (ie. "This answer is correct if it has the word 'George Washington'") but we found that it didn't have sufficient use cases to justify implementation.
• Graphing: Certain instructors ask students to graph the answer to problems. This would require to much work in determining the axis, labels, and curves. Also, it is difficult to grade since the graphs are hand-written so would never exactly resemble the solution graph.
• Math equations: For math exams, many professors ask students to write down a certain math equation. This requires not only character recognition, but also a sophisticated method of comparing the response to the actual answer due to the simple fact that a single equation can be written correctly in many ways.

So the original concept for the manual grading came from the lo-fi prototype when we walked through the necessary functions. The basic idea is that, even though a teacher might need to manually grade a test, we would like to lessen that burden as much as possible through the use of the pen interface. We realized that we already knew which regions contained answers and we could simply display these answers to the teacher through our interface. This allows the teacher to quickly batch process the grading process by question or by student.

This function evolved from it's inception in the lo-fi prototype to the one implemented in the interactive prototype. In the lo-fi prototype we struggled with if we should display data to the teacher sorted by question or by student. We decided we would try to give the teacher a choice by allowing them to sort by column. When we went to implement this in the interactive prototype we discovered that we could give the users the best of both worlds by having students as the rows and questions as the column. Selecting the intersection of the row and column would show the student's response in a separate part of the interface. This allowed us to present the data in a compact way that was also easy to understand and navigate.

Another small improvement we made was to separate "grade" files into their own directory and have the FileChooser dialog start up in that directory. This way the user has a much easier time picking out the right file.

Lo-fi idea

Interactive Prototype

• Class Setup Evolution

This feature was not included in our lo-fi design as it is not part of our three representative tasks. We did however realize that it is a necessary feature as it is related to grading, statistics and even test taking.

The intention of this feature is to manipulate class rosters. Other than being a useful tool in itself for an instructor, we needed to include student rosters so that we can relate a test submission with a particular student. So we will use the information saved from this module to display student grades. This would also help solve the problem of student-pen associations during test taking. Ideally, there will be a printout of the roster and students could make a mark next to their names just before taking a test with their Anoto Pen. This would create the specific pen-student association and all the input from the pen afterwards will "belong" to this student.

Roster manipulation was fully functional for the Usability Study. From the input we got during the interviews however, it has been drastically changed:

• Users were bothered by the fact that they needed to include student ids. A student id did not concern some of our interviewees and the fact that it had to be a unique field made it worse. We also provided poor feedback if a non-unique id was entered. To deal with this issue, we first thought of hiding the id field and simply autogenerate it and store it in our database. Once we implemented this we realized it had other, more serious problems. During our experiments, one of our users populated the student id as he was used to include it with the student information on most respects. Hiding was not an option therefore since it is generally THE defining piece of data for a student. So instead of hiding it, we give the option to the users to ignore it. A hint on the form informs the user that if they do not wish to enter student ids, they can add names and a student id will be generated automatically. We also included feedback if an invalid id was entered.

Feedback

• A second, more serious problem was that now we had no clear way to enroll a student in multiple classes. Instead we saved the same student name under multiple entities. This was bad for our system if we wanted to to get all the grades for a particular student. Related to this was the fact that users did not like having to type in student information more than once. At first we thought we should enable copy/paste between classes but this was both hard to implement and confusing for a user. So instead we decided to maintain a master student roster, where users could edit student information and then from there add students to individual classes. This, combined with the student id field which we maintain, enabled us to easily enroll the same student entity into more than one class.
• One of the interviewees also remarked that he was generally worried with programs crashing before he could save changes. Even without such problems, if the user accidentally closed the application, the changes would be lost. The reasons we didn't save automatically for our Usability Study was firstly efficiency and secondly to enable the user to discard bad changes. We decided to change this however because we felt that the need to click "Save" or answer yes to a popup was unnecessary and time consuming. Instead we save automatically (not as tie consuming as we thought it would be) and only request verification for actions that are not easily reversible. Namely, deleting a class and deleting students. All other operations are extremely fast to do, so even if a user regrets an actions he took, the consequences are small.
• We also made a small change to satisfy a user request, who said the classes list should be sorted or sortable. We felt it was sufficient to sort it alphabetically once the module was loaded.
• Instead of having a separate Help window, we felt it was better to include all the helpful needs on the form. We feel that these are enough for the user to immediately know how to use the module. We also added the ability to delete students and drop students from a class.
• The buttons on the form were made bigger and include more text. Beauty was sacrificed for usability. We needed to make sure the distinction between the two tables was clear. We also tried to utilize the jswing features to help the user as much as possible by disallowing actions like editing the class roster table, adding listeners to selections etc.
• The popup windows were adjusted to open up close to were the user clicked the button. This was to reduce the distance the mouse needs to travel in order to respond to the popup.

Usability Study

Final Prototype

• Explain which evaluation technique was most valuable to your prototypes usability and why?

Even though the lo-fi prototype revealed some major flaws in our design and helped us decide the overall structure, the Usability Study (and the hi-fi presentation) was the one that revealed most of the problems. Unlike the lo-fi interviews, the testing during the Usability study was less awkward (no pieces of paper flyng around) and the users felt more comfortable. They were really more strict about the design as they had higher expectations. Comments we got from the usability study indicated that some modules needed complete revising (like the class setup) while others were better and needed little change (grading). During the usability study we also had the chance to observe finer details like the time taken to complete tasks, how the user behaves in front of the computer etc. For example we figured out that the wizard type interfaces were causing a bit of confusion, and that the class setup form was overloaded and unpractical. Furthermore, during the usability study we got a chance to ask users what their opinion was on the look and feel of the whole system. Finally, the usability study, although our sample group was very small, helped us estimate the computer knowledge of our target users (navigating the file system, meaning of terms, printing etc). We decided it makes more sense to use common language and avoid computer lingo, as many programs so often do.

Final Interface

• Functionality Description

Import Test

Import test is the main entrance into the ET system. A user selects an already created test file (must be in PDF format) off of their hard drive. They then are prompted to create an answer key for this test using the Anoto pen interface. Third they are asked to review their answer key. Lastly it is saved to disk in a special folder created for this purpose.

Print Test

Print Test is where a teacher would go when they are ready to administer a test to their students. They first select a PDF document from their hard drive of the test they wish to administer. This PDF has the Anoto pattern overlay from the Import Test step above. The second step of this procedure prints this PDF to their standard printer.

Browse

This is where a teacher can see which tests they already have imported into the system. It is a simple interface that browses the area on the hard drive where Import Test creates all of the necessary files.

Class Setup

Users can add, edit, delete student names and if they want ids into a master student roster. They can also add or remove classes and then enroll students from the master roster into their classes.

Manual Grading

This is where users see and edit test grades. While autograded questions are assigned points, other questions need to be graded by the teacher.

• Interface Design Description

Import Test

Import Test is broken into 4 main steps

Step 1

Step 1a

1)Select Test from hard drive. This is where the user selects an existing test PDF file from their hard drive.

Step 2

Step 2a

2)Create Answer Key. This is where the user uses the test printed with an Anoto pattern overlay to interact with the ET system to create an answer key that will be later used to compare against student tests.

Step 3

3)Confirm. This step confirms the strokes made in the previous step. If the user sees a mistake they can go back and create another answer key.

Step 4

4)Save. This step simply saves all necessary information about this test to disk in a special directory created for this test.

Print Test

Print test is a simple interface broken into two steps

1) First the user selects which test they would like to print from the hard drive. The interface defaults to the test directory created from the Import Test steps.

2) The user is presented with the system print dialog box for standard printing of the test.

Step 1

Step 1a

Step 2

Browse

This interface allows the user to see what tests they have already imported in the Import Test step. They can delete tests here and rename directories if they wish as well.

Class Setup

Here we see the interface used for inputting class information.

On the left hand side there is a list of classes. Under the list there are two buttons to add or delete a class. Deleting a class asks for confirmation while adding a class asks for a unique class name. The popup will indicate an error message if a duplicate class name (or empty) is entered.

Once the user selects a class, its roster is shown in a table in the middle of the form. The table has two columns, student id and name. The user can sort the table by any column but cannot edit this table directly. Under this table there is a button to drop selected students. The table allows multiple selections and the selection mode used is by row. So if the button is clicked, any selected students are dropped from the class.

In order to edit student information the user is instructed to use the master roster. This is yet another table next to the class roster. To add a student, the user can insert data into the name or id fields of the last table row, which is always empty. If a name is entered, an id is generated automatically. The user can also edit the information of existing students. In that case, any classes in which the student is enrolled will reflect the change. Finally, the user can delete students by selecting one or more rows and clicking "delete students". After a confirmation, the selected students are deleted from the roster and the classes they are in.

To enroll students into the currently selected class, a user needs to select one or more rows from the master roster table and click "enroll". Any selected students that are not already enrolled in the class will be added.

Everything is saved and updated automatically upon edits and actions. This way there is no need to include a "Save" option.

Manual Grading

The grading form is very simple and practical. The grades are output to a table where rows are students and columns are questions. So a cell represents a question grade for a particular student. Ungraded questions are empty.

When the user selects a cell, the image of the student response is shown under the table. This enables the user to easily review the answer without going back and forth between the hard copy of the test.

Once the user changes the value of a cell, the new grade total, indicated at the end of each row, is computed.

There is some general information about the test and question on the right and a help button for more detailed instructions.

At the bottom, the user can save the changes made, return to the main menu or grade another test. If he tries to leave without saving, a popup window appears to verify.

• Describe your implementation. What was most difficult to implement and why?

Import Test

The challenge here came from the need to change a PDF document into an image so that we could superimpose user stroke information over that image. This required going out and getting an open source package called jpedal, learning how to incorporate it into the project, and finally learning how to use it. Once we got this all working we realized that a teacher might have a multipage test and our interface only allowed for the first page to be written on in realtime. Even though we see how we could implement a multipage interface for this time became an obstacle to this and we decided to leave it at one page support only.

The second challenge was implementing an undo function into the user mark area. The state machine required to keep track of questions and answers became very confusing and hard to implement. However, we recognized this as a necessary feature and made it work eventually.

Class Setup

The harder part of this module was the UI itself. The need to account for as many of the events as possible meant we needed multiple listeners on the various JComponents. The logic of the feature worked on a database like system where roster data were saved under the folder "data/classes/. This folder always contains two files. The file classes has a list of class names. The file roster has a list of <id>,<name> pairs for each student. For each element in classes, there is a corresponding file. This file has a comma separated string of ids, representing the students that are enrolled in the class. Using this file system, we retrieve all the necessary information to output the form. The tricky part was making sure the file system couldn't be corrupted and all the data were recorded and saved correctly in real time. Also hard was to come up with the proper structure of this form, since our tests had some contradicting results. For example, one user didn't want to have student ids while others assumed and expected it.

Manual Grading

For the output there is a folder data/grades/ where gradedata are saved for each test. If the user chooses to grade a test, we first need to process the student annotations and try to autograde any multiple choice questions. To do this we compare the answer key generated during the test import with the annotations of each student. If in the answer area the only annotation is close to the correct answer area, we indicate this answer as correct. Otherwise we mark it incorrect. For questions without an answer key, we make no effort to grade it.

When the user selects a cell, we need to produce an image of the student answer. This is done by rendering a PDF from the student response, getting the coordinates of the question area and producing an image out of the PDF over those coordinates.

Autograding and image generation were hard to implement due to the fact that they were "new territory". Despite our best efforts, the autograding is not perfect and may result in incorrent output.

• What was left unimplemented
  • Class Setup - Renaming Classes. The reason we left this out was to make the form as simple as possible. This feature seemed unimportant so we ignored it.
  • Manual Grading - Multiple Students. Partly because we only have one pen and partly due to the complexity of distinguishing annotations between pens, we couldn't implement the functionality of having pen-student associations and multiple students taking a test. For this reason we also left out the idea of printing a roster on Anoto Paper and having students mark a checkbox next to their name to associate the pen they have with the student.
  • Import Test - Multipage imports of tests. Because of the complexity of converting a multipage PDF to multiple images and then getting those images to correctly correspond to the user strokes as they were created, we decided to leave this feature out. Only one page user tests are permitted at this time.
  • Statistics - This was the lowest prioritized item in our project. While it would have been nice to have pretty bar charts and graphics it was not identified as critical to getting the system working. Simply stated, if we had more time this module would have been implemented in its entirety.

Oral Presentation (20 points)

• Media:Final_Prototype.zip
• Media:GroupETcode.zip

Poster

• Media:ET_Poster_Final.zip

---