LoFi-Group:ET

From CS160 User Interfaces Fa06

Jump to: navigation, search

Contents

[edit]

Introduction and Mission Statement (6 pts)

Our project is an Anoto-enhanced teachers' grading assistant tool. The demands on the modern intructors are many and difficult. A wikipedia article, current issues in teaching, paints a picture of the varied demands for a teacher's time. A typical teacher has 5 classes a day with ~30 students in each class totaling ~150 students to: administer testing to; evaluate performance; keep records of a students performance. Therefore, our project is targeting high school teachers who spend a lot of time grading students' quizzes and exams, and then organizing them into grade books. They should be spending more time on students instead of spending time doing grading and paperwork.

We are conducting this experiment to better understand the needs of a teacher's grading system. Through Low-Fi prototyping, we can discover problems and ambiguities in our grading assistant application. We will use this feedback to make early rapid prototyping decisions to decrease our development time, increase productivity, and provide a product to teachers that is more on target to their needs.

Mission Statement: To increase the productivity of high school teachers by lessening the burden of paper work to allow them more time to focus on students.

  • "Computer" role: manipulating paper interfaces to the users
  • Observer role: observe and note-taking during inqueries
  • Greeter role: hand out consent form, introduce the project
  • Observer role: observe and note-taking during inqueries
  • Facilitator role: ask users questions during inqueries
[edit]

Prototype (12 pts)

  • Main menu serves the function of directing what task users could do
  • Each rectangular box is a clickable button
  • When clicking each box, it goes to the sub-menu respectively
  • The top left line serves as directing purpose. It will be on each page to allow users to jump to upper level pages
  • Import test serves the function of converting word or pdf documents into Anoto-based tests
  • Each box is a step to complete the task in a wizard mode
  • During each step, the box is highlighted and the other ones are greyed-out
  • The help button gives instructions of how to mark on the printed test (middle of DSC00070.jpg)
  • Teachers are expected to only read "help" once and be able to perform the task in the future
  • The back and next buttons help to go back and forward among the steps, so if a test is not printed successfully, user could come back and print again
  • When done importing, the screen shows "test successfully imported" as in (DSC00071.jpg)
  • This function will organize the tests and allow users to browse old tests in the database
  • There are different sortings in this functionality, such as sort by subject, date of creation, name of tests, etc
  • There is also search function
  • (top picture of DSC00070.jpg)
  • Consists of two steps which are highlighted or greyed-out when we are on the respective task
  • Step 2 is just going to be printing the test out; it is a very important step in our test creation procedure
  • There are 3 different functions in this menu
  • Add class will add a class into the database
  • Add student will add student(s) into the databse within each classes
  • Browse will go to each class and look for a specific student or the performance of the entire class; it will be associated with the grading and stats section
  • Clicking of any of the three will go to one picture of the following (DSC00069.jpg)
  • Grading
  • There will be two types of grading, auto grading and manual grading
  • Auto grading will be on problems like mutiple choices and manual grading is on everything else like short answers and essays
  • Manual Grading (DSC00066.jpg)
  • The fields on the paper will be divided
  • We will have each students' names be on the side of their problems if the teacher tries to grade a same problem on all tests, it makes the teachers' job much easier without doing the actual sort themselves
  • Other options are available too, such as sorting with name and the entire test
  • The automatic grading part will check the mac addresses of pens and associate them with each students.
  • The counter box shows how many have been processed, it stops when auto grading is done
  • Finish button closes the auto grading
  • We can look up or modify grades and stats for a single student, class, or a test
  • A screen like this will show up when we go to look up for stats of a student (bottom of DSC00070.jpg)
  • The back button goes back to grades & stats or we can use the chain menu on the top
[edit]

Method (12 pts)

  • Participants: (1 paragraph)
    • Customer 1 - Jane
Jane participated in our Contextual Inquiry. Jane is a high school chemistry teacher with 6 classes of standard chemistry. She is a 30 year veteran with a varied background teaching in many school districts and teaching many different subjects. She currently uses a computer based program to track students and their grades in her classes. We thought she could provide good feedback for our application since she is familiar with computer based "gradebooks" and already understands what works best for her and what she has come to expect from new systems.
    • Customer 2 - Jack
Jack is a begining teacher at the high school level. This is his 3rd year teaching science and math subjects. He is rather computer savy but is far from a programmer, we can say he is computer literate. We chose Jack because A) He was available and agreed to our study and B) He is probably most representative of the future teaching workforce. Jack's feedback could potentially help us to see where this technology should be in a few years.
    • Customer 3 - Theo
Theo is a GSI at a college and will soon be a professional instructor. He has excellent computer skills and is also very busy. He needs to teach for financial support but his main focus is his pHD. He is therefore interested in a technology that would save him time and effort on administering and grading tests. We chose Theo in order to get a range of teaching styles and test administration styles to better help us make a general tool that is useful to all teachers.
  • Environment: (1 paragraph)
We visted each of our participants in a home environment seated at a large desk. Qingyun acted as the computer, Anthony as the observer, Antonis as the greeter, and Joe as the facilitator. Qingyun sat across from Jane, Joe sat next to Jane to get a better perspective on what she was seeing, and Antonis and Anthony sat to the side as far out of sight as possible. This allowed for as fast of a "computer" as possible with minimal distractions from group members. This also kept our participant from checking our faces for feedback on her actions.
  • Tasks: (1 page)
    • Import Test (Medium Task)
For this task the user is supposed to import one of their existing tests from their filesystem to our application. This involved first, understanding why one would want to import a test into our system and second, navigating our button type interface.
First the user needs to identify the "Import Test" button on the main menu screen and press the button. Then the user will follow a wizard type process of 1) choosing an already created test sitting on their hard drive in any format. 2) Print out the test on anoto paper and mark to create an answer key. 3) Sync the pen back to the computer and verify the marks given in the previous step. 4) Save the test in our application with associated answer key and meta data such as number of points and date.
  • Print Test (Easy Task)
This task meant for the user to print out a test for administering to students in a test environment of the teacher's choosing. This task involved understanding the adminstration of test paradigm and how to navigate our user interface to complete this goal.
First the user should identify the print button on the main menu and press it to move to the next menu. In this menu the user should follow the wizard type interface to bring up the print dialoge box and print out however many tests are needed to be administered. At this point we will simulate the tests being printed, handed out to test taking students, and returned to the customer in pen and paper form.
  • Grading (Hard Task)
The final task we had the user preform is the final grading of tests and finalizing the grades to a gradebook in our application. This task required the teacher to finish the test creation -> test administration -> grading loop our application is centered around. This task also required preknowledge of the previous tasks before being able to complete this task since it integrates several main concepts together. The task involved interfacing with our application to ensure correct and complete grading was achieved by the entire test process.
First the user will identify and press the "Grading" button to goto the next Grading menu. Here the user will be prompted to start docking student pens and a list of the students and their questions will start appearing behind the dialoge in a list format. The teacher should scan the list ensuring the grades marked on the right side of the list are accurate. Then the teacher should fill in any missing grades on the right due to short answer type questions. When done, the user should click the "Finish and Save" button on the bottom of the page to commit the grades to a gradebook.
  • Procedure: (1 page)
First, we determined the times in which all four of us can meet with customers. We then gave the customers the option as to which time slot is best for them. We assured to alocate at least an hour so there was no rush and we could compile our data without any time constraints.
Before our first study, we did a quick run through of our different tasks. We decided that each of us will be assigned one of the testing roles (Greeter, Facilitator, Computer, Observer) for all the customers. This way, we each would only have to specialize to do one role. Except for the Greeter, who becomes and Observer once he's done greeting.
When it was time to give the study, we first all met beforehand in a separate location to gather the materials and quickly go over our roles once again. This is so that we can arrive at the customer's house fully organized and prepared so as to not cause any possible confusion. Once arriving at customer's house, we all introduce ourselves. We explain what each of our roles during the lofi prototyping study will be. The Greeter, Antonis, then proceeds to explain our project and hands them the consent forms to fill out. During this time, the Computer (Qingyun) sets up the prototype on a table to prepare the study. Once Antonis is done Greeting, he joins Anthony in being an Observer and then the Facilitator (Joe) takes over. Joe then tells the customer which task he/she is expected to perform. During the different steps the user takes, the Facilitator asks the customer various questions regarding the decisions he/she makes. The Computer, Qingyun, performs the actions on the prototype in response to the user's actions. While this is going on, Observer 1 (Anthony) focuses on the low-level details of the study and Observer 2 (Antonis) focuses more on the big picture. Both Observers write down any obvervations they deem interesting, focusing mainly on problems with the prototype. After each task, the Facilitator spends a few minutes asking the customer more questions regarding how they felt about the difficulty or intuitiveness of the task. We then would proceed to the next task.
After we were finished with administering the study, we thanked the customer and left their home to go to a separate location. Here, we all discussed the observations of the study and came up with possible solutions to the problems the customer came across.


  • Test Measures: (1/2 page)
We had two Observers for the study. Observer 1 (Anthony) was focusing mainly on the low-level details, while Observer 2 (Antonis) focused on the overall flow of each task. Observer 1 had a stopwatch which he started once the Facilitator finished giving the instructions. The stopwatch was stopped once the task was deemed complete. During this time, Observer 1 also took note of which steps in the the process were causing bottlenecks.
We started off by a small explanation of the system and of each task. This way we wanted to observe how the user reacted to an unknown interface without a detailed help guide beforehand. As the users worked on the tasks and asked questions, we had a better understanding of what is important to include in our instructions or help documents.
[edit]

Results (12 pts)

Summarize the results of the experiment from your process data.

Our experiments with our various customers went very well. We finished on time with minimal procedural errors. Everyone preformed their tasks and stuck to the script we had planned for the event. The tasks we had the customers preform provided us with key insights, not only into the user interface itself, but to the broader topic of the usefullness of our project to the teaching community.

For the below we will be using this ratings scale:

  1. I don’t agree this is a usability problem.
  2. Cosmetic problem
  3. Minor usability problem
  4. Major usability problem: important to fix
  5. Usability catastrophe: imperative to fix


  • Import Test (Medium Task)
Customer 1 key observations
  • Customer seemed to understand the name of this task and understands why a test must be imported.
  • Customer wanted to skip step 2 (answer key creation) and complete this task later. Didn't see how to make this happen. Missed the next and back buttons. Finally clicked on the next step to advance. We should allow for both actions. Think this was more of a lo-fi prototype issue. Problem level 2.
  • Customer asked if a test could just be an answer sheet for a test. This was a minor problem level 1.
  • Wanted a dock when done dialoge box for step 2 to make more clear how to advance. Problem level 3.
  • Step 3 has issues if the user decides to skip step 2. There is no docking and verifying if step 2 was canceled. Problem level 3
  • Verify step was missing a "done" button. Problem level 2
  • Verify step missing an option to redo the test key creation. Problem 4
  • Customer was suprised when the final save dialoge box came up when she pressed "Save". Maybe we should replace this name with "Save as...". Problem level 2
Customer 2 key observations
  • This customer had trouble understanding why it was necessary to import a test into our system in the first place. They thought that when we said they could use their existing tests we meant that there was no additional processing required. Problem level 1.
  • After some explaination this task went smoothly until the mark generation portion. They were lost on what to do with the printed test on anoto paper. They did not press the help button. Problem level 3.
  • Remarked that they liked the ability to define points for each question.
Customer 3 key observations
  • After our short explanation of the task, the customer had no questions as to its need.
  • He asked why he had to have the test on his computer. We explained that this was necessary for the printing of the test on Anoto paper.
  • He would prefer detailed instructions to be more accessible than clicking 'Help'.
  • He was reluctant to proceed with the marking. He felt he had to verify with us. This is a small and expected issue since the user is a beginner to the system.
  • At the end of the task he wandered what happens if he makes an error. Does he have to recreate the whole test, or is it possible to just correct the error. Although this is vague, it is an observation we should take into account.


  • Print Test (Easy Task)
Customer 1 key observations
  • This customer intended to print out one copy of a test and copy it on a copy machine. Our current system prints out a different pattern for every page of every test for uniqueness. Problem level 5
Customer 2 key observations
  • This customer completed this task with out any problems or questions.
Customer 3 key observations
  • This task caused the customer to inquire about the Anoto paper. How much does it cost, how do you get it, etc. He understood that technological limitations meant he needed to print all copies of the test.
  • Grading (Hard Task)
Customer 1 key observations
  • The customer had trouble interpreting our lo-fi version of this screen. Problem level 2
  • After describing it to her more fully, she winced saying that she would really prefer a summary view of the tests just turned in by her students. Problem level 4
  • User did not understand how to integrate her short answer manual grading with the autograded portion of the test. Wanted a more clear "done for now" option and a "finalize" option to commit the grades to the grade book. Problem level 3
Customer 2 key observations
  • Customer did not like needing to dock pens one by one to capture student tests. Problem level 1.
  • Customer seemed to understand this portion but we have a feeling they just didn't want to appear like they didn't understand. Some good feedback was possibly missed.
  • Remarked that if they used this system they would try to make more multiple choice tests to avoid doing this manual grading portion. Liked the autograde feature. When asked if it would be better than a scantron(tm) they remarked yes, since it meant they did not have to double enter grades.
Customer 3 key observations
  • Customer was not too happy about the need to dock the pens one by one. We explained that streaming will be possible in the future, but it is still limited.
  • He asked if the manual grading was by student or by question. He would prefer it if it was one question at a time.
[edit]

Discussion (12 pts)

Discuss your results. What did you learn from the experiment? How will the results change the design of your interface? Was there anything that the experiment could not reveal?

All of the teachers are exstatic about the mission of our project. They completely agree that the demands on their time is overwhelming and they wish they had more tools at their disposal to help lift some of this burden. However, they do not just want one time saver to cost them more time in another area. So our application must be user friendly, intuitive, and effiicient at achieving goals preformed by teachers everyday.

This lo-fi prototype helped us to understand many aspects of our project. First, we have a better understanding of the amount of work, issues, and technology needed to really bring our project to a successfull finish. Just in creating the lo-fi prototype we started to understand that the scope of this project may be larger than can be completed in the amount of time given. Second, some key assumptions about how a teacher administers tests made on our part are proving to be wrong and we may need to rethink some of the technology decisions we have made thus far.

For example, the easy task of printing out an already imported test we thought would be the simplest of tasks and not provide much insight into our user interface. On the contrary we came across our largest design flaw by testing this task. The standard procedure a teacher now preforms when giving a test is as follows: 1) generate a test using Word, Excel, etc. 2) Print out ONE copy of the test. 3) Copy it as many times as necessary for the number of students taking the test. Our system works similarly with one crutial difference, we print a test for each individual student changing the anoto pattern on each page so that each test is unique. This way we can corrolate tests with students by having the teacher hand out Bobby's test to Bobby. When the system syncs with pen x we can still identify which test belonged to Bobby. However, it was pointed out that it would be way to costly to print ~150 tests both in ink and in time on a slow printer. We now need to move to a solution where each pen is bound to a student either, just for the test, or for the entire year. One solution to this problem is to print out a roster on anoto paper. When a student "checks out" a pen a check is placed next to their name and the association is made by our system.

Another area where our interface can be improved is by making it more flexible. We have many screens where serial type tasks are completed in order before allowing the user to move on to another task. While this may decrease user errors by forcing certain procedures it does not allow for the batching of tasks. Take the following senario, a teacher first picks up our system and would like to get it up and running for the upcomming semester. One of the first tasks they would naturally complete first is the importing of tests into the system. Currently we make the user create an answer key for each test imported as it is being imported. The user may just want to get all of the test into the system first and then add answer keys when the test actually approaches since they may want to make changes to ordering, questions, etc. We will now revisit these wizard type interfaces and see about making them more flexible to accomodate out order processing.

The lo-fi prototype did have some short comings. It was very hard to present data that was changing dynamically and required us and users to keep alot of data about state in our heads. This can lead to problems since we can never really be sure our and their mental models of the prototype match. Our Grading screen involved a list structure that was hard to convey in the paper format and we believe the implementation details of this screen were not fully explored. Users seemed to just "get" when this screen appeared and didn't dwell on the details by changing sorts or reorganizing the data in any way. This leads us to believe some important feedback was probably missed.

[edit]

Appendix (6 pts)

[edit]

Informed Concent Form

[edit]

Technology, system and task explanation

[edit]

Lo-Fi Prototype Pictures

Image:DSC00061.jpg

Image:DSC00062.jpg

Image:DSC00063.jpg

Image:DSC00064.jpg

Image:DSC00065.jpg

Image:DSC00066.jpg

Image:DSC00067.jpg

Image:DSC00068.jpg

Image:DSC00069.jpg

Image:DSC00070.jpg

Image:DSC00071.jpg


[add comment]
Retrieved from "http://vis.berkeley.edu/courses/cs160-fa06/wiki/index.php/LoFi-Group:ET"