PilotStudy-Group:WorldAccordingToG

From CS 160 User Interfaces Sp10

Introduction

The premise of our project is to create an iPhone application to help teachers take attendance, with specific emphasis on middle school. Since our users operate in a fast-paced environment which demands their full attention, it was necessary that we eliminate all but the essential functionality to make our interface crystal-clear. Our application concentrates on the user being able to enter attendance quickly and with minimal navigation, while allowing users to carry out significant and complex tasks, such as data analytics and class participation records.

We are in our middle phases of the process of interface design for this application, and at this point, we have our interactive prototype that can test with real users, who may not have the same level of ease in navigating the interface as us designers. This pilot study experiment we are presenting is one of the most important stages in our design process. The experiment that we are conducting involves having users to test the interactive prototype of our application by giving users three tasks, observing how they are able to perform these tasks, and collecting and analyzing their comments. It helps us identify usability problems with our interface that may not be exposed in our Lo-Fi prototype. It allows us to get direct feedback from our intended users, correct usability problems, and improve our interface design. We can analyze this experiment to detect whether our current interface design is missing something that users expected, or if it provides the functionality that we intended to have, and finally make improvements for the final product based on the interview analysis.

Implementation and Improvements

First, we would like to thank all of our fellow students for the great feedback we received after our presentation. Several glaring errors were brought to our attention and fixed even before we tested with real users.

The attendance view table cells are no longer colored using saturated colors. Instead, we have changed the display of the attendance status to look like a button, which affords clicking.
During our user testing, users, on multiple occasions, accidentally changed the attendance status of students when attempting to scroll through the roll sheet. Clicking on a student name now does nothing.
Users did not discover the ability to change dates by tapping the current date, so we have added a gradient background to make the date look more like a button.
Saving and creating notes in the attendance view requires only two clicks instead of the previous four.
The idea of roll sheets seemed to confuse many people, so we combined the classes and roll sheets tabs into one "Classes" tabs. Class information (name, start and end time, enrolled students) can be accessed via the Info button in the top right of the attendance view.

The attendance history view seemed to confuse the user, as the attendance are separated in different pages. Now, we combined them together and put them into a calendar view. Also we added a graphical statistical analysis that can show the attendance history in the last 30 days.
The students in the "All Students" view are sorted and searched by last name. Our test users told us that they often remember students by their first name. So we change it to sort by first name.

Method

Participants

Our first user tested with us back at the beginning of the semester. The other two users responded to an email sent to local middle schools. The text of the email is attached in Material section.

User 1

Age: 34
Sex: Male
Grade Levels Taught: 9-12
Teaching Time: 8 years
Education Level: MS
Experience: None

User 2

Age: 38
Sex: Female
Grade Levels Taught: 6-8
Teaching Time: 15 years
Education Level: MS
Experience: Has an iPhone/iPod Touch

User 3

Age: 55
Sex: Male
Grade Levels Taught: 8
Teaching Time: 30 years
Education Level: MS
Experience: None

Apparatus

Our users tested our application on a second generation iPod Touch while seated at a table inside their classroom. For timing our tests, we used timing applications on our cellphones. We recorded notes using pen and paper, and later typed up these notes for archival purposes.

Tasks

Task 1

Mark students absent
Market students tardy
Mark students excused
Write a note excusing a student

During this test, we made sure to note any critical incidences in which the user incorrectly changed the attendance of a student. We also made sure to note any time a user seemed frustrated by the number of operations required.

Task 2

Create a new student
Add the student to a class
Mark tardy
Write a note to say it’s their first day

In this test, we tracked which tab the user first selected when adding a new student to a class. We also tracked any critical incidences in which the user was confused about how to add a student to a class.

Task 3

Check the number of absences for a student
Change the date of the roll sheet
Change the attendance status to present

Procedure

For our testing, we met the user in their teaching environment (usually a class room) and had them sit down at a desk or table. After introducing the user to the interface, we handed the user the tasks to look over. Two people stood behind the user, looking over their shoulder, taking notes as the user performed the three tasks. One of our members tracked the time during the tests.

Testing Procedure

Have the user sit down in at the table
Instruct the user to fill out the Informed Consent form
Introduce the user to the application and the interface.
For each task:
- Hand the user the task, letting them read through and ask any questions.
- Let the user start the task, starting the timer
- Watch the user complete the task, writing down any critical incidences.
- Stop the time once the task is finished
Ask the user for opinions regarding the current application layout
Ask the user for suggestions for features / improvements.
Thank the user for his/her time

Test Measures (5 points)

Biographical information (age, sex, education level, iPhone familiarity): These are important independent variables for categorizing user groups and finding correlation with the dependent variables.
Time to task: We measured how long it took each user to complete each task, as well as subtasks within those tasks. One of the most important goals in our interface is to make it quick to use, even for novice users. Analyzing time-to-task data against iPhone familiarity might also give a good measure of the learning curve of our app.
Taps to task: Without filming the tests we weren't able to completely count the number of taps or navigational gestures performed by each user to achieve each task or subtask, but by recording descriptions of the actions of the users we were able to get an estimate of this datum. This measurement is very similar to the time to task measurement, however taps to task is much easier to directly control by interface design, and can be engineered to depend less on familiarity or other personal user data.
Number of errors to task: We measured how many errors in interface navigation the users encountered on their way to performing each task or subtask. This data is complementary to the taps to task data, but provides a more direct measurement of confusion.
Number of stalls/hints: We counted the number of times a user gave up or asked for a hint. This is a good way to judge not only how intuitive the interface is, but how important a help file might be.
Enthusiasm measure: Though highly subjective and dependent on many extraneous factors, the number of exclamations of joy or dissatisfaction by test users is an interesting thing to count. This data can give a general feel for how fun or annoying our interface was to use.

Results and Discussion

User 1

Task 1

The user immediately tapped on the first student’s name and saw that it toggled the attendance status to “A,” which he figured to represent “absent.” The rest of the absent students were marked correctly.
The user tapped on the name again, and saw that the status changed to “T,” which they correctly interpreted as “tardy.” For the second student to be marked tardy, the user tapped on the attendance symbol itself, which is a button, but has not been implemented. The user tapped about four times before we realized the bug, and we told him to click on the name.
The user says that he is “not sure” how to complete this task, but taps the name again, and changes the status to “excused.”
Sees the note icon on the right of the student entry and correctly enters a note. Clicked the “Done” button on the keyboard, was confused briefly, until he saw the “Save” button.

Time: 1:34

Task 2

Correctly navigate to the “All Students” tab, but taps on a student with last name starting with a “P.” Once on the student information screen, the user is not sure how to go back, but ends up clicking on the “All Students” tab again to go back. The user then tried to tap on the “P” heading above students with last name beginning of “P.” Eventually, the user sees the “+” button on the title bar and then successfully creates a student.
On the student information screen, after creating the student, the user saw the button to add classes, and adds the student to the class. However, he mistakenly believes the “remove from class” button as an indicator that he successfully added the students.
To change the attendance status of the student, the user correctly navigates to the roll sheet. However, there is a bug, so the student is not yet immediately added to the roll sheet. We told him to navigate to the class list, and then back into the roll sheet. The student then appears and the user marks the student tardy and writes a note without any trouble.

Time: 1:57

Task 3

The user correctly navigates to the “All Students” tab and clicks on the student’s name. However, he does not know to scroll down to view more information. The user then goes back and forth between the two tabs. After a hint, the user scrolls down to see the number of absences.
The user sees the date on the roll sheet, and starts to click the arrows to go back in time. However, he wonders aloud if there is a faster way, and after a hint, he finds the date picker.

Time:2:13

Changes to Make

Attendance status is a clickable button, but does not do anything – will allow it to toggle attendance status
Update roll sheets with new students immediately

Things to Consider

This user had many suggestions and insight that we did not get from our lo-fi testing.

Is the excused status referring to the absence or tardy?
There are different types of tardies
Excuses happen through the attendance office, with a parent note or phone call
Would like to see how many students in a class were absent for a particular day
Zooming on roll sheet – to see all students at once
Flipping through attendance statuses may be too much – 8 at his school; usually only present, absent and tardy are under the teacher’s control

User’s Overall Thoughts

This user liked the iPhone platform for its mobility, and thought it was especially nice for PE teachers. Overall, the user thought the interface was simple and easy to learn. In particular, the user liked the notes feature.

Our Thoughts

Most of the mistakes and hesitation was from the lack of experience with the iPhone (i.e. scrolling and the buttons on the title bar). Our application had bugs, and if were fixed, the time to task completion and number of mistakes would be reduced. The user was very friendly and eager to provide feedback. We received many ideas to make our application even better. Throughout testing, the user seemed enjoy using the application.

User 2

Task 1

Breezed through the first task with no problems

Time: 0:42

Task 2

(0:40) First checked the roll sheet, but saw the student wasn’t there. Tapped the “All Students” tab and clicked the “+” button to add a new student.
(1:15) After creating a student, the user left the student information screen without adding to the class, and tried to find the student in the roll sheet. When she saw the student wasn’t there, she immediately went back and added the student to the class.
Marking the student tardy and writing the note were done without trouble.

Time: 1:55

Task 3

First, the user attempted to check the number of absences in the roll sheet, but soon realized she needed to go to the student information screen in the “All Students” tab.
(0:38) She correctly navigated back to the roll sheet to change the attendance. She saw the arrows on the date bar and kept clicking the back arrow. We hinted at a faster way, and she was able to call up the date picker.
(0:58) The attendance status was changed without problem.
The user was momentarily confused that there was no "save" function.

Time: 1:06

Things to Change

We noticed that all attendance data is tied together for each of the students’ classes – a tardy marked in one class changes the status in the other classes the student is in. We need to make sure the statuses are kept independent.
The user ran into the same errors as the first test.

Things to Consider

Would like to see other classes’ attendance for her students – i.e. to see if the student is just missing her class or not
iPhones are easily stolen
Women’s clothing generally have no pockets, so it may be difficult for her to carry around the class

Our Thoughts

This user was definitely familiar with the iPhone interface, and it showed in the speed that she accomplished the tasks. It was good to get some criticisms of our application, and the speed at which she completed the tasks was very promising.

User 3

Task 1

(0:30) Understood the attendance status symbols without trouble, and marked all students with their correct attendance statuses.
(1:20) Did not see the “Save” button on notes, so was confused for a bit, until he found the “Done” button.
User said, "Ooh yes, I own this."

Time: 1:55

Task 2

(0:30) Started in the roll sheets, and kept hitting the date arrows. The user did not notice the date changing. After a short while, the user accidentally hit “All Students” (1:00) and was confused on how he navigated there. The user then incorrectly navigated back to the roll sheets (1:30), where he remained confused. We give a hint, and then the user then correctly adds a student. (3:00)
The user sees that he needs to add the student to the course and does so correctly. Once the class was added, the user is not sure if the information was saved, but then sees the “Save” button.
While trying to change the new student’s attendance status, the user accidentally changed other students’ statuses while trying to scroll. The user did not notice until we brought it to his attention.
(5:30) After this, the user changed the status and wrote the note with not issues.

Time: 5:55

Task 3

(0:45) Started in roll sheet, but makes a “guess” and goes to the “All Students” tab. However, the user is unable to find the number of absences (1:05), so he goes back to the roll sheets and taps around (hits the notes icon, back to Roll Sheets button, date arrows) (1:45). After a hint, the user navigates back to the “All Students” tab and taps on the student. However, he does not know how to scroll and find the information. With another hint, the user is able to find the number of absences (3:25).
The user has been changing the date, but had not noticed it (4:45). At first, the user tried to just change the status of the current day. After a hint, the user brings up the date picker (5:35), and has some trouble using it (doesn’t know to scroll on each section), but eventually changes the date correctly.
(5:35) User says, "This is kind of cool."
No issues with changing the status.

Time: 6:45

Changes to Make

This user was the only user to click on the absences section of the student information screen, and we saw that the absences don’t appear in chronologically order.

Things to Consider

Would like some way to get to the student information screen from the roll sheet, instead of having to go the “All Students” tab
Linking iPhones between teachers would be very beneficial – can share notes about student personal problems, issues between students, etc.

User’s Overall Thoughts

The school’s student information system, PowerSchool, is confusing, and many teachers have issues with it. This app would be very helpful on field trips, especially since students often wander to other groups and teachers lose their roll sheets. In addition, the user though the iPhone was a great platform – when a student walks in late, the teacher will not have to interrupt the flow of their lesson to mark the student tardy. Lastly, the user thought the system was easy to use, and felt like he could figure it out with a little more practice.

Our Thoughts

We could definitely see the lack of experience with touch screens, since the user had trouble with some basically touches. However, it is nothing that a little training cannot fix. The user was very patient and gave us great feedback and encouragement after the testing was finished.

Overall Analysis

After all was said and done, we are happy with the results of our initial testing. Our testers were very helpful and provided us with a great deal of feedback. Of course, there are some issues to work out. All three of our users did not think to click on the date to get the date picker, so we will need to figure out some way to make that more clear. The two inexperienced iPhone users did not know to scroll down on the student information page to check for the number of absences. In addition, the “Done” button on the keyboard caused some issues with these users as well. We will consider giving save functionality to that button as well. However, I do not expect inexperience with the iPhone to be a huge factor if/when this product is released to market. Teachers had a development day devoted to learning their current student information systems (PowerSchool, Aeries, etc.), and we would imagine another such day would be dedicated to learn our application. For a first run though, the tests went relatively quickly and with more practice, we have no doubts that the teachers would learn our system in a short time.

We received a good number of feature requests, and we’re definitely working on implementing them. We are going to implement attendance statistics and visualizations. All of our testers talked about connecting the application with the school’s student information system and other iPhones. Our ultimate goal is to get this done, but it may not be feasible by the end of this course.

Appendices

Consent Form

Test Tasks

Task 1: Taking attendance

It’s Friday morning and it’s time to take attendance in your Geometry class. You see that Paige, Julia, Kristine, and Claire are absent. Mark them absent. A few minutes into class Julia and Claire show up late. Change their attendance status to tardy. At the end of class Claire presents a note saying she was away at a wedding. Mark her as excused, and leave a note indicating that she was at a wedding.

Task 2: Adding a new student

A student comes into your class late who you don’t recognize. The student introduces himself. His name is Gus Parker, and he is new in your class. Add Gus to your list of students, and then add him to your Geometry class roster. Mark Gus tardy, but leave a note by his name in the roll sheet for today indicating this it’s his first day.

Task 3: Viewing attendance history

A student in your Geometry class named Alex May comes to you after class. He is worried that he will fail your class if he has one more absence (the maximum allowed is 5). Look up Alex’s attendance history to see how many absences he has. Alex also claims that one of the dates he is marked absent (March 15) is incorrect, and he shows you a worksheet he did that day to prove he was there. Change his absence on that day to mark him present.

Raw Data

Test Notes 1

Improvement Screenshots

Before and After: Roll Sheet View
Before: Table cells are colored using saturated colors. The current date looks like a simple label	After: Table cells now are much cleaner. The date also now has the look of a button

Before and After: Roll Sheet View: add notes
Before: Saving and creating notes in the attendance view requires four clicks	After: Saving and creating notes in the attendance view now requires only two clicks