PilotStudy-Group:RRBG

From CS 160 User Interfaces Sp10

Jump to: navigation, search

Contents

Introduction (5 points)

Many teachers of mathematical subjects use whiteboards during their lecturers. While all teachers plan the content of their lectures beforehand, few plan how they will actually use the whiteboard during class. This can result in poor organization of content on the board, which in turn can lead to student confusion.

The Whiteboard Planning App helps teachers plan out the layout of the lecture content that they'll be writing on a classroom's whiteboards. It also gives teachers the option to establish how much time they want to spend discussing each board. The app allows them to view their planned boards while presenting their lecture so they know just where, when, and how to put each element on the board. Our goal is to make lecture planning quick, easy, and effective for inexperienced teachers, and to help them utilize their whiteboards efficiently and in a way that is easy for students to view and follow.

Currently, the Whiteboard Planning App is in the interactive prototype phase--it runs on the actual iPhone hardware, but it is not yet complete with respect to user experience. By conducting a pilot usability study, we hope to identify usability problems with our interface that could not be exposed with our Lo-Fi prototype study (for instance, issues stemming from the size of the device screen or the real experience of entering text and taking photos on a mobile device).

Implementation and Improvements (15 points)

  • Users now have the option to manage the timing of their presentations. They set how long the presentation will last, and then can specify how much time should be spent on each board. This is accomplished via a slider interface that allows the user to see how far in into the lecture the current board is and how much time after this board has been dedicated to other boards. The interface prevents the user from accidentally setting times that exceed the total duration of the lecture by stopping the slider from dragging too far. In presentation mode, a notification at the bottom of the screen alerts the user as to how much time remains for a given board.

  • Boards can now be numbered according to which physical board in the lecture hall the content should be drawn on. This helps users track their boards in case they want to leave something up for a majority of the lecture while other boards are getting used/erased.
  • Board elements can now be dragged onto the board (replacing the old "tap to select/tap to place" interaction technique).
  • Addition of a Custom board element.
  • Minor improvements to the look and feel of the icons.

Method (10 points)

Participants

User 1

(User 1 was 'TA' in our Contextual Inquiry)

User 1 is a male Computer Science student in his twenties. He's currently a TA for an introductory Computer Science course, and he has TAed the course in the past. He has about a year of experience planning and giving whiteboard/blackboard-based classroom presentations. He does not own an iPhone, but he has developed code for one and is familiar with typical usage patterns and interaction techniques.

We went back to User 1 primarily because of his experience with iPhone and Mac OS development. We thought that getting feedback from someone who both fit our target audience and had UI design experience himself would be valuable.

User 2

User 2 is a male EECS graduate student in his twenties. In his research and teaching, he covers topics such as machine learning and optimization. He has more than two years of experience planning and giving whiteboard/blackboard based classroom presentations. He owns and regularly uses an iPhone.

We chose User 2 because the subjects he teaches are highly mathematical; he fits our target user group well. We also selected him because he had never heard of our app before (by contrast, the other two users both participated in our Contextual Inquiry). We thought that getting a completely fresh, unbiased opinion would be valuable.

User 3

(User 3 was 'Prof2' in our Contextual Inquiry)

User 3 is a male in his 40s, a lower- and upper-division Electrical Engineering professor with 8+ years lecture experience and a number of years TA experience prior, and does not use an iPhone.

He was chosen because of his experience with effective boardwork and the quality of his feedback from the Contextual inquiry.

Apparatus

We elected to conduct our usability studies using the actual iPhone device, rather than the simulator, because we felt that the size of the device and the real-world touch-screen experience were crucial to the success of our application.

We used a point-and-shoot camera to record the test sessions, though we were careful only to record the test subjects' hands as they interacted with the device.

We conducted the studies in various offices and common spaces around Soda Hall and Cory Hall that had large tables with ample surface area (for the user to place the device and the observers to write notes).

Tasks

Task 1: Create a presentation

Our hardest task, planning the layout of a board, takes the longest and is broken into sub-tasks to cover what our app is capable of. In order to cover all the functionality we implemented, this task is now more specific than we previously reported. We (a) gave the users 7 different items to put on the board, and we had the users choose the layouts they would feel comfortable using. We also had them (b) rearrange existing boards and delete them, too.

In this task, we were looking for 2 main things: how well the user grasped the board layout concept, and how easy or difficult the interface for rearranging and deleting boards was.

Task 2: Annotate board element with photo of notes

Our moderately difficult task involves using the iPhone camera. Our program has placeholders for the type of content that will go on the board, but the presenter will likely want to refer to their full set of notes. The user was given a printout of Maxwell's equations to photograph for later reference.

In this task, we were looking to validate our interface for annotating a board element and for notifying the user of an annotation.

Task 3: Set board timing

This new task requires the user to think about how long their lecture will last (20 minutes in our experiment) and to decide how much time to spend going over each board.

In this task, we were examining the usefulness and usability of our timing interface.

Task 4: Give presentation, call up photo of notes, flag a board

Our easy task is giving the presentation, using what the user had previously planned. Since our app stores a photograph of the lecture notes, the user was asked to refer to these notes in the middle of the lecture. The user also needed to mark a board element for later revision.

In this task, we were primarily concerned with how quickly the user could understand and use the presentation interface.

Procedure

(See the links in "Additional Materials" for greater detail)

The procedure for this study was very similar to that of our lo-fi study. We had one person act as facilitator, one as the video recordist, and the remaining people as observers. We first had the user sign the consent form, then we demoed the app, and finally we began the test proper by giving the user our tasks in succession. At the test's conclusion, we discussed peculiar behavior with the user and asked for his feedback.

Test Measures (5 points)

For each task and for each user, we measured:

  • Completion time: To determine whether the app allows a productive workflow or bogs users down with cumbersome interaction techniques.

For each screen and for each user, we measured:

  • # of errors made, divided into the standard four categories:
    • Slips
    • Mistakes
    • Lapses
    • Mode errors

The frequency of errors helps us determine which sections of the app need the most attention, and the type helps us decide what form that attention should take.

Results and Discussion (25 points)

Test Results

(See "Statistics" in the Appendices for complete quantitative data and summary statistics)

Task Completion Time

User 1 tended to take longer because he explored the application more. User 2 took the least amount of time because he did not explore much and because he was the most familiar with iPhone interactions (being the only one of our users who owns one). On average, we feel that the times we observed were acceptable, with the possible exception of Task 1.b (where users got bogged down by hard-to-execute drag-and-drop functionality--see the discussion below)

Common Errors

  • Board Editor: Slips came from users failing to drag certain elements (particularly "Proof") from the element list onto the board. "Proof" is the top-most element in the list, and users tend to drag elements to the middle of the board. Doing this with "Proof" often triggers the Scroll View's scroll instead of element dragging. The two lapses came from users attempting to annotate a board section before dragging an element to it.
  • Lecture Overview: Users had a lot of trouble both initiating and following through with drag-to-rearrange and drag-to-delete. Common types of errors include: Thinking that double-tap initiates a mode (instead of a quasimode), holding to initiate a mode (i.e. iPhone home screen) or a quasimode, and simple single dragging (which triggers the Scroll View scroll).
  • Timing: This section was largely unproblematic. User 1 had a few minor slips in which he missed the slider thumb and failed to drag it.
  • Presentation: User 1's disproportionate number of slips came from a bout of failures to tap the flag icon in a small board region. Mistakes came from users continually trying to tap un-annotated board sections, thinking something would happen.
  • Help Section: User 2 tried to slide through the help (instead of using the forward/back buttons)

What We Learned

From Problems Users Had

The biggest thing we learned was that our current Lecture Overview screen has some serious problems. Even after viewing the help (and, in some cases, eventually being explicitly told what to do), users consistently failed to perform drag-to-rearrange and drag-to-delete properly. It seems that a fully modal (rather than quasi-modal) interaction technique would be less error-prone and more concordant with user expectations.

We also learned that the toolbar in our Board Editor could use some work. It's too crowded (some users accidentally hit buttons next to the one they intended to hit), and the "+" button seems unclear (users verbally expressed confusion as to what this button would do).

Users also expressed confusion at being unable to view the photos that are attached to a particular board section. The camera icon indicates that there *is* a photo attached, but not which one. Some users initially thought that they had failed to attach the photo correctly because they could not view it.

We also learned that users have trouble taking a photo in landscape mode (they tilt the phone correctly, but it doesn't register). Our Presentation Mode photo viewer does not correct for this, causing photos to show up at strange and inconvenient sizes. Users expressed frustration with this.

From What Users Want

All the users expected some way to quickly move between boards from within the Board Editor. They did not tend to use the "+" button, which adds boards from within the editor (perhaps because they did not understand it, as one user vocalized), but they wanted to quickly move between existing boards.

Users liked the flagging feature but felt that it did not convey enough information. They wanted some way to record WHY they had flagged an item for revision, expressing concern that having the flag icon alone would not be enough to jog their memory.

Users had similar feelings about physical board numbers and timing: good ideas, but not enough control. One user wanted to see board numbers that better reflected the layout of physical boards in his lecture hall (How many columns of boards? How many boards per column?). Users also wanted some way to see how far, number-of-boards-wise, they have progressed during Presentation Mode. Timing, while appreciated, was also too coarse for some users. They wanted different options: time for the current board, time for the whole lecture, etc.

There was a lot of disagreement over the style of the lecture overview. One user was fine with a linear list. Another wanted to see a more hierarchical system with boards grouped into sections. Yet another wanted to see a system whose arrangement of boards mimicked the physical layout of the boards in their lecture hall. It seems there may be no one "right way" to display this overview. Allowing multiple views is one option, but the amount of configuration required for this could be a huge deterrent.

We also asked users if they would prefer to author their lectures in a desktop environment. Interestingly, they did not. Users reported that the touch drag-and-drop interaction made bearable a task that would be too tedious on a desktop. One user thought that a desktop would be useful if he could use a tablet to draw equations--in other words, having a mouse and keyboard didn't mean much to him. He also thought that taking pictures of hand-drawn equations, while not as high-fidelity a capture as tablet input, would suffice.

Changes Planned

  • General improvements
    • Get rid of "Accept/Cancel." Just have "OK." Having both clutters interface, makes it easy to accidentally hit one. None of the interactions are critical enough to merit "Cancel."
  • Lecture Overview Interface improvements
    • Change rearranging from double-tap-and-drag quasimode to more iPhone standard hold-then-drag mode (like the Home Screen)
    • When rearranging boards, zoom out so user can see more of the lecture at one time.
    • When rearranging boards, add visual indicator that boards can be rearranged (i.e. the way Home Screen icons jiggle when in edit mode)
  • Board Editor Interface improvements
    • Add extra scroll space at top and bottom of element chooser and turn off bouncing so elements are easier to drag off of scroll view.
    • Add clickable affordance to board number textbox (no users explored--they didn't realized it could be clicked)
    • Be able to view photo after you take it.
    • Show thumbnail of picture instead of camera icon.
    • Space out toolbar icons more
    • Make purpose of "+" button clearer (or remove it)
    • Add more layouts (Four-way split was specifically requested)
    • Add a way to move between existing boards within the Editor
  • Time Editor Interface improvements
    • Change colors of before/after time indicators (red/green convey negative/positive connotation inappropriate for this context)
    • Show (numerically) how far into the lecture the current board is.
    • Mark where each board ends (tick marks, perhaps)
  • Presentation Interface improvements
    • Show thumbnail of picture instead of camera icon.
    • Option to add voice annotation when flagging. (Helps users remember why they flagged something).
    • Show timing when viewing image annotation
    • Tap timer to show different values (current time, time left, etc).
    • Indicator (perhaps arrows?) that screen can be scrolled to show next/previous boards (not obvious to all users)
  • Help Interface improvements
    • Minimize need for (perhaps eliminate?) help. One user didn't expect to see help--he figured that the interface should be totally discoverable just by poking around. We feel that this is a good goal to strive for.
    • (If help kept) Make help navigation consistent with rest of interface (slide to scroll instead of tapping next/previous buttons).


Improvements Under Consideration

(More experimental, lower priorities--all based on specific user requests)

  • Replace board numbers by some way to define physical layout of boards in the room. This would likely be a drag-and-drop canvas in which users could drag around little board icons to approximate where the physical boards in the room are. Then, in the Board Editor, there would be a toolbar button that allows the user to select a board from this screen.

  • Ability to add "impromptu" boards in the middle of lecture. When users are taken off-track by an unexpected student question, they can photograph the unexpected content they wrote on the board. The app automatically adds a new board into the lecture that displays this photo.

  • Ability to export lecture to a .pdf that can be distributed to students.

Appendices (5 points)

Additional Materials

Raw Data

Critical incidents: File:RRBG criticals.txt

Statistics

Completion Times

Tasks User 1 User 2 User 3 Mean StdDev
Task 1.a 7:10 5:00 5:42 5:57 1:06
Task 1.b 2:40 1:38 1:47 2:02 0:34
Task 2 1:20 0:33 1:17 1:03 0:26
Task 3 2:08 0:50 1:05 1:21 0:41
Task 4 3:09 1:10 2:20 2:13 1:00

Board Editor

Measures User 1 User 2 User 3 Mean StdDev
Slips 6 6 1 4.33 2.89
Mistakes 2 2 2 2 0
Lapses 1 0 1 0.67 0.58
Mode Errors 0 0 0 0 0

Lecture Overview

Measures User 1 User 2 User 3 Mean StdDev
Slips 13 13 4 10 5.20
Mistakes 3 4 3 3.33 0.58
Lapses 0 0 0 0 0
Mode Errors 0 0 5 1.67 2.89

Timing

Measures User 1 User 2 User 3 Mean StdDev
Slips 3 0 0 1 1.73
Mistakes 0 0 1 0.33 0.58
Lapses 0 0 0 0 0
Mode Errors 0 0 0 0 0

Presentation

Measures User 1 User 2 User 3 Mean StdDev
Slips 9 0 0 3 5.20
Mistakes 1 1 0 0.67 0.58
Lapses 0 0 0 0 0
Mode Errors 0 0 0 0 0

Help Section

Measures User 1 User 2 User 3 Mean StdDev
Slips 0 0 0 0 0
Mistakes 0 1 0 0.33 0.58
Lapses 0 0 0 0 0
Mode Errors 0 0 0 0 0

[add comment]
Retrieved from "http://vis.berkeley.edu/courses/cs160-sp10/wiki/index.php/PilotStudy-Group:RRBG"
Personal tools