PilotStudy-Group:BluJay

From CS160 User Interfaces Fa06

1 Blujay Flashcards Pilot Usability Study

Blujay Flashcards Pilot Usability Study

latest working package: FlashCards.zip //refer to README.txt for instruction

Introduction (5 points)

"Functionally, its great!"
"I dont use flashcards often however if there
was a program like this available then I would
be more inclined to study with flashcards"
           -Freshman Engineer Jeff

[edit]

System Introduction

We are evaluating a second iteration of the BluJay digital flashcard system. Our target user group for this system are students who study subjects in which flashcard making is frequently used as an effective study tool, such as chemistry, math, biology, and various foreign languages. During our research of flashcard usage we have found many regular flash card users recognize the difficulties in trying to efficiently organize and study their flash cards. This system is designed to address several of these difficulties with paper flashcards including improving organization and allowing for randomized study order, and keeping track of the comfort level that the user has with the different flashcards in the deck. The flashcard system is also designed to leave the basic flashcard paper system as unchanged as possible so that users of the software will not feel like they are making any sacrifices to using our software, and also, will never be worse off than if they were using traditional paper flashcards.

[edit]

Purpose and Rationale of the Experiment

The purpose of this experiment is to study how users interact with our system and determine what interface modification we can make so that our system will increase user comfort and improve their efficiency. Specifically we have three primary aspects of the program which we have implemented and are seeking to evaluate on people unfamiliar with our system. We would like to evaluate how easily they learn to use the system as well as what features they feel are missing or burdensome. We intend to do this by monitoring the subject as they produce flashcards using the Anoto pen on our paper interface. Afterwards we will observe how well they can manipulate the cards in the browse mode and then how effective they find the study portion of the system.

[edit]

Implementation and Improvements(15 points)

The largest new feature we have added to our prototype is implementing the fully pen-paper digital interface. In our interactive prototype which we presented last week we did not have the import function fully operational. Now the streaming feature is fully operational. This worked out much better than our initial vision for the device using the batch stroke processing. There are two major reasons why this streaming solution is superior. The first reason is that actually trying to produce even a reasonable number of unique flashcards would be a major hurdle with the R3 toolkit. The toolkit is designed fundamentally to produce unique patterns on a single page. It is not inherently good at setting up a batch solution with a open ended number of flashcard pattern and corresponding XML reference documents. Instead of this batch idea, we have implemented a streaming version which greatly improves the feel of making digital flashcards. Using our streaming solution there is no ambiguity about whether or not the card was correctly recorded, or if it looks right. The import screen shows the flashcard you are working and shows exactly what strokes the pen has read. This means that if there is a problem with the pen or the Bluetooth connection, the problems will be clear immediately and will not be a surprise later. Even if the import will not start working (a worse case system failure scenario), the flashcards and pen will still work in their original capacity as a plain paper-pen technique.

In addition to the basic stroke capture, the JPEG storage and image manipulation tools are also now fully implemented. In the previous version, we had fields set up where the user could describe their flashcard flipping style, however we had no hard code to process this preference so that the cards would be interpreted by the software correctly. In this most recent prototype we have finished the image processing tools so that the flashcard can be flipped horizontally or vertically and the strokes will be correctly displayed on the computer screen.

In the previous iteration of the interactive prototype, much of the underlying card handling was not implemented and only the skin was in place. Now there is vastly expanded interconnection of the user interaction window and the underlying flashcard engine. For example, in the browse panel, most functions including delete, copy, move, set familiarity level have are fully functional. Also, in the interactive prototype we employed placeholders to represent the flashcards in the system; now those placeholders are replaced by real flashcard data that can be browsed by categories. The browse features in conjunction with the completion of the import has made this a usable program for organizing flashcards.

An additional feature that we implemented in the most recent version of the software is implementing the study portion of the software. In the previous version the interface portion looked complete, however, most of the controls did not serve any function. Now the study buttons such as next, flip, different card viewing options (front/back/both sides) are fully functional, although the familiarity settings on the study panel was not functional for the testing. Completely features include a 'return to options screen' button from the study mode. Previously there was no way to get back to the options screen once the 'study' had started. Another feature is the disabling of traversal buttons that don't apply. This behavior helps the user determine when they've reached the end of a list, so that the user doesn't have to keep clicking 'next' when next doesn't exist!

This most current version of the software also corrected several problems with correcting errors. In the interactive prototype, there was no way to go back to alter certain preferences. i.e. once certain preferences were set in the study mode, these would remain in effect until the software is restarted. In this version we changed the software so that clicking on the study tab would reinitialize the study preferences to default and allow the user to set up the study from scratch.

One aspect of this prototype which remains wizard of oz is fully updating the flashcard list. After the user has finished the browse tasks, i.e. he has moved the math flashcards out of the GRE folder, one member of the group would quickly restart the program so that the file list would be reloaded again. This was a minor aspect of the program which we only learned about during the pilot testing, and saw the math cards presented in the same group as the GRE cards. This will be an aspect of the software which we will have fixed when we produce the next version of the software.

[edit]

Method (10 points)

[edit]

Participants

All the participants are selected from Unit 2 dormitory; many of whom were recruited in Unit 2 study lounges. We set our requirement for our pilot testing users to having at least some experience with flash card making and studying.

CHC: Male, CS major, Freshman, Have used flash cards for SAT & biology, Interviewed for lo-fi testing.

CF: Female, Architecture major, Freshman, Use flash cards to study for various classes, First time user.

JL: Male, Natural Resource major looking to switch to Engineering, Freshman, Use flashcard for studying class material sometimes, First time user.

RC: Male, EECS major, Freshman, Have used flash cards for SAT in high school, but have barely used flash cards since, First time interviewee.

[edit]

Apparatus

The equipment we used was a Dell laptop computer running Windows XP. For the interface we used an integrated trackpad and Nokia's Anoto digital pen(Model SU-16). We connected the pen to the computer wirelessly using a Bluetooth dongle. We provide each subject with two plain cards to use a control for their flashcard making process, and 9 BluJay flash cards(8 needed, 1 as a spare). The experiments were conducted in Unit 2 study lounge where the users normally study. We employed a digital stopwatch to record the time it took the different subjects to complete a flashcard.

[edit]

Tasks

Create & Import (difficult)

The first task we asked of our subjects was to produce a sample set of flashcards. In this pilot study we wanted to make the production of the flashcard emphasized on the interface and not on actually studying so we provided a list of cards describing the flashcards that we wanted them to make. (See appendix) During this task we asked the users to use the Anoto digital pen to copy the information on the list provided to the flashcards as quickly as they feel comfortable. This task also requires the user to select a destination directory for the flashcards to be assigned, as well as specify how they would like to flip the card. This task is finished when the user has clicked the finish import button.

Organization: Move/Set familiarity level (medium)

In the second task that we are asking our user to perform, we want them to use the browse part of the program to do organizational work on the flashcards that they have just finished making. Most of the cards will already be in the correct folder if they have selected the correct folder in the import menu. However, we also included two flashcards which do not belong in the group. They were asked to make two math flashcards which were miscategorized into the GRE group. The user must select these flashcards and move them to the math folder. They will do this by selecting the two math cards and then clicking on the menu at the bottom and changing the menu selection from GRE to Math. Then they click the transfer button to change the directory. Also from this screen the user will also be asked to set their comfort level for each flashcard.

Study Flashcards (easy)

In a third task, the users are asked to study the GRE cards through the software interface by selecting the folder of their interest, choosing the words of desired difficulty level, and traversing through the list with a few basic buttons such as previous, next, flip(to see the other side). In addition, users can modify difficulty levels of cards as they master new information; they can also choose the mode of display so that the reverse side or both sides can be studies first.

[edit]

Procedure

As a first part of the experiment, we asked each subject to read and sign our informed consent document which stated the basic tasks we would ask them to do and also informed them of their rights as a subject in our experiment.

Each subject was given a short tutorial on how to use the digital pen and the features of the software. We then handed the subject a paper with basic goals for the experiment, and the vocabulary list that we chose for them to use to make the flashcards. We offered them the opportunity to look through the list of objectives for the experiment and to ask questions of us before starting.

Flashcard production

During this phase of the experiment we kept a stopwatch running and timed how long it took the subject to produce each of the assigned flashcards. The first two were on regular paper to be used as controls to measure the time it would take with regular pen and paper to produce one flashcard. Afterwards, we asked them to use the Anoto pen on our digital paper to produce the remaining eight flashcards. Our instructions for our subject were to produce the cards as quickly as they felt comfortable. The goal was to allow us to gain some statistical insight from a limited number of trials. Without providing compensation to the users, we felt it was inappropriate for us to ask them to give us more than 20-30 minutes of their time, and as a result we asked them to produce fewer flashcards than what would have been ideal. At the end of producing each digital flashcard we made sure the subject properly checked the import box before starting a new card. During this flashcard production part of the experiment we also made the subject produce two flashcards which would be put in the wrong directory so we could evaluate how effectively they would correct this error in the next screen.

BluJay Browse

When the user was operating in the program's browse mode we asked them to perform several operations which were depicted on the instruction sheet. These included setting Familiarity level with the different flashcards as well as moving two misclassified cards to the correct directory. As a first step we asked the users to move the two math flashcards out of the GRE directory and into the math directory. This was done by marking the checkbox at the corner of the math flashcards and then changing the directory below to math. Then they pressed the move button to move the files to the math directory. We also asked them to set familiarity levels to the cards from the browse screen. (Please refer to the appendix for detailed instructions)

BluJay study

When the user had finished all the browse tasks we asked them to use the study mode. In this mode we gave very sparse instructions and allowed the subject to choose how best to study the material. We felt that studying is an individual process and we would gain very little by asking a subject to study by a procedure list.

As the user performed all of these tasks, we used a stopwatch to take timestamped notes about the rate the user went through the import procedure and the browse feature. We also looked for places where the subject behaved unexpectedly, or the program behaved in an undesirable way. We encouraged the subjects to speak out loud about what they were thinking and what they were trying to do as they did different actions within the software.

At the end of the experiment, we would ask the user to rate the application in terms of visual presentation and ease of use, and ask them for additional thoughts and feedbacks on the interface. If there were special operations they performed during the experiment, we would ask them to clarify their thoughts and actions to help us figure out what went wrong.

[edit]

Test Measures (5 points)

Qualitative analysis:

This test was designed to provide qualitative feedback about the system in addition to statistics. Therefore, we asked the user to speak aloud during the various tasks and we wrote down useful comments and ideas for how to change the software. Several things which we were particularly interested in was whether or not they would understand the button names and would follow the anticipated flow of the program. We also took note of how the program responded to large deviations from our anticipated flow through the software. We were alert for signs that the program was behaving unexpectedly as a result of a user mistake, or bring up error messages. If the users found an error, we would then try to determine a way of building in a case where the user could be guided to using the software correctly instead of bringing up an error dialog.

Quantitative analysis:

One of the key constraints for our system which we feel is critical to its viability as a study tool, is that producing the flashcards using the digital pen-paper interface will take no longer than it would on regular paper. While it would be possible to make this system without the pen at all, we feel that inputing flashcard content into a computer directly by a conventional method would significantly slow the flashcard production process. In our test we would like to measure how quickly our users can make flashcards on our system and compare this time with the length of time it takes the user to make flashcards on regular notecards. To do this we propose a controlled experiment to examine how quickly users can make flashcards with regular paper and compare this to how quickly they can make the flashcards on our system initially and after a small amount of practice.

Our procedure is as follows:

Present the subject with a list of flashcards that they are to make.
Allow the subject time to look over the full list.
Time the subject as he/she makes the first two flashcards on regular paper.
Time how long it takes for the subject to produce each flashcard thereafter with the Anoto system.
Examine whether time to make flashcard approaches time to make a regular flashcard

This experiment will help us understand several important aspects of how users improve with our system. We will learn the initial learning curve uses show for our system, and we will also be able to predict how effective they get at flashcard production with practice.

To control this study we will ask each of our subjects to make several cards that have been chosen to be a fairly equal in difficulty level as well as provide important raw data in the following software tests.

[edit]

Results (10 points)

[edit]

Import

[edit]

Key Issues

All users were able to successfully import the 8 cards
Two users began writing on digital card before starting the import panel at the beginning
Two users wrote on the back side of the card first, then quickly discovered their mistake
Most users struggled with the DONE button at some point during the importing task (due to the shift of the pen)

[edit]

Measurements & Analysis

Time it took for the flashcard users to produce a regular paper flashcard:

mean: 18.25 seconds

standard deviation: 3.84 seconds

Time it took to complete a digital flashcard:

mean: 31.34 seconds

standard deviation: 12 seconds

Time spent delayed on trying to tap the done button:

mean: 5.34 seconds

standard deviation: 8.5 seconds

Time spent making digital flashcards, omitting time with done button delay:

mean: 26 seconds

standard deviation: 7.8 seconds

Student's t-test comparing the time to make a regular card and a digital card

The whisker box A represents the dependent variable of this experiment. i.e. the time to make the flashcard. The B box represents the time taken to make a regular flashcard.

The left hand graph shows the relationship between the average total time to make a digital flashcard versus a regular one. The middle graph shows the difference when we see when we subtract the time fumbling for the done button. The right shows the results of the last two flashcards made, presumably when the subject is most practiced.

Null Hypothesis: There is no difference between making a digital card and a paper card

t= 3.01 corresponding to a probability of 0.46%, meaning that we reject the null hypothesis that there is no statistical difference between these two activities. Producing flashcards with our system is slower than making regular paper flashcards.

Null Hypothesis: If we exclude problems with the done button, then making digital flashcards takes the same time as making regular cards

t=2.69 corresponding to a probability of 1.1%, meaning that we reject the null hypothesis. Even without "done" button delays, making a digital flashcard is slower.

Null Hypothesis: After the user has practiced with the digital pen, there will be no time difference between the digital and regular flashcards. (last two digital flashcards took the same time as the regular paper cards)

t=2.17 corresponding to a probability of 4.8%. This is on the cusp of being unable to reject the null hypothesis. 5% is normally the cut off point, however, we had few users and did not have enough time to complete the practice. From the trends in the data, we believe that a user who has practiced using the digital pen may be able to produce digital flashcards at a rate which is essentially equal to the rate at which they can produce regular flashcards.

In this graph, the data from our limited number of subjects and with limited numbers of trials is insufficient to conclude that the subjects are definitely improving with practice. However, the trend of our data is that the mean time to produce a flashcard does diminish as they gain experience with the Anoto technology. The trend suggests that they will become better but will eventually plateau as the flashcard production rate becomes bottlenecked by the speed they can write.

[edit]

Organization

[edit]

Key Issues

Two users click the MOVE button before selecting a destination folder: cards were moved to "unknown destination" (deleted).
Two users expected the settings to change immediately after they selected the desired rating without clicking SET COMFORT LEVEL button.
Many users had to scroll up/down multiple times due to the large size of our "thumbnails".

[edit]

Measurements & Analysis

Number of clicks to complete MOVING task specified:

Minimal/correct number: 5

Avg number of click: 4.5 / Stdev: 1.66

Avg number of clicks different from correct: 1.5 / Stdev: 0.866

Number of clicks to complete SETTING COMFORT LEVEL task specified:

Minimal/correct number: 12

Avg number of click: 16 / Stdev: 4.74

Avg number of clicks different from correct: 4 / Stdev: 4.74

Time (sec) spent on the browse panel:

Avg: 135.25 sec

Stdev: 32.9 sec

[edit]

Study

Since studying panel was rather straightforward, all the users were able to traverse through the cards they made without any obvious difficulties. Thus no important measurements other than key observations were taken from this task. We omitted recording any values on this study because studying is an inherently unique activity and we could not think of a reasonable way to pilot this experiment. To test the efficacy of this study style, we would need large populations of students preparing for a similar exam. Such as study a population that used the software to study for the SAT and a population that did not use the software.

[edit]

Overall

Rating on visual presentation of the application (1-5):

mean: 3.625

std dev: 0.414578099

Rating on ease of use (1-5):

mean: 4

std dev: 0

[edit]

Discussion (15 points)

[edit]

Discussion of Statistics

The results of the statistical analysis show that our goal of making the digital flashcards is not inherently as quick and easy as making it on normal paper. However, while there is a statistical difference between making the cards on digital paper vs. on normal paper, we can not conclude that producing digital flashcards is functionally slower than normal flashcards. In this task we asked the users to produce flashcards from a list that we provided, and even after a limited amount of practice, the subjects were 20% slower producing the digital flashcards than the regular flashcards. However, if they were making flashcards for a class or an exam, making the flashcard might be only a very small percentage of the time spent on each thing that they intended to study. For example, if the user's goal was to learn 20 chemistry structures, they might spend four minutes researching each structure and deciding what to write on the card. Once the card is finished, they may plan to study 10-20 times for a total of 10 minutes. In such a case, spending an additional 4 seconds on the flashcard production component would be inconsequential on the grand studying scheme, and this slight delay would be unlikely to cause the user to shun the digital flashcards because they take slightly longer.

Furthermore, we did not ask the subject to produce so many flashcards that they became an experienced software user. Thus, there was a degree of novelty to watch the strokes appear on the screen, which caused them to make the flashcards more slowly. Using the plot of time to produce the card vs. practice, we attempted to show that the time to make cards would decline according to a power law, and would eventually asymptote at a level similar to regular flashcards.

[edit]

Improvements to Testing

One issue that occurred with the pilot study is that the participants had trouble following directions. Once the program was presented to them, they kind of “took off” and did their own thing with the interface. On the one hand, this shows that they are getting into the program and that they are excited about it, however, on the other hand, it means that they are likely to do the wrong thing. For example, a few of the participants wrote on the wrong side of the flash cards (or were about to). Because of the limitations of computer interpretation of flash card data, “sidedness” has become relevant. However, some of the participants didn’t realize this right away.

Part of the problem may be due to lack of physical directions to follow. In this experiment, we gave directions mainly through vocal commands and gesturing to help guide the testers. However, this approach is more casual than having a set of paper directions. It’s possible that given a clear set of numbered directions on paper, the tester may be inclined to follow it more exactly. This way we can prevent errors from occurring such as the one when a user clicked the “done import” button prematurely, because he thought he had to change which folder to import to in the middle of a session, when in fact that task was not specified.

Another improvement we could make to the experiment is to take more data (and specify which data to take more accurately). Instead of just taking down the times for “important events,” it would have been useful to specify exactly when events started and stopped. Some of the data taken is subject to a bit of guesswork because the clock was always running. However, in the event that a user gets stuck because the Anoto pen is not functioning, or because they misunderstood the directions, the time accrued does not reflect the interface design.

Additionally, instead of taking only temporal data, it would also be useful to take event-driven data, such as the number of clicks necessary to perform a task. One of the trends we noticed was that a user who got the task done more quickly also did not click around as much. Thus we think it would be interesting to see the correlation between number of events needs, the KLM model, and the actual time taken to perform a task. The additional information may also help guide us in designing specific interface elements.

For a serious study, we would probably consider adding a more in-depth questionnaire at the end. For this study we only included a two-question “on a scale of one to five” verbal questionnaire. This is useful because it qualifies the quantitative data from the tester’s viewpoint. In the moment, it’s not always easy to tell what they think of the interface, however adding this section will help with that.

[edit]