PilotStudy-Group:PollPrecision
From CS160 User Interfaces Fa06
Contents |
Introduction
Electronic voting systems have been a topic of much debate of late. There have been concerns and allegations of tampering and insufficient record-keeping, making vote verification difficult and recounts nearly useless. Yet the value of such voting systems is also great. They make counting and analysis of votes much simpler, and are less prone to human error in data entry because the voter interacts directly with the system. PollPrecision's aim is to address the criticisms of such systems and to simplify the task of conducting surveys, taking polls, and recording and interpreting the collected information. We accomplish this with a suite of three separate applications that work closely together. The first, our ballot designer, allows users to import an image of their completed ballot, and then add Anoto readable fields to it (PollPrecisionization). The second is an application that allows users to view and analyze the data collected. The third is the process by which voter actually votes, where we provide feedback that their votes are being recorded electronically.
In this experiment, we aim to evaluate our progress thus far with PollPrecision. By interviewing potential users of our product and allowing them to try our interactive prototypes, we can course-correct and improve our interface. We will closely observe our interviewees' actions and attempt to use some quantification of our observations to make judgments of the current interface for our system. This will allow us to make informed adjustments to our interface and find out what works and what does not. Going forward, this will give us direction and guide our decision making process with regard to the PollPrecision UI design.
Implementation and Improvements
What follows are the implementation changes and improvements that we have made to the system since the interactive prototyping stage. What is discussed are either changes we have already made or changes that are in the works, some of which stem directly from the interviews conducted in the pilot usability study.
- The most siginificant change in functionality in our system that we have improved since submitting our interactive prototype was the inclusion of some interactivity with the Anoto pen in our feedback system (for the Voting task). This was one of the primary concerns that was encountered during the interactive prototyping stage, where we relied too much on canned information and not enough on information actually gathered from the user. While we didn't get this fully implemented, we did make progress here enough to allow at least some interaction using the pen.
- We have continued to implement functionality since conducting our interviews, in many ways based upon the feedback we received. Most of our changes have been to the voting and ballot-making tasks, which are the ones that have the greatest impact on the users. The voting task is significant because this is the way that most users of the software would interact with the system. The ballot-making task is also important, because it requires the user to actually edit or modify a ballot. This is significant because the rest of the process is dependent upon the successful design of a ballot. Observing the results of the voting process is important, but the data is made available through database queries as well as through our software, so we put more focus on the other two tasks.
- For the ballot-maker, we have made some changes to the naming of some of the editable elements of the ballot. For example, we are changing "Category," (a measure to be voted on using the ballot, such as President, Prop 85, etc.) to "Item" (although we're still not completely satisfied with this new name). Work has been done to improve the PollPrecision-ization of the ballots, as we were missing a step somewhere. Ballots were printing with patterns, but the region information was not being generated properly. This is not fixed, but progress has been made. Additionally, more of the back-end to our system is implemented, facilitating use of the voting system as well as the results viewer.
- The results viewer has been improved to do less wizard of oz-ing. Most of this is not really visible to the user because it is impossible to generate enough data manually for anything meaningful to be produced for the results viewer. What we did do, however, was make progress to hook the viewer up to the data collections and structures that are common to all of the tasks in the PollPrecision system.
- As mentioned above, the voting system now uses some level of Anoto pen interaction to function. In addition, we're considering removing a lot of the visual feedback from the software and instead using subtle physical feedback (such as the pen buzzing) to indicate to users that their vote has been registered correctly. Votes would still be registered by docking the pen, but the need for a console inside the voting booth would be diminished. This may actually simplify the voting process for many who would otherwise be confused or thrown by the high-tech interactions. It might also be discomforting to some voters that some computer is monitoring their pen strokes while they write them, fearing an invasion of privacy. No decision has been made on this front yet, but it is a change we are seriously considering.
The large majority of our changes are under-the-hood type of changes, but they do have implications for the future of our system in this project. They give us the potential for expanding on the functionality of the system, and turning this interface prototype into a prototype for the entire system's design. We hope to make additional changes in the weeks to come to implement even more of the system.
Method
Participants
- Participant 1: Tech Savvy Male Voter- Participated in all 3 tasks
- White, middle-aged male
- technically savvy (worked in computer industry)
- voter, has voted in recent California election
- Interested in designing surveys/ballots using technology as well as on paper
- Was selected because he represents a large portion of voters. How he interprets the system will reflect how others technically savvy people would think about the system.
- Participant 2: Middle-aged Female Voter - Participated in the moderate and easy task
- lifetime voter
- recently voted in California election
- not a heavy computer user
- Participant 3: Elderly Male voter- Participated in the easy and difficult tasks
- White, elderly-male
- Experience with voting procedure
- College educated, former teacher for 39 years
- Participant 4: Student Voter- Participated in the moderate and difficult tasks
- White college aged male
- Voted recently in the local election
- Comfortable with computers and GUI's
Apparatus
We ran our program from Eclipse IDE on a personal computer running on Windows XP. The Anoto pen was connected to the computer via Bluetooth and was streaming. We printed the ballots with the dot patterns prior to the interviews, which took place in 4th floor Tolman Hall. Paper and pen were used to take notes. A stopwatch was used to task times and careful observation to keep track of button clicks.
Tasks
- Easy task: The easy task was for the participant to vote. It assumes that the voter has signed in and has the pen. During this experiment, we looked for any mistakes that the participant made, how often he looked at the feedback mechanism (highlighting his choices on the computer screen), and his confidence of the voting process.
- Medium task: The medium difficulty task was to import data and view the histograms for each category. This task is fairly straightforward, as the participant just looked at the histograms. We looked for how he was going through the different histograms, and if there were any ambiguous wording or interactive components.
- Hard task: The hard task was to add a category and a field to an existing ballot, and produce a ballot with dotted patterns. We looked for gaps between our model of the system and the participant's perceptual model. We also looked for how easily the participant was able to use the system, clicking errors, any part of the system that was difficult to learn and understand.
Procedure
We could not round together enough people qualified to test each interface accurately so we had to do a bit of role play to get complete our experiments. From 4 participants we had to test 3 separate interfaces so we had each participant test one or two of them so we can get a larger sample. For each of the three tasks, we asked each participant to act as user of the part of our system while making an effort to think out loud. We made sure that they knew what role they would be playing but allowed them freedom to play around with the interface so we could gain insight on how people learn our interface. This role-playing requires some creativity by the participants, but the feedback that we got gave us a good insight of the general usability of our user interfaces.
To test the ballot designer interface we started our ballot-design application and verbally told the selected participant the task that we wanted him to perform: to create a PollPrecision ballot (a ballot that works with the Anoto-pen) from an already existing ballot in PDF format. Once we gave them the task, we let the participants experiment with the interface. We did not offer any help until the participant was obviously stuck or asked for help. If the participant talked out his thought process like we asked them to, it gave us a good insight on how intuitive our interface is. We observed the number of clicks for a few different parts of the interface (add the parts). We took special notice of the user’s development of their conceptual model and at what points in their learning process they strayed from our model. Once the participant has figured out our model for designing a PollPrecision ballot we instructed them to begin creating the Ballot for the first few items. This is where we begin our experiment of the usability of our interface. Each user was brought up to the same competency of the interfaces model and we can measure the time it takes the participant to finish the task, and also the accuracy in which they perform the task.
Next the selected participant acted as a voter that has already signed in and received the pen and the ballot. We asked the participant to complete the ballot using the pen as if he is really voting. We did not force the participant to use the screen feedback. We tried to take notice of the participant's locus of attention. We noted on how often and to what extent the visual feedback was useful.
For the final part of our pilot study we had the selected participant act as a poll administrator by checking the results of an election. We asked them to tell us winning candidate and the number of votes he received by viewing the histograms. We then asked him to change the histogram to view another race and asked for the result of another item on the ballot. Throughout this process we took note of the persons thought process and the number of clicks it took to navigate the interface. As we stated earlier, the only problems we expect to have with it is navigation.
Test Measures
Easy Task (BallotViewer, taking the ballot):
- Total time to complete ballot, used to compare relativity with respect to #2 across all participants
- Number of locus changes from ballot to feedback screen, used to observe how switching locus affects total time
Moderate Task (PollGraph, histogram):
- Time to figure out who won the first ballot item, used to see how quickly they can locate the most-obvious statistics
- Number of clicks to figure out who won the first ballot, useful in understanding #1
- Number of clicks to change to a different ballot item, useful to see how easy it is for them to access hidden data
- Time to figure out who won the second ballot item, used to see how quickly they can access hidden data
- Number of clicks to figure out who won the second ballot, used to see how easily they can access hidden data that they want are looking for
Difficult Task (Ballotmaker):
- Time to learn general conceptual model, used to see how intuitive our interface is. (This was measured as the time the user spent before successfully creating the first category)
- Number of clicks to completely assign first category, used to see how effective our interface is
- Time to complete first 3 ballot items, used to see how quickly they can use our interface
Results
- Participant 1: Tech Savvy Male Voter- Participated in all 3 tasks
- Easy Task Results: When using the pen Participant 1 showed a lot of interest in the pen and how it worked rather than the task of voting. Once he got over the novelty of the pen and the screen changing with how he voted, his focus remained mostly on the paper. He only looked up when he changed pages to see how the screen reacted.
- Moderate Task Results: On this task participant 1 learned this interface fairly quickly due to its simplicity. The buttons were labeled clearly enough and he was easily able to find what he needed.
- Difficult Task Results: Participant 1 played around with the ballot design interface for a few minutes, but it soon became clear that he did not have a grasp of the model we used in our interface. The notion of "category" confused him and he began naming categories "Bubble fill in 1" to go with the options. He didn't realize that the each category corresponded to each item on the ballot. Once he figured this out, because he is tech-savvy and knows his way around the classic windows GUI he began trying word and other common short cuts. He complained that there was no way to copy and paste and select multiple regions while assigning categories.
- Participant 2: Middle-aged Female Voter - Participated in the moderate and easy task
- Easy Task Results: Participant 2 took little notice of the pen and kept her head down on the ballot focused on her selection despite us telling her to use the feedback screen where necessary. She finished the voting task quickly and kept to herself while voting. She checked over her selections after she finished using the screen.
- Moderate Task Results: For Participant 2 the results task went fairly smoothly. She compared the results to the ballot and noticed the nominees were out of order. She also had a little problem deciphering the UI despite because the lay out was not something she felt immediately familiar with.
- Participant 3: Elderly Male voter- Participated in the easy and difficult tasks
- Easy Task Results: Participant 3 complained frequently about the bulkiness of the pen and that his hand barely fit around it. There were also instances where the pens internal feedback mechanism buzzed at him and he wondered if he was doing something wrong. He didn't seem to trust the feedback screen. One time he marked a selection lightly, which our system picked up, but he eventually decided on a different selection, and never noticed the inconsistency with our feedback.
- Difficult Task Results: Participant 3 was not very familiar using computers, he admitted to owning one, but only used it for pictures and writing. He had trouble grasping the overall task and why it was even necessary. After some explaining he understood that the regions had to be marked so we can read what the voter wants to select. We eventually got him up to a working level where we thought he could participate in our experiment and he made many mistakes and took a long time, but seemed interested in the process and never game up.
- Participant 4: Student Voter- Participated in the moderate and difficult tasks
- Moderate Task Results: Participant 4 found the visualization to be bland and made a few design suggestions. He had no problems finding who won what ballot item, and cruised through this test easily.
- Difficult Task Results: Participant 4 stated that he frequently used Photo Shop on his computer and asked immediately if this interface was similar to it. We declined to say but he still seemed eager to learn. He had trouble finding the next page button. Later in the experiment he also commented that there were too many options in the category section pointing out that there is only a "choose exactly 1" and choose less than" option, and every other option shouldn't exist for voting. He also commented that the text overlay near the regions should have a way to be suppressed.
In the Ballot maker, the participants all took a long time to create a "category" (grouping) of selection items, and this can be attributed to the lack of a batch-setting functionality; currently, the ballot makers can only choose one item at a time and select the appropriate category it falls under. With the batch-setting capability, we believe that we can reduce both the number of clicks and time for mental preparation drastically. The fact that once the category was chosen for an item display a label next to that item gave the ballot makers problems, and also probably attributed to why the participants took so long creating the ballot: they couldn't see the words underneath the labels and further, selecting new regions became more difficult with more words on the page. We believe that this result will hold with a larger user population.
The pen buzzing frequently (due to the bad-quality printing) irritated all of the participants, and we imagine that this annoyance will hold with a larger user population, and thus must be fixed immediately.
Discussion
- The word "category" confused ballotmakers, because each ballotmaker had different conceptual understanding of that word. We might change this to "groups" or "items".
- Our ballotmaker currently does not support batch creation/editing of regions, and as Nate Timberlake pointed out, having this feature would make the ballotmaking process more efficient
- When the "categories" were created using the ballotmaker, they were also labeled next to the items, and thus cluttered the page. This made the ballot harder to read, and selecting new regions more difficult. Perhaps we might have a toggle on/off of the labels.
- The PDF ballot on the ballotviewer was hard-to-see for some of the participants, and they wanted to "zoom in", just like they could with Acrobat Reader. Unfortunately, we did not implement this feature. Do we want to implement a zooming feature? If we do, we might need to consider Grab-and-scroll feature like in Acrobat Reader, which would mean that we would introduce a new mode. With the PDFs varying a lot from user to user (because each user would be creating his own ballot in PDF), we'd have to consider ways to deal with this.
- One of the weaknesses of our pilot study with regard to our project is that it is very hard to address all of our tasks (or even more than one) with just one user, so the amount of feedback we get on each of the tasks is difficult to base conclusions on. We get good feedback, but there are problems of lacking the sheer quantity of actionable feedback.
- Some of the feedback we got on our voting task design was that the feedback mechanisms we had envisioned weren't all that useful or necessary. The question then arises: is having this feedback detrimental to the voting process, and how might we address this (either by changing/improving it or removing it altogether)?
- One valuable thing we learned is that the more functional the prototype, the more relevant the feedback on the system can be. Ideally, in doing the UI design, we would be building over a back-end that is already functional, so it's just a matter of hooking the UI in to the existing functionality, rather than trying to implement the back-end as we go. This would free us up to focus on the UI rather than the implementation details of the system.
- Rather than have a feedback screen, perhaps the only feedback we need is the pen buzzing to verify that the vote has been received.
- One fear that we really haven't addressed fully prior to now is that some users might be intimidated by the feedback screen while voting, while others won't find it useful and might in fact be confused by it. How might we address this problem? Is the screen even necessary? Can it be optional? If it were optional, how might it be presented to voters? How does this complicate the lives of vote administrators? These are questions that we must ask ourselves as we press on with the project.
- The feedback screen also unnecessarily complicates the voting process. I anticipate that streaming the data while the person votes will lead to security and data storage problems because the pen does not store data when it is streaming. Also any interactive on-screen features would detract and over complicate the entire process. The feedback screen also gives feedback that the user already has in front of the voter on the paper. Instead of giving feedback that we have registered a vote, maybe we should use the feedback screen for more auxilary functions such as directions helping the voting process.
Workload breakdown
- (21%) Keenahn Jung
- Write-up on results, appendices,
- Contributed to the discussion section
- Final editor
- (21%) David Eitan Poll
- Did write-up of implementation changes and improvements to the system
- Contributed to the discussion section
- Worked on continued implementation of the ballot-maker application
- Wrote introduction and rationale for project
- (15%) Heung Tai
- Contributed to the discussion section
- Improvements on code on PollGraph (medium task)
- (21%) Hiroki Terashima
- Did write-up of Method and Test Methods with Eric
- Contributed to the discussion section
- Interviews
- Improvements on code on Ballotviewer (easy task)
- (22%) Eric Vacca
- Did write-up of Method, and Test Methods with Hiroki and Results with Keenahn
- Contributed to the discussion section
- Interviews
- Worked on the overall interface which the 3 tasks communicate
Appendices
Consent Form
PollPrecision_consent:Consent_Form
Reproduced here:
Consent to Participate in Research
Researchers: Hiroki, Keenahn, David, Herman, Eric
Description
Your participation in this study will aid us in designing and improving our user interface for a voting system. We will be interviewing several subjects in this study. Your anonymity and confidentiality will be preserved. Please answer questions honestly.
Procedures
We will give you three tasks to complete using our mock computer interface. We will not tell you how to accomplish these tasks. We will answer general questions, but will not give specific information about how to complete the tasks. Please verbalize your thought process, as we will be asking you questions about how you are interacting with our interface.
Risks/Discomforts
For our study we will be asking you to manipulate pieces of paper/cardboard with ink on them. If you are allergic to these materials you will not be allowed to participate in our study. If you are uncomfortable with the task or with answering our questions, alert a researcher immediately and we will stop the study.
Benefits
Your participation in this study will be unpaid.
Confidentiality
Neither your name nor any other identifying information will be revealed in our documents and subsequent findings. Your confidentiality is guaranteed to be maintained.
Signature
I have read and agree to the conditions outlined above.
______________________________ (Name) _______________ (Date)
Demo Script
Welcome to our study. Please take a seat and we will begin shortly. First, we will have you read over and sign this consent form. Remember that this study is unpaid and that your participation is completely voluntary. Thanks. We are conducting this study to aid in the design and refinement of our user interface for a voting system. You will be assigned a few tasks and then asked to complete them using our interface. Please note that no elements of the interface are set in stone, and that it is still in the alpha stage. That means you can feel free to make wild suggestions about what you would like to see in the interface. Note: we will not tell you how to accomplish these tasks, nor will we answer questions directly about the interface. If at any time you feel frustrated with the tasks, keep in mind that it is not a judgment on your abilities to interact with our interface, and probably it is a problem with our interface. We ask that you verbalize your thought process as much as possible, and we will be asking you questions during the tasks. Do you have any questions? Then let's begin.
- Low difficulty task: (Show them the ballot page) Please fill out this ballot. Thank you.
- Medium Difficulty Task: (Show them the data analysis page) Please tell me who won the presidential vote and who won the senatorial vote. Thank you.
- High difficulty task: (Show them the ballot designer) Please add three items to the ballot. Thank you.
Ask them questions. Thank you for your participation. Your input has been very useful.
Raw Data
Incident Logs:
- Participant 1:
- Easy Task: 0:53 done playing with the pen and starts filling out the ballot quickly
- Moderate Task: It's all over in 7 seconds, no incidents to report
- Difficult Task:
- 0:05 immediately clicks the "open" button, which does nothing at this time
- 3:32 imports the PDF fine, draws a region, but does not understand our model of categories and regions. Incorrectly labels a region. We intervene
- 5:23 Successfully enters first item and category
- Participant 2:
- Easy Task: No incidents to report, filled out ballot with occasional peeks at the screen.
- Moderate Task: No incidents to report, clicked through the UI with no problems.
- Participant 3:
- Easy Task:
- 0:23 The pen buzzes for the first time, he asks if he did something wrong.
- 2:54 Finishes the first page, checking the feedback screen often
- 6:23 Does not notice that one of the highlighted fields on the screen is NOT one that he intended to mark on the ballot
- Difficult Task:
- 4:49 Finally, after almost five minutes of asking him questions (never telling him directly how), he understands the basic mechanics of how to import a file, including navigating the Swing file explorer, which is similar to the windows explorer but with slightly different icons
- 11:24 He successfully enters the first item and category! Again, this is after much guidance.
- 13:57 After he enters the first item, the other two come easily and he is done.
- Easy Task:
- Participant 4:
- Moderate Task: No incidents to report, whizzes through the UI at hyperactive speed.
- Difficult Task:
- 0:03 Immediately asks if this is similar to the Photoshop UI.
- 0:10 Clicks the "open" button, which is currently not supported
- 0:54 Does not take him long to find and import the PDF file
- 1:32 Tries to import another PDF file.
- 2:50 By almost shear button mashing, he manages to draw the first region and create a category for it
- 4:45 Takes his time a bit more with the remaining two regions. Spends time aligning the regions and he is done.
Quantitative Measures (See above for mapping)
| Participant | Easy-1 | Easy-2 | Moderate-1 | Moderate-2 | Moderate-3 | Moderate-4 | Moderate-5 | Difficult-1 | Difficult-2 | Difficult-3 |
| 1 | 213 | 3 | 3 | 1 | 2 | 7 | 2 | 323 | 54 | 452 |
| 2 | 477 | 11 | 6 | 2 | 2 | 11 | 2 | N/A | N/A | N/A |
| 3 | 586 | 32 | N/A | N/A | N/A | N/A | N/A | 673 | 77 | 837 |
| 4 | N/A | N/A | 2 | 1 | 1 | 6 | 1 | 170 | 38 | 285 |
| AVG | 202 | 6 | 3.67 | 1.33 | 1.67 | 8 | 1.67 | 148 | 27 | 266 |
| STDEV | 144.14 | 4.58 | 2.08 | 0.58 | 0.58 | 2.65 | 0.58 | 151.59 | 14.8 | 214.17 |
Charts
