LoFi-Group:PollPrecision
From CS160 User Interfaces Fa06
Contents |
Contributors
- Keenahn Jung - Mission statement, interviewed voter, poll worker, and ballot designer (experiments 3, 4, 5) added to appendix, contributed to method, results, and discussion
- David Eitan Poll - Hard task prototype/details, interviewed voter, added to appendix, contributed to method, results, and discussion
- Hiroki Terashima helped with design of all interfaces, took pictures, interviewed voter, contributed to method, results, and discussion
- Eric Vacca - helped with interviewing for experiments 3, 4, 5, designed interface for easy task, added to appendix, contributed to method, results, and discussion
- Heung Tai - designed interface for medium task, contributed to method, results, and discussion
Introduction and Mission Statement
Electronic voting systems have been a topic of much debate of late. There have been concerns and allegations of tampering and insufficient record-keeping, making vote verification difficult and recounts nearly useless. Yet the value of such voting systems is also great. They make counting and analysis of votes much simpler, and are less prone to human error in data entry because the voter interacts directly with the system. The mission of the PollPrecision project is to address the criticisms of such systems and to simplify the task of conducting surveys, taking polls, and recording and interpreting the collected information in an easy to use and secure manner.
Prototype
What follows is a description of our prototype for the three tasks we selected. The tasks involve all of the different target users of our system: voters, ballot-designers, and poll administrators.
Easy task: Voting Process
This takes place after voters are registered and polling places are set up. There are 3 steps in the process and multiple UIs.
First, the poll administrator turns on the given computer, and it boots directly into the Poll Precision software. The admin will be given a welcome screen to let him know the software is loading. The software then goes directly to a screen prompt that is awaiting input to begin the check in process.
When a voter arives, he will give identification in exchange for an Anoto Pen. He will then sign the check in sheet that is sync'd with the aforementioned computer prompt. When the check-in sheet is signed, the computer prompts the poll worker to give the voter a ballot.
The voter then takes the pen and the ballot into a voting booth with a voting screen. The voter will follow the directions on the ballot and make their selections. When a selection is made, the corresponding selection will be highlighted on screen so the voter can verify the computer is interpreting the input correctly. When the voter is finished he returns to the poll admin, who stores the ballot, docks the pen and offers the voter a confidential reciept.
Moderate task: Analyzing Results
This task is to be done after the polling is complete. The poll administrator clicks a button to submit all the recorded ballots to the central system. A dialog box will ask the administrator to confirm the submission of the polling to the central system. After the administrator confirms, no more polls will be accepted. The administrator can see the results of the polling by using another application in our suite. The application screen is divided into two halves, the top and bottom. The top part has a heading that informs the administrator that the results were submitted and when. There is a scrolling list to let him or her choose which category to look at. Some category examples are president, policy A, judge, etc. After the administrator makes his selection, the bottom half of the screen will display the results using a histogram as presentation. In addition to simply looking at the results, there is a button at the top part that can print out the results (the histogram with the appropriate caption).
Hard task: Incorporating a Ballot into the System
The task of incorporating a ballot into the system is an important one. It serves as a starting point that would allow PollPrecision to be adapted to the needs of various voting scenarios across the country. We don't presume to know how to make a good ballot, nor can we predict every possible situation. We can, however, provide the tools necessary to turn a traditional ballot design into something that our system understands. This task allows such an incorporation to occur.
In principle, this should be the first step taken when designing a PollPrecision poll or vote. Our prototype aims to allow a user to import a ballot designed in his favorite publishing application (as a PDF, for example), and define regions that the Anoto system will be able to work with.
The user begins by importing (see Import button in the interface photo below) a PDF of the ballot to be incorporated into the system. He then creates a series of categories (see the Categories button and dialog) for each of the ballot measures to be addressed. The category dialog is where some of the most important information in this task is acquired. The user must specify a name for the category and specify how selections will be made on the ballot.
There are a variety of selection types that we have suggested:
- Write-ins -- These might be a write-in candidate's name or a survey written response field (asking for feedback, additional comments, etc.)
- Selections -- The actual form that these take might vary...
- Choose ___ N -- Often, polls and surveys require that the voter choose some number of the available selections. For most traditional votes (such as for a candidate in a political race), voters choose exactly 1. However, it is conceivable that surveys will request that voters choose exactly N, at least N, fewer than N, more than N, or no more than N. Thus, we must allow the user of the ballot-incorporation software to specify what type of selection is expected.
- Types of selections (different forms of user input that we should support recognition of):
- X marks the spot (X through the selection region to make the selection)
- Circle
- Fill-in-the-box/bubble
- Check-off
- Any/all of the above accepted
These categories will ultimately be used for naming in the database when data is collected about the survey.
Shown below is a photo of the low-fidelity prototype of the system described for accomplishing this task.
Next, the user must define Anoto-enabled regions on his ballot. These will be areas on the ballot that respond to Anoto pen input. He does this by choosing the region tool and dragging regions onto the image of his ballot. He can edit and delete these regions by dragging them around (see the move mouseover in the photo). He can also select the region and set its category (already defined through the categories dialog). Finally, he can save his PollPrecision-enabled ballot and print it with the Anoto patterns automatically filling the regions defined in the application.
Method
Tasks
The easy task was for the voter to simply go through the entire voting process (show ID, sign in, take ballot, vote, return ballot), while we acted as the poll administrator and the computer.
The medium difficulty task was to import data and view the histograms for each category.
The difficult task was to add a category and a field to an existing ballot.
Participants
All names have been changed to protect identity. We selected our participants based on their relevant experience in voting, administering surveys, and designing ballots. They were all people we knew, so this may have introduced bias. We performed the experiment in environments comfortable for the participant (usually their office or home). We conducted the experiments in pairs, where one researcher played computer silently while the other researcher played the other roles. We focused on qualitative, process data and did not measure reaction time or count number of errors.
Another important point is that our tasks were not all directed at the same user. In fact, each of the tasks was targeted at a different user. We did the best we could to collect comprehensive information, and so we made an effort to interview people from each of our user groups so that we could cover all of the tasks for which we prototyped.
In each experiment, we opened with some banter to put the subject at ease while he signed the consent form, then we introduced the study and gave him the instructions for the task.
Experiment 1: Voter "Tom"
"Tom" is a college-age voter, who most recently voted in the California special election. He is not particularly technically savvy, but is also not computer illiterate.
Tom was interviewed in his home, as a real voting environment was unavailable. However, the environment played little role in our interface. The greatest exception to this is the privacy of the voting booth, although we simulated this by isolating Tom with the screen (and the person playing the computer) and his ballot.
It should be noted that Tom was selected for the experiment because he fit our target user group of "Voter," and would be able to express the needs of the typical voter.
Tom was asked to walk through the voting process, talking aloud as he tried to navigate the system. He first came to the interviewer acting as the poll worker, showed his ID, and signed in on the sign-in sheet. He was then given a ballot, and directed to walk into the ballot box to cast his vote, then return to the poll worker to turn in his ballot and pen.
When Tom turned in his ballot and pen (dropped the ballot in the ballot box, handed the pen to the poll worker), the poll worker docked the pen and gave him a receipt indicating that he has voted.
The entire process took less than 5 minutes, and there were only a few points when actions slowed down and Tom needed to think about what to do next.
For more information, please see the corresponding section in the Appendix
Experiment 2: Voter "John"
John is an “experienced voter” who is comfortable with new technology. We conducted the test with two group members; the first was a note-taker and acted as the poll administrator and the other member controlled the interfaces and simulated the feedback. John entered into our simulated polling station, and after showing ID was given a pen and told to sign in. John was then given a ballot and we conducted a few experiments on him to test our interface. In this experiment we were studying the focus of the voter paying special attention to where his eyes move, and where his focus (and locus) of attention is.
Quantitatively in this experiment we were looking to see if discrepancies between the paper ballot and the screen were caught by the voter. Also we kept count of the number of times the voter double checked his selection when it appeared on the real time feedback screen.
Experiment 3: Voter "Jimmy"
Jimmy is a computer savvy student who has voted before but has never designed a ballot or administered voting. We conducted the test with two researchers: the first was a note-taker and acted as the poll administrator and the other played computer. Jimmy entered into our simulated polling station, and after showing ID was given a pen and told to sign in. Jimmy was then given a ballot and we conducted a few experiments on him to test our interface. In this experiment we were studying the focus of the voter paying special attention to where his eyes move, and where his focus (and locus) of attention is.
Quantitatively in this experiment we were looking to see if discrepancies between the paper ballot and the screen were caught by the voter. Also we kept count of the number of times the voter double checked his selection when it appeared on the real time feedback screen.
Experiment 4: Ballot Designer "Billy"
Billy is an extremely savvy, highly experienced survey designer/administrator. We consider him to be an expert in this field. We conducted this study in his office. We conducted the test with two researchers: the first was a note-taker, interviewer, and greeter and the other played computer. We tasked Billy with adding a write in field and a checkbox field to an existing ballot, and to add a new category. We studied mainly high level things, like where he was looking in the interface, what he clicked on etc. Quantitatively, there was not much to keep track of, but we were looking to see if the user could figure out the meaning of region and category, and whether or not he saved his work before printing/exiting.
Experiment 5: Poll Worker "Timmy"
Timmy is a tech savvy student with some experience administering surveys. He also was a volunteer poll worker. We conducted this study in his home. We conducted the test with two researchers: the first was a note-taker, interviewer, and greeter and the other played computer. Timmy's task was to simply import data and view the different histograms. Qualitatively, we looked to see if he grasped the basic elements of our interface and understood what they did. Quantitatively, we kept track of how long it took the user to complete the task, and whether or not he printed the histogram.
Results
Experiment 1: Voter "Tom"
Ultimately, the experiment with Tom was fairly successful. He had no significant problems using the system (probably partially becuase the process for voters involves instruction from the poll worker to begin with). He made a few comments when he got stuck and had to think carefully about what he was doing. He commented that it would be helpful to always have a "what should I do next?" prompt on the screen, and that he liked the visual feedback that his vote would be tallied correctly. He also commented that the boxes on the ballot that he had to check with the pen were perhaps too small for effective use.
For raw process data, see the corresponding section in the Appendix
Experiment 2: Voter "John"
The experiment with John yielded many good observations which should be taken into consideration on future iterations. John was guided through the entire process and offered some good insights and suggestions from his point of view. He raised the issue of input field sensitivity when he touched the pen in one of the selection boxes and the choice was highlighted on the screen, even though he didn't intend on voting for it.
John worked very quickly with his head down. He never looked up to check the feedback screen until he was done making selections. Because he worked with his head down, we supposed there were some errors in his vote. He noticed every discrepency from his intended vote and the on screen vote.
Experiment 3: Voter "Jimmy"
Jimmy went through the voting process fairly quickly and had no major problems. He had voted before and had used a tablet PC before, so it was completely natural for him to touch something on the paper and see a representation appear on the screen. He spent about equal time looking at the ballot and looking at the screen. For the experiment, we faked some errors in recognition, and Jimmy caught them right away since he alternated between looking at the ballot and the screen.
Experiment 4: Ballot Designer "Billy"
Billy offered many, many suggestions and complaints for our interface (see raw data for an enumeration of all of them). His biggest complaint was that it was wholly unclear how to accomplish the task using the interface, which disappointed him since the application really has one function. Some main points he raised were that the wording was unclear, that it was strange to HAVE to right click a region to add it to a category (there should be multiple ways of doing this), since right click is usually used for hidden functionality, not primary actions. His attention was spent mostly on the left pane of the screen, which makes sense since that's where the interesting stuff takes place. He also had no problem just clicking around and exploring, as any good computer scientist would do. We speculate that not everyone is so patient or willing to tinker and that we should include detailed instructions just in case. He also desired more buttons on the toolbar, similar to MS Word or Visual Studio, that would allow him the basic functionality he was used to (cut/copy/paste, align, etc). He was an experienced user, so thankfully he saved his work before printing/exiting.
Experiment 5: Poll Worker "Timmy"
Timmy went through this process fairly quickly but also offered a lot of good suggestions. Mainly, he desired consistency with existing interfaces, such as MS Word. Thus, he wanted to see the "file," "Edit," and "help" menus, as well as the print button as an icon instead of the word print. He was able to view all the histograms we provided as well as print them. He thought our application was actually lacking in functionality and wished there was more to it, such as the ability to view raw data, and export it to other formats. We agree that this functionality is necessary and will add it.
Discussion
- [David] One of the things I think is clearly important is that the process of voting not be changed for the voter. As far as he or she is concerned, voting should be as simple as making a selection. If he or she can take advantage of the information on the screen, great, but it shouldn't be necessary and shouldn't confuse or intimidate people in order to vote successfully.
- [David] One of the testers noted that on-screen guidance could be extremely useful, and I think we should support this without interfering with the critical information displayed on the screen. Voters should be guided automatically through the process of voting, both on the ballot and on-screen. The more feedback they get, the more confident they can be in the process.
- [David + Eric] We discussed the difficulty of the voting task versus that of examining the results, and think that perhaps voting should be considered a moderately difficult task, whereas examining the results should be considered a simple task. Voting involves far more human input, and thus seems more difficult.
- [Eric + Hiroki] With our current design, after the voter is finished with the ballot, our system has no way to know if the voter is still reviewing selections or has left the voting booth. We should consider methods to ensure that nobody else can see the voters selections. Possible methods could be a check box signifying the end of input or a clear screen button on the feedback screen.
- [David + Eric] A potential sticking point for voting: the screen. If too much is happening there, the user can become distracted or confused. Further, it shouldn't change the voting process, it should mainly act to reinforce it. We propose that the screen should be fully passive, indicating the selections the user has made on the screen, but not requiring the user to take any actions for the explicit purpose of changing what's on the screen. In other words, the voter should be able to vote entirely without the screen. Nothing on the ballot or in the system should depend upon the screen. Rather, the screen should simply be an additional medium for providing feedback.
- [David] Another thing to consider is that small regions for marking with a pen can pose a problem for voters as they are hard to use. This came up in our experiments. Perhaps we could suggest to the ballot-designer to make their regions larger (using a dialog of some sort) if they attempt to create ballots with regions below a certain threshold.
- [Eric] The sensitivity of the input regions is very important, especially when considering passive feedback. If the user sees a direct cause and effect between their actions and the screen it could be a source of distraction. Also it is important to consider what to do with stray marks on the page and how to handle when they enter an input field
- [Keenahn] The biggest and most consistent complaint I got across interviews was that the interface was just unclear or unintuitive. We need to resuse well known paradigms like the "file" "edit" "help" menus, cut/copy/paste buttons on a toolbar, print button as a printer icon instead of the word print, etc. Also, we need to pick a better vocabulary for what we are calling the different elements of the ballot, because "region" and "category" are unclear, even though we use the words consistently.
- [Keenahn] I think we may need to be a little more explicit in showing (instead of hiding) functionality for our ballot designer. A good interface is a balance of showing and hiding information, but in this case I think it is ok to err on the side of showing too much, since there really are not that many functions to show.
Appendix
Consent Form
PollPrecision_consent:Consent_Form
Reproduced here:
Consent to Participate in Research
Researchers: Hiroki, Keenahn, David, Herman, Eric
Description
Your participation in this study will aid us in designing and improving our user interface for a voting system. We will be interviewing several subjects in this study. Your anonymity and confidentiality will be preservered. Please answer questions honestly.
Procedures
We will give you three tasks to complete using our mock computer interface. We will not tell you how to accomplish these tasks. We will answer general questions, but will not give specific information about how to complete the tasks. Please verbalize your thought process, as we will be asking you questions about how you are interacting with our interface.
Risks/Discomforts
For our study we will be asking you to manipulate pieces of paper/cardboard with ink on them. If you are allergic to these materials you will not be allowed to participate in our study. If you are uncomfortable with the task or with answering our questions, alert a researcher immediately and we will stop the study.
Benefits
Your participation in this study will be unpaid.
Confidentiality
Neither your name nor any other identifying information will be revealed in our documents and subsequent findings. Your confidentiality is guaranteed to be maintained.
Signature
I have read and agree to the conditions outlined above.
______________________________ (Name) _______________ (Date)
Demo Script
Reproduced here:
Demo Script
Welcome to our study. Please take a seat and we will begin shortly. First, we will have you read over and sign this consent form. Remember that this study is unpaid and that your participation is completely voluntary.
Thanks. We are conducting this study to aid in the design and refinement of our user interface for a voting system. You will be assigned a few tasks and then asked to complete them using our mock interface. One of the researchers will act as the computer and show you what happens in response to your actions on our interface. Please note that no elements of the interface are set in stone, and that it is still in the alpha stage. That means you can feel free to make wild suggestions about what you would like to see in the interface. Note: we will not tell you how to accomplish these tasks, nor will we answer questions directly about the interface. If at any time you feel frustrated with the tasks, keep in mind that it is not a judgement on your abilities to interact with our interface, and probably it is a problem with our interface. We ask that you verbalize your thought process as much as possible, and we will be asking you questions during the tasks. Do you have any questions? Then let's begin.
- Low difficulty task: (Show them the ballot page) Please vote for a senator. Thank you.
- Medium Difficulty Task: (Show them the data analysis page) Please examine the data given to you. Thank you.
- High difficulty task: (Show them the ballott designer) Please add a new item to the ballot. Thank you.
Ask them questions.
Thank you for your participation. Your input has been very useful.
Voter Experiments-- Raw data
David Poll... Played poll (forgive the irony) worker and computer
Eric Vacca... Note-taker
- 3 -- Minor usability problem -- "What is the next step? What should the voter do next?" needs to be displayed on the screen while voting... User knew what to do, but only from experience and commented that he could have gotten lost if he weren't making an effort to get every step right.
- 2 -- Cosmetic problem -- Size of regions/fields needs to be larger. This is more of a problem with the ballot design, but a reasonable consideration. We can suggest minimum region sizes, perhaps.
Hiroki Terashima... Guided the voter and acted as the computer
Eric Vacca... Played poll worker and was Note-taker
- 4 -- Major usability problem: important to fix -- When the voter is finished, the feedback screen remains for the voter to check over his selections, but remains on at least until pen is docked. Voter confidentiality is violated.
- 3 -- Minor usability problem -- How sensitive is the field? Giving the user active feedback almost encourages experimentation with the sensivitity of the system. The voter was interested in what happened when he simply touched the input field.
Keenahn Jung... greeter, interviewer, note-taker
Eric Vacca... Played computer
Easy task:
- 2 -- Cosmetic Issue: needs a confirmation screen after voting.
- 2 -- Cosmetic Issue: Meaning of "check in" is unclear.
- 1 -- not really a flaw: User did not understand the point of the receipt.
Medium task:
- 4 -- Major Usability Problem: Not clear how sign in sheet will be organized or how a user prints out the sign in sheet.
- 5 -- Major Usability catastrophe: Not clear how the administrator gets to the "accept votes for station" screen. There is no security for this.
- 1 -- not really a flaw: No user interface for the server at the centralized voting center.
- 2 -- Cosmetic Issue: "Accept votes" and "continue voting" should really be "yes" and "cancel" and the question should be stated more clearly.
- 4 -- Major Usability Problem: Not clear the sequence of screen clicks required to get to the screen with the data. Not clear what they would have to click on to import the data (or export it to other formats).
- 3 -- Minor Usability Problem: It is not clear the purpose of the drop down menu of categories. "Select the category to view result" should be "...results" to be consistent with the rest of our app.
- 2 -- Cosmetic Issue: Not clear where the top field that displays the date the results were sent gets that info. Is this the date of the last sent data?
- 4 -- Major Usability Problem: Missing standard menus like "file" "Edit" "help" etc.
- 2 -- Cosmetic Issue: Print and exit buttons should be up top, not off to the side. Not necessary to have an exit button.
- 4 -- Major Usability Problem: Not clear how to view the raw data. Also, need a way of modifying how data is displayed (drag the columns around, reorder them etc).
Hard Task:
- 3 -- Minor Usability Problem: The word "regions" is unclear and should be "field."
- 3 -- Minor Usability Problem: The word "category" is also unclear, but not sure what we should change it to. SUggested "question"
- 3 -- Minor Usability Problem: Need cut/copy/paste buttons for regions. Need to be able to cut/copy/paste regions. Maybe these buttons should be in a toolbar.
- 4 -- Major Usability Problem: Not clear how to associate regions with a category. Needs to be other ways to do it (through menus, or maybe its own toolbar) other than just using right click.
- 4 -- Major Usability Problem: Not very clear how to add regions. There should be a button that just says "Add a region" that adds a medium sized region to the page, which can then be moved and resized. Not clear at all how to delete regions. User guessed to use delete key based on past experience. Should be an option in right click menu.
- 3 -- Minor Usability Problem: Regions should be multi selectable and groupable using the mouse. Should not have to assign each one individually to a category.
- 2 -- Cosmetic Issue: Wording "region category" is confusing. "Category of selected region" might be clearer. Also, the "region category" pane is huge, it should not take up so much space.
- 4 -- Major Usability Problem: Need a way to associate a category with a certain area on the page. Need a way to quickly highlight all the regions in a category.
- 3 -- Minor Usability Problem: Need alignment buttons (left, right, center) and "snap to grid" functionality for regions.
