PilotStudy-Group:Group H

From CS 160 User Interfaces Sp10

Jump to: navigation, search



The system being evaluated is the iDeck iPhone application. iDeck is a social card-playing environment that aims to simulate the freedom of playing at a card table. The app will have the ability for users to connect to either friends or strangers in a game lobby that simulates a card table, and play using the affordances of the iPhone. There the users will be able to communicate with the whole group and play any game they want with the provided deck, just as they would if the users were to meet up in real life. The purpose of this study is to see how well the user testing will be by first trying it out on "pilots" before testing a larger user group. The pilots will be more well versed with the application and the study than users in a user testing experiment. The pilots' main goal will be to see the effectiveness of the application for testing, as well as the process of the test itself. The application requires playing on a network, and this pilot study will be used to see how effective our network setup is, and how we may improve upon it as to create a more realistic scenario for user testing. The pilot study will also be used to help reiterate the app before user testing, so that the application will be closer to the finished product and more appropriate for future user study groups.

Implementation and Improvements

  • Added a "create table" button at the top (in the navigation bar) of list of tables
  • Now simply clicking on a table cell takes you that table.
    • "Join table" button removed
  • Clicking on the name of a user in the contacts page will join their table
    • "Join user's table" button removed
  • Each cell of the table view in the list of tables are now divided into a primary label, secondary label and a image.
  • Each cell of the table view in the list of contacts are now divided into a label and a image
  • The application now has a more attractive main menu with redesigned buttons
  • The table view's background resembles a poker table, instead of only a green background
  • The table view has buttons that lets the user deal, collect back the cards, send messages to other players in the table, and exit the game
  • The application doesn't let the user start unless there is a username in the username textfield
  • Removed Joker field from Table Creation settings
  • Removed the minimum bet field

Some of these changes were implemented after the first pilot study.



For the pilot study, the main goal was to have users who were well-versed enough in technology that they could easily use the iPhone application and comment on the app both for its functionality and design separate that of the card-playing aspect. The should also be focusing on the study itself, and how we could improve the study for a larger user-base that is more focused on the card-playing functionality and may have little to no iPhone experience. The pilots should still preferably have some card playing experience, however.

We posted some signs on the southside of campus where many students live. The sign was a simple piece of paper with an email that you could tear off. The title was, participate in iPhone Card game application test. We found several people in a short period of time with this method, and we choose three suitable participants. The first participant was a 21-year-old male who regularly attends poker tournaments on campus and frequently plays poker with his friends. The participant owns an iPhone and has used other card game applications. For the report's purposes, we shall refer to him as Max. The second participant was a 26-year-old male who has attended several large poker tournaments and occasionally plays card games on web applications. We shall refer to him as George. The third participant was a 23-year-old female who plays online card games in her free time. She does not consider herself a card enthusiast, but likes to play for fun. We shall refer to her as Ann. We believe this provides a range of user experience levels and user types that would be part of the target user group.


We used 2 laptops for the test with the iPhone simulator installed in them. We were unable to use actual iPhones/iPods due to compatibility issues with the iPhone SDK at the time of testing. Given the nature of a networked game, we needed to have a minimum of two players at any given time. Thus, we ensured that there would always be at least one player and one dealer at all tests. Since we only had one pilot user doing the study at a given time, the other laptop was used by a group member to simulate another player and was situated so that the two laptop users could not see each others' screen. The testing was done at a house on the south side of Berkeley campus. Additional tests were done at Milano cafe on Bancroft Avenue.


We decided to switch the medium and the hard task from the LoFi prototype. We also made few modifications to the easy task. So the tasks we asked the particpants to complete were the following:

Easy Task: Join a poker table

Our application should allow users to play a quickstart game, so we incorporated persistence so the user would not have to re-enter his username before playing. The user's goal is to quickly join a poker table by opening up the app, tapping on the "Tables" button on the main menu, and then identify a poker table that he would like to join from a list of available tables. From there, he should intuitively select the table to join.

Medium Task: Play one poker round in the table you just joined

Playing a card game is what we want our users to enjoy from our card game application. We were looking to see if the user was able to handle the buttons and the gestures. We also observed if the icons and images made enough sense to the participants and if they had the correct idea when these are tapped.

Hard Task: Create a BlackJack Table

Creating a new table takes time, and we were looking if the user could create their own personalized table with the settings provided. We also wanted to make sure if the participants understood what the table settings were asking them and how it will affect their gameplay. We also looked if the table settings were enough to meet the participants' needs for them to play their own personalized card game.


Given the nature of our application, the more players the better. The game doesn't really make sense unless another person is also playing the game. With this as a constraint, we required an additional player at all times that could play a game with a person. Therefore, we had one group member as an additional player in one of the laptop computers.

We first started by describing the background of our application to our participant, and then we allowed him to read and sign the consent form [1]. After signing the consent form, the facilitator logged in as "johnDoe" and performed a quick demo [2] of the application . In the demo, the facilitator just briefly described some of the elements in each screen he went through, without revealing any hints on how to perform the tasks. The participant was then asked to complete the three tasks. The easy task started off with the participant facing the main menu, and he/she to find a poker table and then join it. When the task was completed, we let the participant stay in the poker table, where the other player was a team member. The rest of the team were acting as observers and taking notes. Then the participant was asked to complete the medium task, which is to play one round of poker with the team member. Once the participant finished the medium task, he then had to complete the hard task which consisted of creating a blackjack table. The hard task started with the participant in the main menu. At the end of the test we conducted a quick debriefing session to ask the participant what he thought of our application and if any of his expectations were met. We also asked him/her about some incidents that the observers recorded and were unclear about. We finally thanked the participant for their time. The full script can be found in the appendix [3]

Test Measures

The number of occurrences of the following test measures were recorded during each pilot study.

  • Confusion: we are aiming to provide a flexible card application that resembles the real world environment, and we want to avoid any confusion that the user might have due to the way the application is structured. We measured the number of confusions by counting the times the user looked lost or when he/she was making long pauses. Also we encouraged the participants to express their feelings, so we counted any sign of confusion from what they were saying.
  • Frustration: we want to reduce the number of frustrations since these can delay the completion time, or even discourage the user from completing the task at all. We counted the number of frustrations by what the participants were telling us.
  • User Errors: we want to make sure the application is easy to use. We counted the user errors by counting the number the times the user did something that got him unexpected results. For example, when the user taps on the wrong button and realizes he is in the wrong place.
  • System Errors: counting the number of system errors will give us an idea if the application is fully functional and if we need to fix some errors for the next iteration. We counted this by recording the number of times the application crashed or did something unexpected.
  • Task time: our card game application should be as straightforward as playing a real physical game. We counted the minutes each participant took in completing their tasks.

Results and Discussion

Results of the tests

Results of pilot study

The pilot study was very helpful in critiquing our app and our experiment. Detailed logs of the pilot studies can be found in the appendix [4]. The easy and hard task seemed simple enough that all pilots were able to complete fairly easily without any confusion. The hard task was considered hard because it required more thought to plan out all of the fields for a table, and can be reduced in difficulty by creating two tabs, one that holds very simple and often changed fields, and another advanced tab for some of the more specific changes that an advanced user might want. The medium task posed the greatest difficulty for the pilots and will need to be better described in our user testing experiments. We found that the pilots were unclear of what to do at a table since they expected game logic and/or message cues, something found in the other tasks. For example, a user would navigate to the tables page and choose a table with no problems. However, once this user is at a table, problems arise. There are implementation reasons for this, which will be discussed here. First, if a game is already currently in play, a user will show up to a table with cards already dealt, and there is no message indicating to the user that a game is in session and he/she will have to wait until the next round. In addition, if a user goes to a table where there is no players, it is unclear of what to do. These results will be applied to reiterate both the app and the user study experiment so that the participant will not be confused on what to do during the game table.

First participant

Our first pilot was Max and some of the result we found we applied before the second testing. Many of his complaints were addressed by a reiteration of the app before continuing with the pilot study. One of the main problems that caused confusion and frustration was that the app promised functionality that it didn't have. There was a minimum bet field in the table creation settings that Max asked about, because our app did not have a monetary system implemented at the time of the test. We decided to remove this field before the next tester as to avoid confusion, and will put the functionality back in when we have implemented the monetary system. Another aspect of the app that was removed was the field to include jokers into the deck, something we realized that we did not implement and should therefore not include. Max also pointed out the large number of text fields, especially for fields that would only be numerical, like the number of players or decks, and would prefer a simpler solution.

Second participant

Our second pilot was George and he seemed to have an easier time than Max when we removed many extraneous functions and fields. Because of this, George completion time of the hard task was 51 seconds less than Max. For George, joining a table was fairly straightforward. In addition, the number of confusions and frustrations reduced to half with this new iteration of our application. However, George's confusions were due to the lack of feedback of in-game progress and functionality during the game. Finally, creating a table was frustrating due to excessive text entry of settings, and also confusing due to the gesture settings.

Third participant

Our last pilot was Ann, and like George, she faced less confusions and frustrations with the application than Max. She was able to understand the tasks clearly and was quick in completing them. She didn't have any trouble joining a poker table or looking where she could create one. However, she was confused when she was creating a blackjack table because some of the settings were not clear to her in what they were asking. She believes that the table settings should be more self-explanatory because she personally doesn't like to take time reading the help file.

What we learned from the user study

The study was helpful in finding out how users might react to the tasks. The easy task was simple and straightforward, which is what we were aiming for. The medium task proved to be more difficult because there is so much freedom that the task description and the app did not help the users to understand the steps he needed to do to achieve the goals. The hard task was fairly simple, but the pilot study showed us that there were too many text fields in the table creation view, something we can hopefully improve upon before larger user studies. The pilots helped us in pointing out simple design choices that could be improved, such as using a UIPicker for fields in the table creation view that required just numerical input, as well as including help guides on how to play in the table view. The study is helping us to improve the application and let the users in future studies focus on finding less obvious, but equally bad, design choices. The pilots also underlined our main problem, which is that the application is frustrating to use since it has extraneous fields and a lack of direction. An upside is that one of the things we wanted to quantitatively measure was system error and that the app was stable enough as to not have any errors.

The changes we plan to make include having help guides easily visible in the table creation view, as well as the tables view. Obviously if we can find a way to give directions easily without needing a help menu, we will try to implement it. The pilots suggested to streamline the app by removing any extraneous functions or fields, and that the table creation settings be simplified for beginners, while having more advanced options tucked away for more advanced users. The user interface of the contacts will also be improved upon so that there are either online/offline groups, or that a friend system will be implemented. The overall interface will also be improved upon to have better graphics, as well as button layout using the suggestions of the pilots. The pilots also suggest creating a more intuitive table view, one where the buttons are obvious in their functions and where your cards are should also be obvious to the user. The users wanted the ability to message others and that is an important feature that we are working to implement currently.

Other major changes including handling the bug that occurs when someone leaves a table. The bug causes their cards to stay, and there is no notification to the other players; this can be very confusing when another person is waiting for a player to take their turn. We will also implement the gesture functions so that finger gestures will cause appropriate messages to appear when in table view. Another major function to be implemented is how the table will assign a new dealer when the current dealer leaves the table.


Script of Experiment

Greeter: Welcome to our card game application for the iPhone/iPod called iDeck. iDeck is an application that will allow you to play with friends in a virtual world without being in the same physical room. You as an experienced card game player will go through this application and tell us what you think about it; if it satisfies most of your expected needs when playing on a real table. Right now your are facing the main menu of the app, and from this you can go and create a table or join an existing table. While the rest of the team finishes setting up the testing prototype, you can start reading the consent form and sign it.

[Hands the consent form to the tester]

Now our team facilitator will explain how this testing application works and after showing a short demo, he will ask you to complete three tasks.

Facilitator: You are now we are facing the main menu of the iDeck application, and from here you can join a playing table or create a new one. As you can see, there is a username box in here. In the final product, we expect each people to have their own username, but for this testing session, you will be logged in as "johnDoe."

I will now show you how to start using this application. First we need to log in with our username.

[Types in "johnDoe" in the username text field]

Once you are logged in, you can check which contacts are online.

[Taps on the contacts button]

This screen shows you the list of your contacts, which are the people you can play with. We can go back to the main menu using the "Back" button on the Navigation toolbar.

[Taps the "Back" button]

You can also view which tables are currently playing by going to tables.

[Taps on the tables button].

This is the list of all the tables that you can join and are currently playing.

[Taps the "Back" button, back to the main menu]

The switch at the bottom of the main menu screen is to set your status either to online or offline [taps the switch to show how to change from online to offline and vice versa].

Now I will ask you to complete three tasks. I encourage you to think out loud and express any of your thoughts during the whole test. You are allowed to do any iPhone/iPod gestures such as single tapping, double tapping, and sliding one or multiple fingers. The application will respond appropriately to your gestures. There might be gestures which are undefined, and in that case, you will not see any response. You might want to try a different gesture and see if that has a desired effect on the test. None of the team members will be able to help or speak to you. You might at some time feel stuck, but you are encouraged to do your best. Please do not worry if you feel that you are "failing" the test. This interface is on trial, not you. So if you fail to understand something or you can't complete one of the tasks, that's a sign of trouble with the design, not a lack of intelligence on your part.

During the test, the observer will take notes quietly on any relevant reaction you have on the interface. At the end of the test, we will have a debriefing session to ask some questions and gather any of your impressions.

Now that we are in the main menu, your first task will be to join a any table where poker is being played. Please start.

[After the user completes first task, keep the table view open] Now that you have joined a poker table, your second task consists of playing one round of poker. Please start.

[After the user completes the second task, open the main menu] Your third task is to create a blackjack table with personalized features of your choice.

[After the user completes third task] You have now completed the test. What did you think about it? Any comments, questions, impressions?

[Take at most 10 minutes to do a debriefing session]

[After the debriefing session] Thank you for your time, we really appreciate it. Your results will help us improve our interface and create the best card game experience.

Consent Form

You are invited to participate in a study of the user interface design for a playing card iPhone application. We hope to learn about any usability problems encountered trying to accomplish tasks with the interface. You were selected as a possible participant in this study because we believe you fit in our target user group of card enthusiasts.

If you decide to participate, we will conduct an experiment as follows. First, we will demonstrate basic interaction with the interface and perform a simple task. After the demonstration, we will ask you to complete three separate tasks. For each task, we will first describe the task, and then ask you to complete it. However, we will not explain how to complete the task. The total time for the experiment is expected to be about 20 minutes, including a short debriefing session.

Any information that is obtained in connection with this study and that can be identified with you will remain confidential and will be disclosed only with your permission. Your decision whether or not to participate will not prejudice your future relation with us. If you decide to participate, you are free to discontinue participation at any time without prejudice. If you have any questions, please do not hesitate to contact us. If you have any additional questions later, please contact us at [removed for privacy].


You are making a decision whether or not to participate. Your signature indicates that you have read the information provided in this form and have decided to participate. You may withdraw at any time without penalty or loss of benefits to which you may be entitled after signing this form should you choose to discontinue participation in this study.


Signature and Date


Signature of Investigator

Logs of Pilot Studies

First Test: Max

  • Easy Task Time: 12 sec
  • Medium Task Time: 1 min 2 sec
  • Difficult Task Time: 2 min 34 sec
  • Occurrences of Confusion: 10
  • Occurrences of Frustration: 5
  • Occurrences of User Error: 1
  • Occurrences of System Error: 0


  • Easy task: join a poker table
    • sees table button
    • hit tables button
    • sees a lot of tables and scrolls down
    • looks for a poker table
    • sees there are two poker tables
    • sees that one has 4 and the other 3 people in there
    • he likes to play with a lot of people so clicks on the one with four
    • he sees himself in a new table view (End of task)
    • Time: 12 seconds
  • Medium task: Play a game, one round
    • He sees everyone got dealt five cards, a 10 of spades, a 6 of diamonds, a jack of diamonds, a 4 of spades, and 6 of clover.
    • He sees his hand as up but everyone’s cards are down
    • He is not sure if they can see his cards- user state (confusion 1)
    • He is frustrated he cannot ask what to do, or who’s turn it is (confusion 2, frustration 1)
    • He double taps one one of his cards, and nothing happens (confusion 3, user error 1)
    • He looks that another player moved one of their cards into the middle of the table, and another player put in 3 cards into the middle of the room
    • He assumes he can do the same, so he drags three cards into the middle of the table, a , his 4 of spades, his 10 of diamonds, and a jack of diamonds
    • He wonders what is going to happen now (confusion 4)
    • Surprised to see cards from the deck has moved to other player on the right
    • Then he receives gotten cards from the deck
    • Time: 1 min 2 sec
  • Hard Task: Create a blackjack table
    • He realizes he has to go to tables to create tables since there is no create table button in the main menu
    • He sees a bunch of tables and sees on the top right create a table
    • He is overwhelmed by the amount of text boxes (frustration 2)
    • He sees the first thing, table type. He is not sure if his selection will affect the game. If he puts blackjack, will he be able to play poker? (confusion 5)
    • He types blackjack
    • He wishes to have a next button in the keyboard like safari to switch to the next field (frustration 3)
      • Is slow to him
      • Not intuitive
    • Types a name for the table
    • For number of players, he wonders if there is a drop down menu instead of typing a number (frustration 4)
      • Wonders the same for number of decks
    • Why would I need multiple decks? (confusion 6)
      • He only plays with one deck most of the time
    • Cards face up: he puts 0 because he doesn’t want cards to show. “Will this not show my cards, will it affect me or others?” (confused 7)
    • “What’s the point of minimum bet, I don’t have money” (confusion 8)
    • He doesn’t want jokers so he leaves it as is. He likes that is only a slider, yes or no
    • He doesn’t know what to do with gestures. He doesn’t know what gestures mean, he looks confused. (confusion 9)
    • He goes to help. Angry because help button is not at top and visible (frustration 5)
    • He looks what gestures are and has an idea.
    • He is wondering where the messages will show up. He remembers when playing that he didn’t see any message box. (confusion 10)
    • Very confused and doesn’t care much. He leaves it blank
    • Hits create table. (End of task)
    • Time: 2 min 34 sec
  • Results:
    • he is frustrated because he cannot talk to others
    • No help functions so he doesn't know what he can do with the cards
    • No help menu so he doesnt know how to play the game
      • Cannot take initiative because he doesn’t know what to do
      • Have to learn through imitation (what if they’re all new?)
    • Doesn’t know if it’s his turn, could be fixed through messagebox

Second Test: George

  • Easy Task time: 16 sec
  • Medium Task time: 1 min 40 sec
  • Difficult Task time: 1 min 43 sec
  • Occurrences of Confusion: 3
  • Occurrences of Frustration: 3
  • Occurrences of User Error: 0
  • Occurrences of System Error: 0


  • Easy task: join a poker table
    • hit tables button
    • glances through and thinks out loud, "Hm, which table should I join?"
    • Scrolls a bit, then joins poker table with three users
    • He sees himself in a new table view and says, "That looks cool, it's like an actual table." (End of task)
    • Time: 16 sec
  • Medium task: Play a game, one round
    • He was dealt a 7 of hearts, 7 of spades, 3 of diamonds, 5 of clubs, and queen of clubs
    • He recognizes his hand is face up but everyone’s cards are down, but is a bit confused about the position of his hand in the table view (confused 1)
    • He thinks about what he is supposed to do and is unsure what is happening as far as the game's progression goes (confusion 2)
    • He sees other users cards being dragged towards the table and does the same.
    • At the end of the round, he is a bit frustrated by the lack of feedback about what is going on in the table (frustration 1)
    • Time: 1 min 40 sec
  • Hard Task: Create a blackjack table
    • Initially he is unsure where to create at table so he checks the Tables menu.
    • He taps on the create table button
    • He types taps on the Table Type textfield and types in BlackJack. There is frustration when he mistypes some letters. (frustration 2)
    • He expresses more frustration when continuing to enter all the options for the table settings. (frustration 3)
    • He understands most of the options up until the gestures. He looks at the help menu but is not sure how the gestures are used. (confusion 3)
    • Time: 1 min 43 sec
  • Results:
    • Confusion with in-game functionality and progress, lack of feedback
    • Confusion with gestures settings
    • Frustration with entry of table settings

Third Test: Ann

  • Easy Task time: 9 seconds
  • Medium Task time: 1 min 53 sec
  • Difficult Task time: 1 min 18 sec
  • Occurrences of Confusion: 3
  • Occurrences of Frustration: 1
  • Occurrences of User Error: 0
  • Occurrences of System Error: 0


  • Easy task: join a poker table
    • sees the tables button and hits it
    • goes through each of the tables and taps on the first poker table she sees
    • she sees herself in a new table view (End of task)
    • Time: 9 seconds
  • Medium Task: Play a game, one round
    • The participant says "cool" referring to the graphics in table view.
    • The participant receives a 7 of spades, 3 of clubs, an ace of spades, a 10 of diamonds and a 7 of diamonds.
    • Confused on how to start playing. She is not sure what to do to start playing. (confusion 1)
    • She drags her 3 of clubs card and drags it around to see if it is movable.
    • She sees other player moving one of his cards to the middle of the table
    • She decides to do the same and puts her 3 of clubs card in the middle of the table
    • She gets a little frustrated because the app takes a while to respond. She states that she is used to quick action when she plays poker online in the computer (frustration 1)
    • Time: 1 min 53 seconds
  • Hard Task: Create a blackjack table
    • She goes straight to tables by tapping on the tables button
    • She sees the create table button in the navigation bar and says, "Oh, there it is."
    • Now she is facing the table settings.
    • In table type, she types "Blackjack"
    • In Table name, she types "master table"
    • In number of players, she selects 5
    • In number of cards face up she is not sure what it means. If it's per player or per table. She types 0 because she thinks in blackjack, "you don't show your cards to anyone" (confusion 2)
    • In cards per hand, she selects 2
    • She is not sure what "messages sent using gestures" mean. She then remembers that there wasn't any message system at all while she was playing poker in the medium task. She is confused, but she proceeds to type "hit" on double tap after looking at the placemark in the textfield (confusion 3).
    • For horizontal swipe, she types "stay" after looking at the placemark in the textfield.
    • She taps on the "create table button"
    • Time: 1 min 18 seconds
  • Results:
    • Confusion with starting a game
    • Confusion with game settings - number of cards face up, gestures
    • Frustration with responsiveness of app in game

This page was created by Group H.

Go back to Pilot Usability Study

[add comment]
Personal tools