Pilot Usability Study - Group:Fatal Exception

From CS160: User Interface Design Sp12

Contents 1 Introduction (5 points) 2 Team Members 3 Implementation and Improvements(15 points) 4 Method (10 points) 4.1 Participants 4.2 Apparatus 4.3 Tasks 4.4 Procedure 5 Test Measures (5 points) 6 Results and Discussion (25 points) 7 Appendices (5 points)

Introduction (5 points)

The path to obtaining a driver's license can be very difficult. For a small group of learning drivers, the driving test evokes a great sense of fear because of the highway driving portion. These drivers lack coordination, good reaction times, and driving sense required for highway travel. Our mission is to provide an environment for these learning drivers to practice highway driving skills without being afraid of crashing. This will help ease the transition into highway driving, rather than the experience being a 'dive into action' ordeal. Our goal will be satisfied if drivers who previously stalled on getting their license because of fear become more motivated to get their license.

The experiment is to measure ease of use of interface elements. Does this interface provide a comfortable driving experience? Is it easy to use?

Team Members

Wenjie Zhou : improved acceleration constants, improved cursor, interviewed Sophia, contributed to writeup

Brandon Young : Implemented video player for task 2, interviewed Leslie, contributed to writeup

Neel Rao : Improved visual elements, improved braking, improved help screen, interviewed Valerie, contributed to writeup

Omar Ali :

Implementation and Improvements(15 points)

• Improved cursor movement recognition
• Implemented a limited video player to simulate task 2
• Better control, both hands can be used as a cursor
• Made it easier to select buttons, fixed button selection glitch
• Improved gesture recognition while driving
  • Acceleration constants made more realistic
  • Steering wheel angle calculations more accurate
  • Braking feels more like braking in a real car
• Working on implementing a 3d prototype mapped to gestures
• Better icons for each task help users identify them
• More help options for first time users
  • Explains how the cursor works
  • Explains button selection

Method (10 points)

Participants

Participant 1: Sophia Shi

She is a full time student in community college in Oakland. Sophia has had her license for years, but when it comes to driving, she is still extremely nervous, especially when hitting on the freeway. She is terrified for driving because she fears being in an accident. She is extremely fearful of changing lanes, both locally and on the freeway. Changing lanes may seem very easy on the freeway, but with her being scared of speeding up, she is basically stuck on the right lane where people merge in from on-ramps. With so many drivers going over 75MPH zipping around each other without signaling, she is again stuck on the same lane. It seems that she would get the most benefit from our hard task. This application is also helpful for removing her from the fearful environment which interferes with remembering her driving skills. She has never used a Kinect before.

Participant 2: Leslie Valencia

Leslie is a 22-year-old college student from San Fernando Valley, California and does not have a drivers license. A driver’s license wasn’t necessary in her household, and she does not know how to drive in general. That said, she is interested in learning how to drive. She tried driving before, but didn’t feel comfortable driving by herself more than 30 mph. She has little sense of space and finds herself drifting from side to side, and can’t drive in a straight line. She has also never driven on the highway before and never received driving lessons. She also perceives cars as destructible and possibly dangerous. This application is suitable for removing her from that driving anxiety to improve her skills. Our hard task is also suitable for improving her tendency to drift side to side. She has never used a Kinect before.

Participant 3: Valerie

Valerie is a college student from AZ. She does not yet have her license because her driving test requires highway driving. She is comfortable with everything else, but the highways still phase her out. At those high speeds, Valerie feels like anything could go wrong. She feels that in wrong turn, or in a quick turn of the head something crazy could just appear in front of her. She believes that she will freak out and react horribly. Eventually Valerie wants to get her license and she knows that all it requires is a change of mindset. Highway MyWay is perfect for her because she use it to get comfortable with her ‘highway sense’ so she knows she won’t freak out when she’s actually driving. She has never used a Kinect before.

Apparatus

Laptop w/ kinect hooked up to a larger tv screen. We set up a chair for the users to sit in and made sure they had enough space for gestures. We had additional laptops for taking notes and recording additional information. We had a notepad to jot down any quick notes.

Tasks

• Intro Tasks: These are the demo tasks we used to describe how our interface works and maps with gestures. We showed the start menu and described how buttons are pressed, and then showed our task menu and how to navigate it. After we demonstrated these tasks we asked our testers to try it themselves.

• Easy Task:In this simulation, user need to brake as soon as possible when the car in front of user makes a sudden stop. Start with bring right foot back to slow car down, and move right foot towards left foot in a braking motion to brake when the car in front of user stops.
• what are we looking for in testing of Easy Task: we want to test if the users can easy control the speed of car by moving their foot. Moving forward for speeding up and moving backward for slowing down. And we also need to test what is the feelling of user about putting two feet together to fully brake the car(where 0mph shows up).

• Medium Task:In this simulation user will practice merging onto the highway. Make sure to merge at the proper time and not hit any cars! When the lane is free, use hands as a steering wheel to merge.
• what are we looking for in testing of Medium Task:we want to test if the users can easy control the speed of car by moving their foot. Moving forward for speeding up and moving backward for slowing down.And also to test if users are able to merge to highway in a right timing by using our application.

• Hard Task:In this simulation user will practice changing lanes. Raise and lower his or her left hand to signal, and steer with both hands. Move his or her right foot forward to accelerate and bring it back to stop accelerating. Move his/her right foot towards the left foot to brake. He or she have to quickly swerve out of the lane by turning his or her hands when an obstacle presents itself.
• what are we looking for in testing of Hard Task: In this task, we want to test if the users is able to choose a right timing to change line by controling the wheel and by controling the speed by move their right foot.

Procedure

We first introduced users to our app and what the goal of it was. Then, if the user didn’t know, we explained what the kinect was and how best to use it. After the intro, we moved onto the application. Before we introduced the three tasks, we showed how our kinect cursor works (use your right hand as a cursor, and hover over buttons for 2s to select it) on the main menu and task menu. Once the testers were familiar with controlling the interface, we asked them to start at the main menu and navigate to the first task. We described to the testers what the purpose of each task was, and we observed how they made gestures and how they believed gestures mapped to our interface. All of our testers understood that using two hands will control the steering wheel. While each task was being performed, we had two team members controlling the interface, making it react with the testers motions. We had another team member recording observations, and the last team member with the testers directing what tasks they should do.

Test Measures (5 points)

General

• How clear is the cursor design? Were users able to figure it out on their own?
  • The cursor is the main and only way of navigating the UI, so it should be slick and easy to use.
• Have users used similar Kinect cursors before?
• Were users able to select the desired buttons? How long did it take to select the start button (without instruction).
  • We want the time to start the game to be very low.

• Was the help menu useful?
• Would you have liked more help?
  • This is to test if our documentation is insufficient.
  • This is the best way we have of telling users what is important.

• Can the users pause the game when asked? Do users pause the game unintentionally?
• How comfortable do you feel sitting, pretending like you are in a car?
  • We want to prioritize user comfort, we are trying to fix discomfort while driving.

Easy Task

• Did the users understand the goal?
• Is the braking animation similar to real life? Did the user learn it immediately?
  • We want braking to feel very natural, as it is a very important reaction time based action

Medium Task

• Did the users understand the goal?
• How is the acceleration gesture? Did the user learn it immediately?

Hard Task

• Did the users understand the goal?
• How is the steering wheel animation? Did the user learn it immediately?

Results and Discussion (25 points)

• Users were not familiar with standard Kinect interface features such as calibration.
  • We shouldn't rely on user intuition, they may have not used a Kinect before.
• The interface was fairly easy to pickup, but users had trouble sometimes recognizing that the Kinect did not detect them.
  • Perhaps we should make the skeletal viewer larger, or tell the user that the Kinect lost their skeleton detection
  • Tell the user how to get detected again.

• When given instruction, users could start the game using the Kinect cursor in an average of 5 seconds.
  • The start menu looks like it's working well so far, we were shooting for a fast startup time.
• Some users complained of a slow cursor.
• The help documentation was useful to our participants.
• Task wise, the users were able to understand all goals except for the medium difficulty task.
  • We need more description/documentation for the medium task.

• Accelerating
  • Users felt like accelerating was a little weird because 'pressing on the pedal' should add to your speed, not dictate it. We could scale the mapping between user foot position and speed to make the movement more realistic.

• Turning
  • Feels very natural and intuitive, all of our users were able to figure this one out. Just pretend like you are holding a steering wheel.
  • One user accidentally paused the game when turning because her hands were too close together.
  • The same user thought the reaction time required to turn in task 2 was too little.

• Braking
  • Mixed response: It's easy to perform but there are also ways to do it that will trick the kinect into thinking youve done it with minimal effort. We could change the way the gesture is detected.

• ‘Two hands together’ to pause the game is a nice design.
• The way to choose a icon and click that icon is not easy to be familiar with at first.
  • Perhaps show a quick help icon that shows users how to use the icon.
• The cursor is not agile enough.
  • We should implement more mouse smoothing and maybe change the timeout delay to 3 seconds instead of 2
• Gestures difficult to learn/not appropriate
  • We are trying to make the gestures as close to real driving as possible. The target users already know how to use car controls and are really using our app to gain comfort in their coordination/reaction time skills.
  • Perhaps we could implement gesture recording so users can put in what they feel is realistic.
• Tasks were simple and easy to understand. They were representative of real fears.
• Not enough feedback
  • Unfortunately all we can give is visual feedback right now, physical feedback is impossible but audio feedback may be possible. Sounds are a very useful cue in real life

Appendices (5 points)

• Pre-Test: "This camera will record your motions and send the information to the program. You can pause the test at any time. To test this interface, we'll be giving a minimum of instructions, but if you get stuck we can help."

• "In order for the kinect to detect you, you will need to stand a certain distance away (demonstrate to user). You will know that the kinect detects you because a skeleton will be overlayed on your body. Look for the skeleton in the upper right corner (demonstrate to user again). Try standing in front of the camera and let it detect you."

• "We will be timing how long it takes you to start the application. Use your right hand as a cursor and select start."
• "What are your opinions on the cursor?"
• "Try it again by selecting back to go to the main menu."
• "Have you ever used a kinect interface before? Does the hover button concept make sense to you?"
• "Now, sit down and try selecting start again"

Raw data

Sophia:

• when in the main menu, it took 6 second for Sphia to select the "Start" button.
• Sphoia says "i am not ready to get into this task now". the hove on 2 second to get into the task seem too short for her. she havenot read all the description for that task yet.
• when Sphia try to selete the Easy task from the left side of the screen, she can not reach this icon using her left hand even she streche her whole body to left.
• after get into easy task, Shphia feel exited when the speed meter is changing as her right foot moves.
• In easy task, Sophia think that is is not reasonable to fully break the car on high way.
• In medium task, Sophia want to pause the task and return to the main menu. she likes the design fo the gathering both hand to pause the task.
• In hard task, the big stone on the row is intersting rather than a real car image.
• Sophia think we should make the background for each task more real.
• Completed easy task.
• Completed hard task.

Leslie:

• Took ~4 seconds to select the "Start" button.
• When turning, paused the game because hands were too close together.
• Said the cursor was not sensitive enough.
• Said the reaction time required for task 2 was too short.
• Completed easy task.
• Completed hard task.

Valerie:

• Took ~6 seconds to select the start button.
• Completed easy task.
• Completed hard task.
• Understood task descriptions/icons.
• Valerie was short and the cursors didn't work as well for her. But she was still able to use it.
• She kept wanting to stand up to select buttons because she felt like she couldn't 'reach them.'
• Complained that there was a lack of documentation for the cursor
• Found an easier way to brake and did that instead, although it didn't look like a real braking gesture at all.
• Since we didn't implement any speed maximums, she would accelerate to ridiculous speeds.

---