User Testing

Introduction

While we feel that virtual travelling is exciting, we want to know if real people would want to use our application. We wonder whether or not it is realistic and engaging enough to create an immersive experience. If not, we want to find out ways we can improve the user interface to achieve that experience.

Does our application and gesture set provide sufficient functionality for exploration? Do our gestures feel natural as opposed to cumbersome? What do people want to do in a virtual world? We hypothesize that users will prefer and appreciate simple gestures over physically demanding ones. We hypothesize that users will be interested in performing actions (e.g. flying, super-speed) that they cannot do in the physical world.

Methods

Recruitment:

1) We set up the experiment in an open, private space nearby a public area from which we could recruit participants. We used Old Union room 122, a private room in the bustling Old Union to run the experiment.

2) Send out an e-mail with a short description and invitation survey to participate in our study. Confirmed participant statistics as of 3/2:

  • # of users: 7
  • 2 male, 5 female
  • 4 Stanford students, 3 exchange students
  • Represented states/countries: CA, OH, WA, Japan
  • 7/7 have been to San Francisco before

3) Filter participants by:

  • familiarity with location used in testing (e.g. San Francisco or New York) to get both expert and novice explorers.
  • geographic origin, to get a more diverse set of testers to see if our application would be deployed globally
Environment:
  • A large room/open indoor space with a wall-sized projection setup (Old Union)
  • Tape marks on the floor for default position and foot gestures to guide user

Study Design
  • Balanced within subjects ABAB experimental design
  • A randomized ordering of different prototypes, to avoid ordering effects
Constants:
  • All users start in a default location (currently San Francisco)
  • All users will use the same walking gesture, with speed/turn granularity
Independent variables:

There are two gestures for turning:

  • A = shoulder turning
  • B = arm turning (like bicycle signals)

“Feature” that they would like to see

  • Augmented reality
  • Moving at “super speed”
  • Flying in a jetpack
Dependent Variable/Outcome Measurement:
  • Time how long it takes to complete the entire test (and subtasks) for each controller
  • Observe whether or not walking and turning speed granularity are utilized
  • Gather user feedback in this survey. We hope users will provide more honest feedback to the “anonymous” computer instead of feeling obligated to be nice during a face-to-face interview with one of us
  • Observe what they do during free time so we can compare what people familar vs. unfamilar with SF choose to do/explore. Do they get bored?
Study Format & Instructions:
  1. (45 sec) Introduce ourselves and our project. Tell them we are making a “natural user interface using the Kinect to navigate a virtual world”. Explain the purpose of the user study to identify “natural” and effective gestures to navigate Google Earth. Explain that there are 2 gesture controllers we are testing, and they can ask us questions or end the study at any time.
  2. (30 sec) Demonstrate the gestures for Gesture Controller 1. Remind user to think aloud.
  3. (1min) Use Gesture Controller 1 to perform the “Generic Test”Generic Test (starting from same location)
    walk 1 block & make a right turn
    walk 4 blocks & make a left turn
    walk 1 block
    [A: inspect tweets; B: nothing (tweets automatically displayed)]
  4. (30 sec) Demonstrate the gestures for Gesture Controller 2. Remind user to think aloud.
  5. (1 min) Use Gesture Controller 2 to perform the Generic Test”
  6. (30 sec) Ask user which controller they preferred, and explain that they can now explore freely. Remind user to think aloud.
  7. (1 min) Allow user to explore freely.
  8. (1 min) Inform the user that testing is complete and ask them to fill out this survey on a computer.

Results

The main results of our User Testing are displayed below:

Demographics:

Number of subjects: 8
Male/Female: 4/4

Turning:

Average time taken (hands): 1:04
Average time taken (shoulders): 1:15
Number who preferred hands: 4
Number who preferred shoulders: 4

Some comments:

  • “Controls were not as responsive as they could be”
  • “It would be nice to control the amount of turning that’s done”
  • “Turning with the shoulders was too slow. I can’t turn into a street while walking, I have to stop, turn 90 degress, and start”
  • “I was too focused on trying to walk straight that I really didn’t pay attention to the sites”

Observations:

  • There was significant latency in the turning gesture, and fine-grained turning was often difficult for subjects. Most had trouble turning into a street and walking down it, often having to stop, turn and re-orientate every few steps.
  • 4 users who preferred using their hands preferred hand turning as it was more obvious and gave them more “control”. They also said it was difficult to keep their eyes on the screen when doing shoulder turns
  • The other 4 users who preferred using their shoulders found hands to be tiring, and found shoulders to be a more natural and intuitive interface
  • Some subjects pointed out that when they used the “hand turns”, they expected a 90 degree turn

Additional Feature:

Super-speed walking:

Our male subjects liked the feature and ability to walk at “super speed”, while female subjects that we tested did not see the need for such a feature. We also found that height matters, as height-challenged users had to stretch their legs very far in order to trigger the gesture, while false positives occured for tall users whose natural step was very large

Flying:

We found “flying” to be a very popular feature among our subjects, who enjoyed the ability to take off from the ground, and travel around the virtual world.

We found an appropriate metaphor to use was that of a jetpack, as users found it to match their impression of walking around the world, taking off vertically, and then navigating the world from altitude using the forward/left-right/altitude controls. Less appropriate was the metaphor of a bird, as most users found the “wings” and tilt/yaw/spin controls difficult to grasp without training.

Users did point out that at altitude, they were not able to “look down” on the city (as the view orientation remained at a straight horizontal plane). They also pointed out that moving forward at altitude seemed very slow, as very little of the view changed.

Augmented reality:

Most users found the augmented reality layer to be interesting, as it showed recent tweets from the area. They found the gesture to be appropriate, and took well to the metaphor of “putting on a pair of glasses”.

General Feedback:

“If this is supposed to be some type of real world representation, the graphics and fidelity of the environment need to be a little better”
“If you use a projector or two you might be able to project the surface on a curved/3D display, to increase realism”

Full results can be found in this Google Document.

Discussion

One of the principal design choices we’ve made for this project is trying to keep the number of gestures to a minimum, since we believe this will lead to an overall better user experience. However, we had many ideas on how different gestures could help users achieve the same task. Thus, our primary goal from user testing was to optimize our navigation gestures. In particular, we developed two different methods for turning, and we wanted to see which gestures felt more natural to users. We hypothesized that the shoulder turning gesture would be much preferred to the arm turning gesture because it seemed to match the way that we turn in real life. Surprisingly, our results were split half and half between users who preferred the subtle shoulder turning and users who preferred the more explicit arm turning gesture. However, based on the explanations for their choices, users suggested that shoulder turning felt more “natural” while arm turning was “easier to control”. Since the speed of turning was set to be faster for the arm turning than for the shoulder turning, we attributed some of the user preference for arm turning to this fact. We also noticed that users would turn more if they wanted to turn more quickly, and they were frustrated when the controllers did not respond accordingly.

Another part of our test was to get feedback on our jetpack idea. We conducted Wizard-of-Oz testing to see if it would be worth implementing. Since we gave users the freedom to design their own flying gestures, we gained some insight as to what gestures would be most intuitive. However, giving them that freedom revealed that there were many issues with flying that we had not considered. The gestures that they designed triggered many false positives that we did not anticipate. The users tended to put their arms down once they reached their desired altitude, but this triggered the user to start falling. Also, we found that the movement speed that was appropriate on the ground was no longer fitting at a higher altitude. Many users wondered why the movement was “so slow”.

Implications

Turning

Based on what we observed from user testing, we decided that we want to continue with shoulder turning, but we need to implement better granularity and control for the shoulder turning to match users’ expectations.

After Wizard-of-Oz tests on a few users, we have decided to implement both the shoulders controller and the hands controller, with both having different turning speeds. The shoulders controller would be a slower speed, and the hands controller would give the equivalent of a 90 degree turn. We found that users adapted to the new controller quickly, using the hands controller to execute major turns, while using the shoulders controller to make more fine grained orientation changes.

Flying

Based on the excitement of the users while they were “flying”, we found reason to go forward with our plan for designing and implementing flying gestures. We Wizard-of-Oz tested a few gestures, and found that using the arms to simulate a “jetpack” was the most appropriate gesture, as it gave the user the impression of turning a jetpack on to increase altitude. We will proceed to user test other gestures to control the jetpack.

We also need to adjust the speed of movement when a user is in jetpack mode to one that gives the impression of movement at altitude.

General Interface

We believe that providing accurate and useful feedback is as important as the intuitiveness and simplicity of the gestures themselves. Thus, we will redesign the GUI to provide better feedback as well as to provide a better, more polished, overall user experience.

Another piece of feedback that we gleaned from the subjects was that the user interface seemed unnecessarily “sanitary”, with a lot of physics that existed in the real world being absent in the virtual. A few highlighted that the entire experience had no sound effects, and that the jetpack had no vibration. We will be exploring more ways in which one can augment the experience.

 

User Test Video