Home | Download/Install | Tutorials

3D Puppetry: A Kinect-based Interface for 3D Animation

Beginner Tutorial: Creating Animations

We include several data for several example puppets and virtual backgrounds for testing our system. You will need access to the physical versions of the puppets, which are mostly toys, to use those examples. Otherwise, you can add your own rigid puppets to the system. You will also need to capture a color model for your skin.

1. Once you have added puppets and created your skin model, modify the save_performance parameter in the included all_puppets_launch.launch file to give a name for your performance. See the Launch Files PDF for more details. Also be sure to mimic the physical performance setup we describe on the Download and Installation page.
2. Execute roslaunch kinect_tracker all_puppets_launch.launch . The software will take set up the data feed from the Kinect and load the data for all the puppets and scenes. Then it will present you with two windows. One shows the raw color-image feed from the Kinect, while the other show the virtual scene.
3. The default scene is a checkerboard surface. See the Controls PDF for a full list of commands. To explore the system, press 't' a few times to toggle through visualizations of the trackable performance volume. If a puppet enters this volume, our system should detect and start tracking it. As a puppet gets close to the edge of the tracking volume, we render it with reduced opacity to warn the puppeteer that tracking is about to be lost.
4. Now bring a puppet into the Kinect's field of view and observe how it is detected and then rendered on screen. Some smaller puppets may require you to move your hand away before identification can occur. If tracking still does not work, try rotating the puppet into a few different poses. Eventually the system should lock on. If the estimated pose looks completely wrong, use 'x' to reset tracking for all the puppets.
5. Try moving the camera. Left-click and drag to rotate the camera and right-click and drag to pan it. Use the scroll wheel to zoom in and out. See the controls list for how to adjust the lighting. Use the 'b' key to switch between backgrounds.
6. The 'd' key toggles shadow rendering on and off. We recommend turning it off for performance capture because its current implementation is slow. We usually only enable it while exporting movies of our performances.
7. When you are ready to record an animation, use the 'r' key to start and stop recording. Note that if you used the launch file to load a previously recorded animation, you will need to clear it first by pressing '_' (shift - ). This command should be used whenever you want to clear the currently stored animation.

Sometimes stories involve more than two puppets interacting at the same time. Usually it is impossible for one person to perform such stories by themselves. So we provide layered-animation functionality, where an animation can be performed in two separate passes and layered together. Begin by recording one layer of the performance using the 'r' key. Then, if you would like to record a second layer of animation, press 'R' (shift r). This mode will play back the first performance as it captures your new performance. You will need to use different puppets for the two layers of animation. Then when you press 'p' to play the animation, it will play back both layers simultaneously.

If you would like to save your current animation, press 'w' . To export the movie, press 'e' . It will playback your animation and record the frames to a movie named according the save_performance parameter in the launch file. It will also save a move that combines this rendered movie with the raw color video frames from the Kinect. That movie will have 'combo' in its name.