Home | Download/Install | Tutorials

3D Puppetry: A Kinect-based Interface for 3D Animation

Beginner Tutorial: Adding Puppets

The best candidate puppets are rigid, textured with multiple colors, and mostly matte in appearance. To add them to our system, the user must provide a 3D model and a set of template image. We outline an example pipeline here:

Obtain 3D Model

We recommend using either ReconstructMe or AutoDesk's 123D Catch iPad app to capture 3D models of your physical puppet. Next, use MeshLab to clean up the models and prepare them for import into our system:

  1. Cull the model. If you used the software listed above, your raw model likely includes elements from the capture environment. Use MeshLab to select those faces and delete them. Then use the Remove Unreferenced Vertex command. Note that if you used ReconstructMe, your model may not have colors. We'll address that issue later.
  2. Measure the dimensions of the virtual model using the tape-measure tool and compare them to its physical dimensions. Use the Transform:Scale tool to make them match.
  3. Use the Translate:Move, Translate, and Center tool to move the model to the origin.
  4. Reload the model and don't rotate the view. Use the Transform:Translate tool to align the puppet so it is facing towards the top of your screen and you're looking up at its bottom side. This makes sure the poses calculated by our tracker are all consistent with each other.
  5. Make sure all the normals are correct. Use the Recompute Vertex Normal tool followed by the Normalize Vertex Normals tool. Save the model.
  6. Your model likely contains many more points than our system needs for tracking, so we'll make a downsampled version. Use the Poisson Disk Sampling tool and choose a sampling radius of 0.002m.
  7. Activate the layer-management window by selecting View > Show Layer Dialog.
  8. Select the Poisson-disk samples layer and save it as a new mesh. We usually add _sub to the name of the original model.

Optional: If a model needs to be colorized:

  1. Edit the SIFT_scanning launch file in the kinect_tracker/launch folder. Fill the ply_names parameter with your Poisson-sampled model file, the model_names and display_ply_names parameters with the original model file from Step 5 above, and a similar name for the SIFT_save_base parameter.
  2. Execute roslaunch kinect_tracker SIFT_scanning.launch
  3. Slowly bring the physical puppet into the tracking space and wait for the system to start tracking it correctly. You may need to press the 'x' key to restart tracking a few times.
  4. Present several views of the puppet and press the 'k' key to transfer colors from the Kinect's point cloud to the puppet's stored model.
  5. Once you've captured as many colors as possible, used K (shift-k) to spread the colors you've captured to the rest of the model.
  6. Press 'l' to save the colored model. It will appear in the models folder with the name you entered for display_ply_names with "_recolored" appended.

Obtain Template Images

Before capturing template images, make sure your physical setup matches the one described in the Installation section. In particular, make sure the Kinect's field of view is filled with the blank, white poster board.

  1. Edit the SIFT_scanning launch file in the kinect_tracker/launch folder. Fill the ply_names parameter with your Poisson-sampled model file, the model_names and display_ply_names parameters with the original model file from above, and a similar name for the SIFT_save_base parameter.
  2. Execute roslaunch kinect_tracker SIFT_scanning.launch
  3. Slowly bring the physical puppet into the tracking space and wait for the system to start tracking it correctly. You may need to press the 'x' key to restart tracking a few times.
  4. Set the puppet on the performance surface, remove your hands and any other objects from the Kinect's field of view, and press 's' to save a template image. Repeat this process several times for different puppet positions and orientations. See the example SIFT templates provided with kinect_tracker. We typically find that around 30 images are necessary for responsive and robust puppet identification.

Add Puppet to Launch File

Now it's time to add your puppet to the system for performance capture. In all_puppets_launch.launch, or whatever launch file you're using for capture, fill the ply_names parameter with your Poisson-sampled model file, and the model_names and display_ply_names parameters with either the original model file, or the recolored file, if you captured one. Also edit the pose_load_names parameter to include the name you put in the SIFT_save_base field when you captured the template images.