Motivation

Collecting motion data is an important tool in controlling robots. Traditional approaches of motion capture usually use labels for passive markers. They suffers from several problems such as occlusions or cumbersome equipments. In the past few years, methods of markerless, unconstrained posture estimation using only cameras has received much attention from computer vision researchers. One of these methods is volume-based approach. Instead of deriving kinematic models directly from 2D images, this method first builds an intermediate 3D volume feature of the capture subject. Then fit a 3D body model is to the volume data. Here we proposed an approach for model-free, markerless, volume-based motion capture of humans. It is centered on generating underlying nonlinear axes (or a skeleton curve) from a volume of a human subject captured from multiple calibrated cameras.

Model-Based Approach

We set up cameras around the area where the observed person acted. Then extract volume data using voxel carving. The focus of our approach is to estimate the skeletal curves of the volume data, from which the kinematic posture can be estimated. The main assumption of our approach is that volume data is generated from a human body model consists of rigid body parts with linear central axes (such as a cylinder) connected by revolute joints. And the degrees of freedom provided by the revolute joints introduce nonlinearity into the volume data. By eliminating the nonlinearity in the data, the body model can be placed into a pose-invariant posture, such as a “Da Vinci” posture. Thus the skeleton curves can be easily found.

1. Collect volume data from multiple cameras placed around the observed area, using voxel carving.
2. Apply Isomap to the volume data of each time frame, And get the "da Vinci" posture in the domensionally reduced feature space.
3. Place concentric spherical shells on the "da Vinci" posture to estimate skeletal points.
4. Connect adjacent skeletal point to form skeletal curves. Then project those points back to original data space.

Volume data of human body	The "da Vinci" posture after applying Isomap	Spherical shells placed at the "da Vinci" porsure

Skeletal curves derived from da Vinci posture	Skeletal curves

After the skeletal curves and points are estimated. The segmentation ov different body parts are relatively easy.

1. Bound the head voxels with a sphere model.
2. Bound the torso with rigid cylinder model.
3. Segment the four limbs to match two-link articulations models.
4. Derive the joint angles from the segmentation.

Head Segmentation	Torso segmentation

Arm segmentation	Segmentation result	Estimated joint angles

Model-Free Approach

The model-free approach assumes arbitrary kinematic topology with rigid linear links. It derive appropriate kinematic model and capture its motion.

Given the volume data, this approach goes

1. Model and pose estimation step: For each frame in the sequence, estimate a kinematic model and pose specific to the frame volume.
• Classify skeleton curve root and branching nodes as certain joints
• Segment skeleton curve branch to find other joints, using skewness of the voxel data long the curves. Those segmented joint points are recorded on the branch.
2. Model refinement step: Produce a normalized kinematic model for the sequence
• Align the specific kinematic models across all frames. Taking human data as example, match the left leg of the previous frame to the left leg of the next frame.
• Collapse models on each other. Again using human data as example, collapse the points representing knees of the left leg in all of the frames to form a sequence of 1-D data.
  
• Given the clusters of joint points of each limb, use density estimation algorithms to estimate normalized common joint locations across models.
  
  
3. Motion refinement step: Estimate motion of normalized kinematic model with respect to the volume sequence
• Re-apply refined model to each skeleton curve in the sequence.
• For each data frame, for each branch in the curve, mark estimated joints locations on the curve.
• Derive joint angles of each joints, and thus the posture of the curves.
  
  

Examples

Jumping jack:

Waving:

Walking:

Movies

Waving:

• Original Images
• Voxels
• Joint angles result
  

Publications

"Towards Model-free Markerless Motion Capture", Chi-Wei Chu, Odest Chadwicke Jenkins, Maja J Mataric'. [pdf]

"Markerless Kinematic Model and Motion Capture from Volume Sequences", Chi-Wei Chu, Odest Chadwicke Jenkins, Maja J Mataric'. To appear in IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2003 (CVPR 2003) [pdf]

Contact

Chi-Wei Chu

Odest Chadwicke Jenkins

Maja J Mataric´