EasyMocap Documentation

Make human motion capture easier.

4,930 stars Python Python 11 files · ~9,497 tokens #motion-capture
RAW Doc

Technical Documentation: zju3dv/EasyMocap

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Doc Quickstart (`doc/quickstart.md`)

Quick Start

First install this project following install

Demo

We provide an example multiview dataset[dropbox][BaiduDisk(vg1z)], which has 800 frames from 23 synchronized and calibrated cameras. After downloading the dataset, you can run the following example scripts.

bash
data=path/to/data
# 0. extract the video to images
python3 scripts/preprocess/extract_video.py ${data} --handface
# 2.1 example for SMPL reconstruction
python3 apps/demo/mv1p.py ${data} --out ${data}/output/smpl --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --vis_smpl
# 2.2 example for SMPL-X reconstruction
python3 apps/demo/mv1p.py ${data} --out ${data}/output/smplx --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male --vis_smpl
# 2.3 example for MANO reconstruction
#     MANO model is required for this part
python3 apps/demo/mv1p.py ${data} --out ${data}/output/manol --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body handl --model manol --gender male --vis_smpl
python3 apps/demo/mv1p.py ${data} --out ${data}/output/manor --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body handr --model manor --gender male --vis_smpl

Demo On Your Dataset

0. Prepare Your Own Dataset

bash
<seq>
├── intri.yml
├── extri.yml
└── videos
    ├── 1.mp4
    ├── 2.mp4
    ├── ...
    ├── 8.mp4
    └── 9.mp4

The input videos are placed in videos/.

Here intri.yml and extri.yml store the camera intrinsici and extrinsic parameters.

See apps/calibration/Readme for instruction of camera calibration.

See apps/calibration/camera_parameters for the format of camera parameters.

1. Run [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose)

bash
data=path/to/data
out=path/to/output
python3 scripts/preprocess/extract_video.py ${data} --openpose <openpose_path> --handface
  • --openpose: specify the openpose path
  • --handface: detect hands and face keypoints

2. Run the code

The input flags:

  • --undis: use to undistort the images
  • --start, --end: control the begin and end number of frames.

The output flags:

  • --vis_det: visualize the detection
  • --vis_repro: visualize the reprojection
  • --sub_vis: use to specify the views to visualize. If not set, the code will use all views
  • --vis_smpl: use to render the SMPL mesh to images.
  • --write_smpl_full: use to write the full poses of the SMPL parameters

3. Output

Please refer to output.md

Doc 02 Output (`doc/02_output.md`)

EasyMocap Doc - Output

En | 中文

Contents

  1. Json Format
  2. Export to .bvh

Json Format

The results are saved in json format.

bash
<output_root>
├── keypoints3d
│   ├── 000000.json
│   └── xxxxxx.json
└── smpl
    ├── 000000.jpg
    ├── 000000.json
    └── 000004.json

The data in keypoints3d/000000.json is a list, each element represents a human body.

bash
{
    'id': <id>, # the person ID
    'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]], # x,y,z is the 3D coordinates, c means the confidence of this joint. If the c=0, it means this joint is invisible.
}

The definition of the joints is as body25.

The data in smpl/000000.json is also a list, each element represents the SMPL parameters which is slightly different from official model.

bash
{
    "id": <id>,
    "Rh": <(1, 3)>,
    "Th": <(1, 3)>,
    "poses": <(1, 72/78/87)>,
    "expression": <(1, 10)>,
    "shapes": <(1, 10)>
}

If you use SMPL+H model, the poses contains 22x3+6+6. We use 6 pca coefficients for each hand. 3(jaw, left eye, right eye)x3 poses of head are added for SMPL-X model.

Attention (for SMPL/SMPL-X users)

This parameter is a little different from original SMPL/SMPL-X parameters.

We set the first 3 dimensions of poses to zero, and add a new parameter Rh to represents the global oritentation, the vertices of SMPL model V = RX(theta, beta) + T.
Please note that the paramter Rh is not equal to global_orient in SMPL-X model. We take this representation because that changing paramters to new coordinate system in origin is difficult(see this link).

In our representation, you can just use R'@(RX + T) + T' to convert the model, and the new global rotaion and translation is simply written as R'@R and R'@T + T'

To compute the joints locations from these parameters, please refer to ./code/vis_render.py. The key steps are:

python
# 0. load SMPL model
from smplmodel import load_model
body_model = load_model(args.gender, model_type=args.model)
# 1. load parameters
infos = dataset.read_smpl(nf*step)
# 2. compute joints
joints = body_model(return_verts=False, return_tensor=False, **info)[0]
# 3. compute vertices
vertices = body_model(return_verts=True, return_tensor=False, **info)[0]

Export to bvh format

To export the SMPL results to bvh file, you need to download the SMPL-maya model from the website of SMPL. Place the .fbx model in ./data/smplx/SMPL_maya, it may be like this:

bash
└── smplx
    ├── smpl
    │   ├── SMPL_FEMALE.pkl
    │   ├── SMPL_MALE.pkl
    │   └── SMPL_NEUTRAL.pkl
    ├── SMPL_maya
    │   ├── basicModel_f_lbs_10_207_0_v1.0.2.fbx
    │   ├── basicModel_m_lbs_10_207_0_v1.0.2.fbx
    │   ├── joints_mat_v1.0.2.pkl
    │   ├── README.txt
    │   ├── release_notes_v1.0.2.txt
    │   └── SMPL_maya_plugin_v1.0.2.py
    └── smplx

The Blender is also needed. The `` is usually ${out}/smpl, which contanis the 000000.json, ... of SMPL parameters.

bash
BLENDER_PATH=<path_to_blender>/blender-2.79a-linux-glibc219-x86_64
${BLENDER_PATH}/blender -b -t 12 -P scripts/postprocess/convert2bvh.py -- <path_to_output_smpl> --o <output_path>

We have not implement the export of SMPL+H, SMPL-X model yet. If you are interested on it, feel free to create a pull request to us.

输出

Json格式

关键点重建的结果会输出到${out}/keypoints3d路径下

bash
<out>
├── keypoints3d
│   ├── 000000.json
│   └── xxxxxx.json
└── skel

每个json里面是一个列表，包含了当前帧的所有人，列表里的每一个元素表示一个人，内容如下：

json
{
    'id': <id>, # 表示人的跟踪的id
    'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]]: # (N, 4)，表示人的关键点坐标，c表示置信度，置信度为0则该关节点不可见
}

关键点的定义使用OpenPose的BODY25格式

导出为bvh格式

这里使用Blender进行导出，测试的Blender版本为2.79。需要先下载SMPL的fbx模型

bash
BLENDER_PATH=<path_to_blender>/blender-2.79a-linux-glibc219-x86_64
${BLENDER_PATH}/blender -b -t 12 -P scripts/postprocess/convert2bvh.py -- <path_to_output_smpl> --o <path_to_bvh>

Doc Dataset (`doc/dataset.md`)

EasyMoCap - Dataset

For convenience, all of the data used by EasyMoCap share the same format.

Input structure

bash
<seq>
├── intri.yml
├── extri.yml
└── videos
    ├── 1.mp4
    ├── 2.mp4
    ├── ...
    ├── 8.mp4
    └── 9.mp4

You can use this commond to extract the videos to images:

bash
python3 scripts/preprocess/extract_video.py ${data} --no2d

After this, the folder will be like:

bash
<seq>
├── intri.yml
├── extri.yml
└── images
    ├── 1
    │   ├── 000000.jpg
    │   ├── 000001.jpg
    │   ├── 000002.jpg
    │   └── ...
    ├── 2
    │   ├── 000000.jpg
    │   ├── 000001.jpg
    │   ├── 000002.jpg
    │   └── ...
    ├── ...
    ├── ...
    ├── 8
    │   ├── 000000.jpg
    │   ├── 000001.jpg
    │   ├── 000002.jpg
    │   └── ...
    └── 9
        ├── 000000.jpg
        ├── 000001.jpg
        ├── 000002.jpg
        └── ...

2D Pose

For each image, we record its 2D pose in a json file. For an image at root/images/1/000000.jpg, the 2D pose willl store at root/annots/1/000000.json. The content of the annotation file is:

bash
{
    "filename": "images/0/000000.jpg",
    "height": <the height of image>,
    "width": <the width of image>,
    "annots:[
        {
            'personID': 0, # ID of person
            'bbox': [l, t, r, b, conf],
            'keypoints': [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
            'area': <the area of bbox>
        },
        {
            'personID': 1, # ID of person
            'bbox': [l, t, r, b, conf],
            'keypoints': [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
            'area': <the area of bbox>
        }
    ]
}

The definition of the keypoints is body25. If you want to use other definitions, you should add it to easymocap/dataset/config.py

If you use hand and face, the annot is defined as:

bash
{
    "personID": i,
    "bbox": [l, t, r, b, conf],
    "keypoints": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
    "bbox_handl2d": [l, t, r, b, conf],
    "bbox_handr2d": [l, t, r, b, conf],
    "bbox_face2d": [l, t, r, b, conf],
    "handl2d": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
    "handr2d": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
    "face2d": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]]
}

3D Pose

bash
[
    {
        'id': <id>, # the person ID
        'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]], # x,y,z is the 3D coordinates, c means the confidence of this joint. If the c=0, it means this joint is invisible.
    },
    {
        'id': <id>, # the person ID
        'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]], # x,y,z is the 3D coordinates, c means the confidence of this joint. If the c=0, it means this joint is invisible.
    }
]

The definition of the keypoints can be found in easymocap/dataset/config.py. We main use the following formats:

  • body25: 25 keypoints of body
  • bodyhand: 25 body + 21 left hand + 21 right hand
  • bodyhandface: 25 body + 21 left hand + 21 right hand + 51 face keypoints

Doc Installation (`doc/installation.md`)

EasyMocap - Installation

0. Download models

0.1 SMPL models

This step is the same as smplx.

To download the SMPL model go to this (male and female models, version 1.0.0, 10 shape PCs) and this (gender neutral model) project website and register to get access to the downloads section.

To download the SMPL+H model go to this project website and register to get access to the downloads section.

To download the SMPL-X model go to this project website and register to get access to the downloads section.

Place them as following:

bash
data
└── smplx
    ├── J_regressor_body25.npy
    ├── J_regressor_body25_smplh.txt
    ├── J_regressor_body25_smplx.txt
    ├── J_regressor_mano_LEFT.txt
    ├── J_regressor_mano_RIGHT.txt
    ├── smpl
    │   ├── SMPL_FEMALE.pkl
    │   ├── SMPL_MALE.pkl
    │   └── SMPL_NEUTRAL.pkl
    ├── smplh
    │   ├── MANO_LEFT.pkl
    │   ├── MANO_RIGHT.pkl
    │   ├── SMPLH_FEMALE.pkl
    │   └── SMPLH_MALE.pkl
    └── smplx
        ├── SMPLX_FEMALE.pkl
        ├── SMPLX_MALE.pkl
        └── SMPLX_NEUTRAL.pkl

0.2 (Optional) SPIN model

This part is used in 1v1p*.py. You can skip this step if you only use the multiple views dataset.

Download pretrained SPIN model here and place it to data/models/spin_checkpoints.pt.

Fetch the extra data here and place the smpl_mean_params.npz to data/models/smpl_mean_params.npz.

0.3 (Optional) 2D model

You can skip this step if you use openpose as your human keypoints detector.

Download yolov4.weights and place it into data/models/yolov4.weights.

Download pretrained HRNet weight and place it into data/models/pose_hrnet_w48_384x288.pth.

bash
data
└── models
    ├── smpl_mean_params.npz
    ├── spin_checkpoint.pt
    ├── pose_hrnet_w48_384x288.pth
    └── yolov4.weights

2. Requirements

Some of python libraries can be found in requirements.txt. You can test different version of PyTorch.

3. Install

bash
python3 setup.py develop --user

Doc Log (`doc/log.md`)

2021.01.24

  1. Support SMPL+H, SMPL-X model.
  2. Upgrade body_model.py.
  3. Update the optimization functions.
  4. Add checking length of limb
  5. Update the example figures.

2021.04.13

  1. Add mirrored-human code.
  2. Add calibration and annotator
  3. Add setup.py

2021.05.27

  1. Remove the code/ folder
  2. Add the support for mano model
  3. Add --write_smpl_full flag

Doc Mv1p V01 (`doc/mv1p_v01.md`)

mv1p.py (v0.1 Legacy Pipeline) -- Multi-View Single-Person Motion Capture

1. 运行命令

bash
# 需要设置 EGL 渲染后端 (无头服务器上 pyrender 依赖此环境变量)
export PYOPENGL_PLATFORM=egl

python3 apps/demo/mv1p.py ${data} \
    --out ${data}/output/smpl \
    --vis_det --vis_repro --vis_smpl \
    --undis \
    --sub_vis 1 2 3 4

注意: --vis_smpl 依赖 pyrender 进行 OpenGL 渲染。在无头服务器 (无 display) 上必须设置 PYOPENGL_PLATFORM=egl，否则会报 pyglet.display.xlib.NoSuchDisplayException: Cannot connect to "None"。测试机器使用 EGL 后端。

命令行参数说明

参数 类型 默认值 说明
path (位置参数) str -- 数据根目录
--out str {path}/output 输出目录
--sub str[] 自动发现 指定使用的相机子目录
--sub_vis str[] [] (全部) 用于可视化的相机子集
--body str body25 关键点格式 (body25/body15/bodyhand/...)
--model str smpl 人体模型类型 (smpl/smplh/smplx/manol/manor)
--gender str neutral 性别 (neutral/male/female)
--undis flag False 是否进行畸变矫正
--vis_det flag False 可视化 2D 检测结果
--vis_repro flag False 可视化三角化后的重投影
--vis_smpl flag False 可视化 SMPL 模型渲染结果
--thres2d float 0.3 2D 关键点置信度阈值
--start int 0 起始帧
--end int 100000 结束帧
--smooth3d int 0 3D 关键点平滑窗口大小 (0=不平滑)
--MAX_REPRO_ERROR int 50 重投影误差阈值 (像素)
--skel flag False 是否强制重新计算骨架
--write_smpl_full flag False 额外写出完整 pose 参数
--write_vertices flag False 额外写出模型顶点
--verbose flag False 打印详细优化日志

2. 输入数据格式

2.1 目录结构

text
${data}/
├── intri.yml              # 相机内参 (OpenCV YAML 格式)
├── extri.yml              # 相机外参 (OpenCV YAML 格式)
├── images/
│   ├── 1/                 # 相机 1 的图片
│   │   ├── 000000.jpg
│   │   ├── 000001.jpg
│   │   └── ...
│   ├── 2/                 # 相机 2 的图片
│   ├── 3/
│   └── 4/
└── annots/
    ├── 1/                 # 相机 1 的 2D 标注
    │   ├── 000000.json
    │   ├── 000001.json
    │   └── ...
    ├── 2/
    ├── 3/
    └── 4/

2.2 相机参数文件

intri.yml -- 内参 (OpenCV YAML 格式):

yaml
%YAML:1.0
---
names:
  - "1"
  - "2"
  - "3"
  - "4"
K_1: !!opencv-matrix         # 3x3 内参矩阵
  rows: 3
  cols: 3
  dt: d
  data: [fx, 0, cx, 0, fy, cy, 0, 0, 1]
dist_1: !!opencv-matrix      # 1x5 畸变系数
  rows: 1
  cols: 5
  dt: d
  data: [k1, k2, p1, p2, k3]
# K_2, dist_2, ... 同理

extri.yml -- 外参 (OpenCV YAML 格式):

yaml
%YAML:1.0
---
names:
  - "1"
  - "2"
  - "3"
  - "4"
R_1: !!opencv-matrix         # 3x1 旋转向量 (Rodrigues)
  rows: 3
  cols: 1
  dt: d
  data: [rx, ry, rz]
Rot_1: !!opencv-matrix       # 3x3 旋转矩阵
  rows: 3
  cols: 3
  dt: d
  data: [...]
T_1: !!opencv-matrix         # 3x1 平移向量
  rows: 3
  cols: 1
  dt: d
  data: [tx, ty, tz]
# R_2, Rot_2, T_2, ... 同理

读取逻辑 (easymocap/mytools/camera_utils.py:read_camera):

  • 使用 cv2.FileStorage 读取 OpenCV YAML
  • 从 Rodrigues 向量 R_{cam} 还原旋转矩阵 Rot_{cam}
  • 计算投影矩阵 P = K @ [R | T]，存入 cameras[cam]['P']

2.3 2D 标注文件 (annots)

每帧每个相机一个 JSON 文件，格式：

json
{
    "filename": "images/1/000000.jpg",
    "height": 1080,
    "width": 1920,
    "annots": [
        {
            "personID": 0,
            "bbox": [x1, y1, x2, y2, confidence],   // 5 个值
            "keypoints": [                            // Nx3, N=25 for body25
                [x, y, confidence],
                ...
            ]
        }
    ]
}

读取逻辑 (easymocap/mytools/file_utils.py:read_annot):

  • personID 被重命名为 id
  • keypoints 转为 np.array，形状 (nJoints, 3)
  • 负置信度被置零
  • 按 id 排序

3. 整体流程

text
┌─────────────────────────────────────────────────────────────────────┐
│  main (__main__)                                                    │
│  ┌─────────────────────────┐   ┌─────────────────────────────────┐ │
│  │ 1. 参数解析 + 数据集创建  │   │ 2. 阶段一: 三角化 (mv1pmf_skel)│ │
│  │    load_parser()         │──>│    遍历每帧:                    │ │
│  │    parse_parser()        │   │    - 读取多视角 2D 关键点        │ │
│  │    MV1PMF(...)           │   │    - SVD 三角化 → 3D 关键点     │ │
│  │                          │   │    - 检查重投影误差              │ │
│  └─────────────────────────┘   │    - 写出 keypoints3d JSON       │ │
│                                └────────────────┬────────────────┘ │
│                                                  │                  │
│                                ┌────────────────▼────────────────┐ │
│                                │ 3. 阶段二: SMPL 拟合            │ │
│                                │    (mv1pmf_smpl)                │ │
│                                │    - 加载 3D 关键点 + 2D 关键点  │ │
│                                │    - 加载 SMPL 模型             │ │
│                                │    - 优化 shape → pose → 2D    │ │
│                                │    - 写出 SMPL 参数 JSON        │ │
│                                │    - 渲染可视化                  │ │
│                                └─────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

3.1 入口 (`apps/demo/mv1p.py:__main__`)

代码位置: apps/demo/mv1p.py:118-141

python
parser = load_parser()          # 定义命令行参数
parser.add_argument('--skel')   # 额外的 --skel 参数
args = parse_parser(parser)     # 解析参数, 自动发现相机子目录

# 创建多视角单人数据集
dataset = MV1PMF(args.path, annot_root=args.annot, cams=args.sub,
    out=args.out, config=CONFIG[args.body], kpts_type=args.body,
    undis=args.undis, no_img=False, verbose=args.verbose)

# 阶段一: 三角化 (如果 keypoints3d 目录不存在或指定 --skel)
if args.skel or not os.path.exists(skel_path):
    mv1pmf_skel(dataset, check_repro=True, args=args)

# 阶段二: SMPL 拟合 (始终执行)
mv1pmf_smpl(dataset, args)

执行流程:

  1. parse_parser() 自动发现 images/ 下的子目录作为相机列表 (如 ['1','2','3','4'])，保存实验配置到 exp.yml
  2. 构造 MV1PMF 数据集对象，读取相机参数、建立图像/标注列表
  3. 先执行三角化得到 3D 骨架，再执行 SMPL 拟合

3.2 数据集初始化 (`MV1PMF` / `MVBase`)

代码位置: easymocap/dataset/mv1pmf.py:14-75, easymocap/dataset/base.py:372-637

MV1PMF 继承自 MVBase，初始化时:

  1. 扫描文件列表: 对每个相机 cam，收集 images/{cam}/ 和 annots/{cam}/ 下的文件列表
  2. 确定帧数: nFrames = min(各相机图片数量)
  3. 读取相机参数 (MVBase.read_camera):
    • 从 intri.yml + extri.yml 读取每个相机的 K, R, T, dist
    • 计算投影矩阵 P = K @ [R | T]，堆叠为 Pall，形状 (nViews, 3, 4)
    • 计算各相机对的基础矩阵 Fall
  4. 创建 FileWriter: 负责后续的结果写入和可视化

__getitem__(index) 获取第 index 帧数据时:

  • 读取所有视角的图片和 2D 标注
  • 如果 undis=True，对图片做 cv2.undistort，对关键点做 Undistort.points
  • select_person 按 pid=0 过滤出单人标注
  • 返回 images (list of ndarray) 和 annots (dict: bbox (nViews,5), keypoints (nViews,25,3))

3.3 阶段一: 三角化 (`mv1pmf_skel`)

代码位置: apps/demo/mv1p.py:44-69

对每一帧执行:

text
for nf in range(start, end):
    1. images, annots = dataset[nf]
       # annots['keypoints']: (nViews, 25, 3)  -- 各视角的 2D 关键点
       # annots['bbox']:      (nViews, 5)      -- 各视角的边界框

    2. check_keypoints(annots['keypoints'], min_conf=0.3)
       # 将置信度 < 0.3 的关键点置零

    3. keypoints3d, kpts_repro = simple_recon_person(annots['keypoints'], dataset.Pall)
       # 三角化: 多视角 2D → 3D

    4. check_repro_error(keypoints3d, kpts_repro, annots['keypoints'], P, MAX_REPRO_ERROR=50)
       # 重投影误差 > 50 像素的关键点置零, 重新三角化

    5. kp3ds.append(keypoints3d)

    6. [可选] vis_det  → 保存到 {out}/detec/{:06d}.jpg
    7. [可选] vis_repro → 保存到 {out}/repro/{:06d}.jpg

最终:

  • 如果 smooth3d > 0，对 3D 关键点序列进行滑动窗口平滑
  • 遍历写出 {out}/keypoints3d/{:06d}.json
  • 如果 vis_repro，用 ffmpeg 合成 {out}/repro.mp4

3.3.1 三角化算法 (`simple_recon_person`)

代码位置: easymocap/mytools/reconstruction.py:93-100

python
def simple_recon_person(keypoints_use, Puse):
    out = batch_triangulate(keypoints_use, Puse)  # SVD 三角化
    kpts_repro = projectN3(out, Puse)              # 重投影到各视角
    return out, kpts_repro

batch_triangulate (reconstruction.py:52-90):

  • 输入: keypoints_ (nViews, nJoints, 3), Pall (nViews, 3, 4)
  • 对每个关节点，构造超定方程组 A x = 0 (DLT 方法):
    • A[2i] = conf * (u * P[2] - P[0])
    • A[2i+1] = conf * (v * P[2] - P[1])
  • SVD 分解取最小奇异值对应的解
  • 仅对 至少 2 个视角有有效检测 的关节点求解
  • 输出: (nJoints, 4) -- [x, y, z, conf_mean]

projectN3 (reconstruction.py:17-29):

  • 将 3D 点投影回每个视角: kp2d = P @ [x,y,z,1]^T，然后做齐次坐标归一化

3.3.2 重投影误差检查 (`check_repro_error`)

代码位置: apps/demo/mv1p.py:33-42

  • 计算每个视角每个关节点的重投影误差 (像素距离)
  • 对误差超过 MAX_REPRO_ERROR (默认 50 像素) 的关键点，将其置信度设为 0
  • 用更新后的 2D 关键点重新三角化

3.4 阶段二: SMPL 拟合 (`mv1pmf_smpl`)

代码位置: apps/demo/mv1p.py:71-116

3.4.1 数据加载

python
# 1. 重新遍历数据集，收集 2D 关键点和 bbox
for nf in range(start, end):
    images, annots = dataset[nf]    # no_img=True, 不读图片
    keypoints2d.append(annots['keypoints'])   # (nViews, 25, 3)
    bboxes.append(annots['bbox'])             # (nViews, 5)

# 2. 从三角化结果加载 3D 关键点
kp3ds = dataset.read_skeleton(start, end)     # (nFrames, 25, 4)

# 3. 堆叠为 numpy 数组
keypoints2d = np.stack(keypoints2d)  # (nFrames, nViews, 25, 3)
bboxes = np.stack(bboxes)            # (nFrames, nViews, 5)

3.4.2 SMPL 模型加载

代码位置: easymocap/smplmodel/body_param.py:32-64

python
body_model = load_model(gender='neutral', model_type='smpl')
  • 从 data/smplx/smpl/ 加载 SMPL 模型参数
  • 加载关节回归器 J_regressor_body25.npy (将 SMPL 的 6890 个顶点回归到 Body25 的 25 个关节点)
  • 模型放到 CUDA 设备上

3.4.3 优化过程 (`smpl_from_keypoints3d2d`)

代码位置: easymocap/pipeline/basic.py:55-78

分三个阶段进行优化:

阶段 A: Shape 优化 (optimizeShape)

代码位置: easymocap/pyfitting/optimize_simple.py:17-82

  • 目标: 用 3D 关键点的骨长约束拟合 shapes (10 维)
  • 优化变量: shapes (1, 10)
  • 损失函数:
    • s3d: 模型骨长与实测骨长的差异 (加权 1.0)
    • reg_shapes: shape 参数的 L2 正则 (加权 5e-3)
  • 优化器: L-BFGS (strong_wolfe line search)
  • 使用 body15 的前 1-14 段肢体 (不含 nose, neck)

阶段 B: 3D Pose 优化 (optimizePose3D)

代码位置: easymocap/pyfitting/optimize_simple.py:305-349

分两步:

  1. 全局 RT 优化 (OPT_R=True, OPT_T=True):

    • 优化变量: Rh (nFrames, 3), Th (nFrames, 3)
    • 损失: 3D 关键点距离 + 平滑 + 正则

  2. Body Pose 优化 (OPT_POSE=True):

    • 优化变量: Rh, Th, poses (nFrames, 72)
    • 损失函数 (各权重来自 load_weight_pose):
      • k3d (1.0): 模型关节与三角化关节的 3D 距离
      • smooth_body (5.0): 相邻帧关节位置平滑
      • smooth_poses (1.0): 相邻帧 pose 参数平滑
      • smooth_Rh (1.0): 相邻帧全局旋转平滑
      • reg_poses (1e-3): pose 参数 L2 正则
      • reg_poses_zero (1e-2): 特定关节 pose 归零正则
    • 优化器: L-BFGS

阶段 C: 2D Pose 优化 (optimizePose2D)

代码位置: easymocap/pyfitting/optimize_simple.py:351-401

  • 优化变量: Rh, Th, poses
  • 额外损失:
    • k2d (1e-4): 模型关节投影到各视角与 2D 检测的距离 (在 bbox 归一化坐标下)
  • 使用多视角 2D 关键点联合约束

所有优化阶段的通用流程 (_optimizeSMPL, optimize_simple.py:244-303):

text
1. 准备参数 (deepcopy, 零化 root pose, 插值空白帧)
2. 前向计算: kpts_est = body_model(poses, shapes, Rh, Th)
3. 计算各项损失 → 加权求和
4. L-BFGS 优化 (FittingMonitor 控制收敛: ftol=1e-4)
5. 后处理 (插值空白帧, tensor → numpy)

3.4.4 结果写出与可视化

python
for nf in range(start, end):
    # 1. 写出 SMPL 参数
    dataset.write_smpl(param, nf)
    #    → {out}/smpl/{:06d}.json

    # 2. [可选] 写出完整 poses (--write_smpl_full)
    #    → {out}/smpl_full/{:06d}.json

    # 3. [可选] 写出顶点 (--write_vertices)
    #    → {out}/vertices/{:06d}.json

    # 4. [可选] SMPL 渲染可视化 (--vis_smpl)
    #    body_model(return_verts=True) → 6890 个顶点
    #    pyrender 渲染到各视角图片上
    #    → {out}/smpl/{:06d}.jpg

    # 5. [可选] SMPL 重投影可视化 (--vis_repro)
    #    body_model(return_verts=False) → 关节点
    #    projectN3 投影到各视角
    #    → {out}/repro_smpl/{:06d}.jpg

4. 输出结果格式

4.1 输出目录结构

text
${out}/
├── exp.yml                # 实验参数快照
├── keypoints3d/           # 三角化的 3D 关键点
│   ├── 000000.json
│   ├── 000001.json
│   └── ...
├── smpl/                  # SMPL 参数
│   ├── 000000.json
│   ├── 000001.json
│   └── ...
├── detec/                 # [--vis_det] 2D 检测可视化
│   ├── 000000.jpg         # 多视角拼接图
│   └── ...
├── repro/                 # [--vis_repro] 三角化重投影可视化
│   ├── 000000.jpg
│   └── ...
├── repro.mp4              # [--vis_repro] 重投影可视化视频
├── smpl/                  # [--vis_smpl] SMPL 渲染可视化 (与参数同目录)
│   ├── 000000.jpg
│   └── ...
├── smpl.mp4               # [--vis_smpl] SMPL 渲染视频
├── repro_smpl/            # [--vis_repro + smpl 阶段] SMPL 关节重投影
│   ├── 000000.jpg
│   └── ...
└── repro_smpl.mp4         # SMPL 关节重投影视频

4.2 keypoints3d JSON 格式

每帧一个文件 ({out}/keypoints3d/{:06d}.json):

json
[
    {
        "id": 0,
        "keypoints3d": [
            [x, y, z, confidence],   // 关节 0 (Nose)
            [x, y, z, confidence],   // 关节 1 (Neck)
            ...                      // 共 25 个关节 (Body25)
        ]
    }
]
  • 形状: (25, 4) -- 每个关节 [x, y, z, conf]
  • conf 为各视角置信度的加权平均值
  • 坐标系: 世界坐标系 (由相机外参定义)

4.3 SMPL 参数 JSON 格式

每帧一个文件 ({out}/smpl/{:06d}.json):

json
[
    {
        "id": 0,
        "Rh": [[rx, ry, rz]],         // (1, 3) 全局旋转 (axis-angle)
        "Th": [[tx, ty, tz]],         // (1, 3) 全局平移
        "poses": [[p0, p1, ..., p71]],  // (1, 72) 身体姿态参数 (24个关节 x 3, axis-angle)
        "shapes": [[s0, s1, ..., s9]]   // (1, 10) 体型参数 (PCA 系数)
    }
]

参数说明:

  • Rh: 全局旋转，axis-angle 表示，3 维向量
  • Th: 全局平移，世界坐标系下的位置
  • poses: SMPL body pose，共 72 维 = 24 个关节 x 3 (axis-angle)。前 3 维是 root pose (全零，因为 root 旋转由 Rh 控制)
  • shapes: SMPL shape 参数 (beta)，10 维 PCA 系数，所有帧共享

4.4 可视化图片

  • detec: 各视角的 2D 检测框 + 关键点，拼接成一张图
  • repro: 三角化 3D 关键点重投影回各视角，拼接成一张图
  • smpl: SMPL 模型通过 pyrender 渲染叠加到各视角图片上 (需要 display 环境)
  • repro_smpl: SMPL 模型关节点投影到各视角的可视化

所有拼接图使用 merge() 函数将多视角图像排列成网格。

5. 运行日志分析

以样例数据 debug/test 运行 3 帧为例:

text
Demo code for multiple views and one person:
    - Input : debug/test => 1, 2, 3, 4        # 4 个相机
    - Output: debug/test/output_doc
    - Body  : smpl=>neutral, body25            # SMPL neutral, Body25 关键点

triangulation: 100%|██████████| 3/3             # 阶段一: 三角化
dump: 100%|██████████| 3/3                      # 写出 keypoints3d
Composing video: .../repro.mp4                  # ffmpeg 合成重投影视频

loading: 100%|██████████| 3/3                   # 阶段二: 加载数据
-> [Optimize global RT  ]:   1.4s               # Shape + RT 优化
-> [Optimize 3D Pose/3 frames]:  17.6s          # 3D Pose 优化 (L-BFGS)
-> [Optimize 2D Pose/3 frames]:   5.3s          # 2D Pose 优化 (L-BFGS)
render: ...                                     # 渲染 SMPL (需要 display)

6. 关键代码文件索引

文件 功能
apps/demo/mv1p.py 主入口: 三角化 + SMPL 拟合
easymocap/mytools/cmd_loader.py 命令行参数定义与解析
easymocap/dataset/mv1pmf.py MV1PMF 多视角单人数据集
easymocap/dataset/base.py MVBase 基类: 相机读取、数据加载、文件写入
easymocap/dataset/config.py Body25 等关键点配置 (关节数、骨架连接)
easymocap/mytools/reconstruction.py 三角化 (batch_triangulate) 和重投影 (projectN3)
easymocap/mytools/camera_utils.py 相机参数读写、畸变矫正
easymocap/mytools/file_utils.py JSON 读写、标注读取、结果写出
easymocap/mytools/writer.py FileWriter: 结果写出和可视化
easymocap/smplmodel/body_param.py SMPL 模型加载、参数工具函数
easymocap/smplmodel/body_model.py SMPLlayer: PyTorch SMPL 前向计算
easymocap/pipeline/basic.py smpl_from_keypoints3d2d: 多阶段优化编排
easymocap/pipeline/weight.py 各模型类型的默认损失权重
easymocap/pipeline/config.py Config: 优化控制参数
easymocap/pyfitting/optimize_simple.py optimizeShape, optimizePose3D, optimizePose2D
easymocap/pyfitting/lossfactory.py 各种损失函数 (3D, 2D, 平滑, 正则)
easymocap/visualize/renderer.py pyrender 渲染器

7. 数据流总结

text
输入:
  images/{cam}/*.jpg           各视角图片
  annots/{cam}/*.json          各视角 2D 关键点 (25 joints, Body25)
  intri.yml + extri.yml        相机参数

         │
         ▼
  ┌──────────────────────┐
  │  Phase 1: 三角化      │
  │  多视角 2D → 3D       │
  │  (SVD/DLT)           │
  └──────────┬───────────┘
             │
             ▼
  keypoints3d/{:06d}.json     3D 关键点 (25, 4)

         │
         ▼
  ┌──────────────────────────────────────────────┐
  │  Phase 2: SMPL 拟合                           │
  │  Step A: optimizeShape (骨长 → shapes)        │
  │  Step B: optimizePose3D (3D kpts → Rh,Th,poses)│
  │  Step C: optimizePose2D (2D kpts → 微调)      │
  └──────────┬───────────────────────────────────┘
             │
             ▼
  smpl/{:06d}.json            SMPL 参数: Rh(1,3), Th(1,3), poses(1,72), shapes(1,10)

输出可视化:
  detec/{:06d}.jpg            2D 检测可视化
  repro/{:06d}.jpg            三角化重投影可视化
  repro.mp4                   重投影视频
  smpl/{:06d}.jpg             SMPL 渲染可视化
  smpl.mp4                    SMPL 渲染视频
  repro_smpl/{:06d}.jpg       SMPL 关节重投影可视化
  repro_smpl.mp4              SMPL 关节重投影视频

Doc Mvmp (`doc/mvmp.md`)

EasyMocap - mvmp

This code aims to solve the problem of reconstructing multiple persons from multiple calibrated cameras. The released code is an easy-to-use version. See Advanced for more details.

0. Preparation

Prepare your calibrated and synchronized system by yourself.

You can download our dataset here.

bash
├── intri.yml
├── extri.yml
├── annots
│   ├── 0
│   ├── 1
│   ├── 2
│   ├── 3
│   ├── 4
│   ├── 5
│   ├── 6
│   └── 7
└── videos
    ├── 0.mp4
    ├── 1.mp4
    ├── 2.mp4
    ├── 3.mp4
    ├── 4.mp4
    ├── 5.mp4
    ├── 6.mp4
    └── 7.mp4

Extract the images from videos:

bash
data=/path/to/data
python3 scripts/preprocess/extract_video.py ${data} --no2d

1. Reconstucting human pose

This step will reconstruct the human pose in each frame.

bash
python3 apps/demo/mvmp.py ${data} --out ${data}/output --annot annots --cfg config/exp/mvmp1f.yml --undis --vis_det --vis_repro

2. Recovering SMPL body model

First we should tract the human pose in each frame. This step will track and interpolate missing frames.

bash
python3 apps/demo/auto_track.py ${data}/output ${data}/output-track --track3d

Then we can fit SMPL model to the tracked keyponts:

bash
python3 apps/demo/smpl_from_keypoints.py ${data} --skel ${data}/output-track/keypoints3d --out ${data}/output-track/smpl --verbose --opts smooth_poses 1e1

To visualize the results, see visualization tutorial

Advanced

For more complicated scenes, our lab has a real-time version of this algorithm, which can perform 3D reconstruction and tracking simultaneously.

If you want to use this part for commercial queries, please contact Xiaowei Zhou.

https://user-images.githubusercontent.com/22812405/123629197-968c0080-d846-11eb-8417-4e6d3a65466d.mp4

Doc Realtime Visualization (`doc/realtime_visualization.md`)

EasyMoCap -> Real-time Visualization

We are the first one to release a real-time visualization tool for both skeletons and SMPL/SMPL+H/SMPL-X/MANO models.

Install

Please install EasyMocap first. This part requires Open3D==0.9.0:

bash
python3 -m pip install open3d==0.9.0

Open the server

Before any visualization, you should run a server:

bash
# quick start:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml
# If you want to specify the host and port:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml host <your_ip_address> port <set_a_port>

This step will open the visualization window:

You can alternate the viewpoints free. Press a to automatic rotate the scene. The configuration file config/vis/o3d_scene.yml defines the scene and other properties. In the default setting, we define the xyz-axis in the origin, the bounding box of the scene and a chessboard in the ground.

Send the data

If you are success to open the server, you can visualize your 3D data anywhere. We provide an example code:

bash
python3 apps/vis/vis_client.py --path <path/to/your/keypoints3d> --host <previous_ip_address> --port <previous_port>

Take the zju-ls-feng results as example, you can show the skeleton in the main window:

Embed this feature to your code

To add this visualization to your other code, you can follow these steps:

bash
# 1. import the base client
from easymocap.socket.base_client import BaseSocketClient
# 2. set the ip address and port
client = BaseSocketClient(host, port)
# 3. send the data
client.send(data)

The format of data is:

python
data = [
    {
        'id': 0,
        'keypoints3d': numpy.ndarray # (nJoints, 4) , (x, y, z, c) for each joint
    },
    {
        'id': 1,
        'keypoints3d': numpy.ndarray # (nJoints, 4)
    }
]

Define your scene

In the configuration file, we main define the body_model and scene. You can replace them for your data.

Examples

To understand our code, we provide lots of results for visualization.

First download the data here and set the path to vis:

bash
vis=/path/to/vis/data

1. Skeletons

Basic skeletons:

bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml write True out ${vis}/output/skel-body25 camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smpl/keypoints3d

Body+Face+Hand:

bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_total.yml write True out ${vis}/output/skel-total camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smplx/keypoints3d

Multiple Person:

bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml write True out ${vis}/output/skel-multi camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/multi/keypoints3d --step 4

2. Mesh

SMPL:

bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_smpl.yml write True out ${vis}/output/mesh-smpl camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smpl/smpl --smpl

SMPLX:

bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_smplx.yml write True out ${vis}/output/mesh-smplx camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smplx/smpl --smpl

MANO:

bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_manol.yml write True out ${vis}/output/mesh-manol camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/manol/smpl --smpl

Advanced

1. Camera Setting

Try to modify these keys to control the location and rotation of the cameras.

yml
camera:
  phi: 0
  theta: -30
  cx: 0.
  cy: 0.
  cz: 6.

2. Scene Setting

We provide some useful mesh in easymocap.visualize.o3dwrapper. If you want to add your own 3D mesh, add it to the key scene.

3. Body Model

At present, we just allow use one type of body model in the scene for fast visualization. So you must set the body model before you run the apps/vis/vis_server.py. If you want to use different models in a scene, you can implement it and pull a request.

Doc Todo (`doc/todo.md`)

TODO

This part is coming soon, please stay tuned.

Doc Tutorial New Task (`doc/tutorial_new_task.md`)

Add new tasks

0. Prepare the data and dataset

1. Add new loss functions

2. Add new optimization

3. Write your own main function

4. Evaluation for the new tasks

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#motion-capture Browse motion-capture catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →