# Technical Documentation: zju3dv/EasyMocap

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/zju3dv/EasyMocap)

### Doc Quickstart (`doc/quickstart.md`)

# Quick Start

First install this project following [install](./installation.md)

## Demo

We provide an example multiview dataset[[dropbox](https://www.dropbox.com/s/24mb7r921b1g9a7/zju-ls-feng.zip?dl=0)][[BaiduDisk](https://pan.baidu.com/s/1lvAopzYGCic3nauoQXjbPw)(vg1z)], which has 800 frames from 23 synchronized and calibrated cameras. After downloading the dataset, you can run the following example scripts.

```bash
data=path/to/data
# 0. extract the video to images
python3 scripts/preprocess/extract_video.py ${data} --handface
# 2.1 example for SMPL reconstruction
python3 apps/demo/mv1p.py ${data} --out ${data}/output/smpl --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --vis_smpl
# 2.2 example for SMPL-X reconstruction
python3 apps/demo/mv1p.py ${data} --out ${data}/output/smplx --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male --vis_smpl
# 2.3 example for MANO reconstruction
#     MANO model is required for this part
python3 apps/demo/mv1p.py ${data} --out ${data}/output/manol --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body handl --model manol --gender male --vis_smpl
python3 apps/demo/mv1p.py ${data} --out ${data}/output/manor --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body handr --model manor --gender male --vis_smpl
```

# Demo On Your Dataset

## 0. Prepare Your Own Dataset

```bash
<seq>
├── intri.yml
├── extri.yml
└── videos
    ├── 1.mp4
    ├── 2.mp4
    ├── ...
    ├── 8.mp4
    └── 9.mp4
```

The input videos are placed in `videos/`.

Here `intri.yml` and `extri.yml` store the camera intrinsici and extrinsic parameters.

See [`apps/calibration/Readme`](../apps/calibration/Readme.md) for instruction of camera calibration.

See [`apps/calibration/camera_parameters`](../apps/calibration/camera_parameters.md) for the format of camera parameters.

### 1. Run [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose)

```bash
data=path/to/data
out=path/to/output
python3 scripts/preprocess/extract_video.py ${data} --openpose <openpose_path> --handface
```

- `--openpose`: specify the openpose path
- `--handface`: detect hands and face keypoints

### 2. Run the code

The input flags:

- `--undis`: use to undistort the images
- `--start, --end`: control the begin and end number of frames.

The output flags:

- `--vis_det`: visualize the detection
- `--vis_repro`: visualize the reprojection
- `--sub_vis`: use to specify the views to visualize. If not set, the code will use all views
- `--vis_smpl`: use to render the SMPL mesh to images.
- `--write_smpl_full`: use to write the full poses of the SMPL parameters

### 3. Output

Please refer to [output.md](../doc/02_output.md)

---

### Doc 02 Output (`doc/02_output.md`)

# EasyMocap Doc - Output
[En](Output) | [中文](#输出)

## Contents
1. [Json Format](#json-format)
2. [Export to .bvh](#export-to-bvh-format)

## Json Format
The results are saved in `json` format.
```bash
<output_root>
├── keypoints3d
│   ├── 000000.json
│   └── xxxxxx.json
└── smpl
    ├── 000000.jpg
    ├── 000000.json
    └── 000004.json
```
The data in `keypoints3d/000000.json` is a list, each element represents a human body.
```bash
{
    'id': <id>, # the person ID
    'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]], # x,y,z is the 3D coordinates, c means the confidence of this joint. If the c=0, it means this joint is invisible.
}
```
The definition of the joints is as [body25](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/02_output.md#pose-output-format-body_25).

The data in `smpl/000000.json` is also a list, each element represents the SMPL parameters which is slightly different from official model.
```bash
{
    "id": <id>,
    "Rh": <(1, 3)>,
    "Th": <(1, 3)>,
    "poses": <(1, 72/78/87)>,
    "expression": <(1, 10)>,
    "shapes": <(1, 10)>
}
```
If you use SMPL+H model, the poses contains `22x3+6+6`. We use `6` pca coefficients for each hand. `3(jaw, left eye, right eye)x3` poses of head are added for SMPL-X model.

### Attention (for SMPL/SMPL-X users)

**This parameter is a little different from original SMPL/SMPL-X parameters.**

We set the first 3 dimensions of `poses` to zero, and add a new parameter `Rh` to represents the global oritentation, the vertices of SMPL model V = RX(theta, beta) + T.
Please note that the paramter `Rh` is not equal to `global_orient` in SMPL-X model. We take this representation because that changing paramters to new coordinate system in origin is difficult(see [this link](https://www.dropbox.com/scl/fi/zkatuv5shs8d4tlwr8ecc/Change-parameters-to-new-coordinate-system.paper?dl=0&rlkey=lotq1sh6wzkmyttisc05h0in0)).

In our representation, you can just use `R'@(RX + T) + T'` to convert the model, and the new global rotaion and translation is simply written as `R'@R` and `R'@T + T'`

To compute the joints locations from these parameters, please refer to `./code/vis_render.py`. The key steps are:
```python
# 0. load SMPL model
from smplmodel import load_model
body_model = load_model(args.gender, model_type=args.model)
# 1. load parameters
infos = dataset.read_smpl(nf*step)
# 2. compute joints
joints = body_model(return_verts=False, return_tensor=False, **info)[0]
# 3. compute vertices
vertices = body_model(return_verts=True, return_tensor=False, **info)[0]
```

## Export to bvh format
To export the SMPL results to bvh file, you need to download the SMPL-maya model from the website of SMPL. Place the `.fbx` model in `./data/smplx/SMPL_maya`, it may be like this:
```bash
└── smplx
    ├── smpl
    │   ├── SMPL_FEMALE.pkl
    │   ├── SMPL_MALE.pkl
    │   └── SMPL_NEUTRAL.pkl
    ├── SMPL_maya
    │   ├── basicModel_f_lbs_10_207_0_v1.0.2.fbx
    │   ├── basicModel_m_lbs_10_207_0_v1.0.2.fbx
    │   ├── joints_mat_v1.0.2.pkl
    │   ├── README.txt
    │   ├── release_notes_v1.0.2.txt
    │   └── SMPL_maya_plugin_v1.0.2.py
    └── smplx
```
The Blender is also needed. The `<path_to_output_smpl>` is usually `${out}/smpl`, which contanis the `000000.json, ...` of SMPL parameters.
```bash
BLENDER_PATH=<path_to_blender>/blender-2.79a-linux-glibc219-x86_64
${BLENDER_PATH}/blender -b -t 12 -P scripts/postprocess/convert2bvh.py -- <path_to_output_smpl> --o <output_path>
```
We have not implement the export of SMPL+H, SMPL-X model yet. If you are interested on it, feel free to create a pull request to us.

-----

# 输出
## Json格式
关键点重建的结果会输出到`${out}/keypoints3d`路径下
```bash
<out>
├── keypoints3d
│   ├── 000000.json
│   └── xxxxxx.json
└── skel
```
每个json里面是一个列表，包含了当前帧的所有人，列表里的每一个元素表示一个人，内容如下：
```json
{
    'id': <id>, # 表示人的跟踪的id
    'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]]: # (N, 4)，表示人的关键点坐标，c表示置信度，置信度为0则该关节点不可见
}
```
关键点的定义使用OpenPose的[BODY25格式](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/02_output.md#pose-output-format-body_25)

## 导出为bvh格式
这里使用Blender进行导出，测试的Blender版本为2.79。需要先下载SMPL的fbx模型
```bash
BLENDER_PATH=<path_to_blender>/blender-2.79a-linux-glibc219-x86_64
${BLENDER_PATH}/blender -b -t 12 -P scripts/postprocess/convert2bvh.py -- <path_to_output_smpl> --o <path_to_bvh>
```

---

### Doc Dataset (`doc/dataset.md`)

# EasyMoCap - Dataset

For convenience, all of the data used by EasyMoCap share the same format.

## Input structure

```bash
<seq>
├── intri.yml
├── extri.yml
└── videos
    ├── 1.mp4
    ├── 2.mp4
    ├── ...
    ├── 8.mp4
    └── 9.mp4
```

You can use this commond to extract the videos to images:
```bash
python3 scripts/preprocess/extract_video.py ${data} --no2d
```

After this, the folder will be like:
```bash
<seq>
├── intri.yml
├── extri.yml
└── images
    ├── 1
    │   ├── 000000.jpg
    │   ├── 000001.jpg
    │   ├── 000002.jpg
    │   └── ...
    ├── 2
    │   ├── 000000.jpg
    │   ├── 000001.jpg
    │   ├── 000002.jpg
    │   └── ...
    ├── ...
    ├── ...
    ├── 8
    │   ├── 000000.jpg
    │   ├── 000001.jpg
    │   ├── 000002.jpg
    │   └── ...
    └── 9
        ├── 000000.jpg
        ├── 000001.jpg
        ├── 000002.jpg
        └── ...
```

## 2D Pose

For each image, we record its 2D pose in a `json` file. For an image at `root/images/1/000000.jpg`, the 2D pose willl store at `root/annots/1/000000.json`. The content of the annotation file is:

```bash
{
    "filename": "images/0/000000.jpg",
    "height": <the height of image>,
    "width": <the width of image>,
    "annots:[
        {
            'personID': 0, # ID of person
            'bbox': [l, t, r, b, conf],
            'keypoints': [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
            'area': <the area of bbox>
        },
        {
            'personID': 1, # ID of person
            'bbox': [l, t, r, b, conf],
            'keypoints': [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
            'area': <the area of bbox>
        }
    ]
}
```

The definition of the `keypoints` is `body25`. If you want to use other definitions, you should add it to `easymocap/dataset/config.py`

If you use hand and face, the annot is defined as:
```bash
{
    "personID": i,
    "bbox": [l, t, r, b, conf],
    "keypoints": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
    "bbox_handl2d": [l, t, r, b, conf],
    "bbox_handr2d": [l, t, r, b, conf],
    "bbox_face2d": [l, t, r, b, conf],
    "handl2d": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
    "handr2d": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]],
    "face2d": [[x0, y0, c0], [x1, y1, c1], ..., [xn, yn, cn]]
}
```

## 3D Pose

```bash
[
    {
        'id': <id>, # the person ID
        'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]], # x,y,z is the 3D coordinates, c means the confidence of this joint. If the c=0, it means this joint is invisible.
    },
    {
        'id': <id>, # the person ID
        'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]], # x,y,z is the 3D coordinates, c means the confidence of this joint. If the c=0, it means this joint is invisible.
    }
]
```

The definition of the keypoints can be found in `easymocap/dataset/config.py`. We main use the following formats:
- body25: 25 keypoints of body
- bodyhand: 25 body + 21 left hand + 21 right hand
- bodyhandface: 25 body + 21 left hand + 21 right hand + 51 face keypoints

---

### Doc Installation (`doc/installation.md`)

# EasyMocap - Installation

## 0. Download models

## 0.1 SMPL models

This step is the same as [smplx](https://github.com/vchoutas/smplx#model-loading).

To download the *SMPL* model go to [this](http://smpl.is.tue.mpg.de) (male and female models, version 1.0.0, 10 shape PCs) and [this](http://smplify.is.tue.mpg.de) (gender neutral model) project website and register to get access to the downloads section. 

To download the *SMPL+H* model go to [this project website](http://mano.is.tue.mpg.de) and register to get access to the downloads section. 

To download the *SMPL-X* model go to [this project website](https://smpl-x.is.tue.mpg.de) and register to get access to the downloads section. 

**Place them as following:**

```bash
data
└── smplx
    ├── J_regressor_body25.npy
    ├── J_regressor_body25_smplh.txt
    ├── J_regressor_body25_smplx.txt
    ├── J_regressor_mano_LEFT.txt
    ├── J_regressor_mano_RIGHT.txt
    ├── smpl
    │   ├── SMPL_FEMALE.pkl
    │   ├── SMPL_MALE.pkl
    │   └── SMPL_NEUTRAL.pkl
    ├── smplh
    │   ├── MANO_LEFT.pkl
    │   ├── MANO_RIGHT.pkl
    │   ├── SMPLH_FEMALE.pkl
    │   └── SMPLH_MALE.pkl
    └── smplx
        ├── SMPLX_FEMALE.pkl
        ├── SMPLX_MALE.pkl
        └── SMPLX_NEUTRAL.pkl
```

## 0.2 (Optional) SPIN model
This part is used in `1v1p*.py`. You can skip this step if you only use the multiple views dataset.

Download pretrained SPIN model [here](http://visiondata.cis.upenn.edu/spin/model_checkpoint.pt) and place it to `data/models/spin_checkpoints.pt`.

Fetch the extra data [here](http://visiondata.cis.upenn.edu/spin/data.tar.gz) and place the `smpl_mean_params.npz` to `data/models/smpl_mean_params.npz`.

## 0.3 (Optional) 2D model

You can skip this step if you use openpose as your human keypoints detector.

Download [yolov4.weights]() and place it into `data/models/yolov4.weights`.

Download pretrained HRNet [weight]() and place it into `data/models/pose_hrnet_w48_384x288.pth`.

```bash
data
└── models
    ├── smpl_mean_params.npz
    ├── spin_checkpoint.pt
    ├── pose_hrnet_w48_384x288.pth
    └── yolov4.weights 
```

## 2. Requirements

- python>=3.6
- torch==1.4.0
- torchvision==0.5.0
- opencv-python
- [pyrender](https://pyrender.readthedocs.io/en/latest/install/index.html#python-installation): for visualization, or [pyrender for server without a screen](https://pyrender.readthedocs.io/en/latest/install/index.html#getting-pyrender-working-with-osmesa).
- chumpy: for loading SMPL model
- OpenPose[4]: for 2D pose

Some of python libraries can be found in `requirements.txt`. You can test different version of PyTorch.

## 3. Install

```bash
python3 setup.py develop --user
```

---

### Doc Log (`doc/log.md`)

## 2021.01.24
1. Support SMPL+H, SMPL-X model.
2. Upgrade `body_model.py`.
3. Update the optimization functions.
4. Add checking length of limb
5. Update the example figures.

## 2021.04.13
1. Add mirrored-human code.
2. Add `calibration` and `annotator`
3. Add `setup.py`

## 2021.05.27
1. Remove the `code/` folder
2. Add the support for mano model
3. Add `--write_smpl_full` flag

---

### Doc Mv1p V01 (`doc/mv1p_v01.md`)

# mv1p.py (v0.1 Legacy Pipeline) -- Multi-View Single-Person Motion Capture

## 1. 运行命令

```bash
# 需要设置 EGL 渲染后端 (无头服务器上 pyrender 依赖此环境变量)
export PYOPENGL_PLATFORM=egl

python3 apps/demo/mv1p.py ${data} \
    --out ${data}/output/smpl \
    --vis_det --vis_repro --vis_smpl \
    --undis \
    --sub_vis 1 2 3 4
```

> **注意**: `--vis_smpl` 依赖 pyrender 进行 OpenGL 渲染。在无头服务器 (无 display) 上必须设置 `PYOPENGL_PLATFORM=egl`，否则会报 `pyglet.display.xlib.NoSuchDisplayException: Cannot connect to "None"`。测试机器使用 EGL 后端。

### 命令行参数说明

| 参数 | 类型 | 默认值 | 说明 |
|------|------|--------|------|
| `path` (位置参数) | str | -- | 数据根目录 |
| `--out` | str | `{path}/output` | 输出目录 |
| `--sub` | str[] | 自动发现 | 指定使用的相机子目录 |
| `--sub_vis` | str[] | `[]` (全部) | 用于可视化的相机子集 |
| `--body` | str | `body25` | 关键点格式 (`body25`/`body15`/`bodyhand`/...) |
| `--model` | str | `smpl` | 人体模型类型 (`smpl`/`smplh`/`smplx`/`manol`/`manor`) |
| `--gender` | str | `neutral` | 性别 (`neutral`/`male`/`female`) |
| `--undis` | flag | False | 是否进行畸变矫正 |
| `--vis_det` | flag | False | 可视化 2D 检测结果 |
| `--vis_repro` | flag | False | 可视化三角化后的重投影 |
| `--vis_smpl` | flag | False | 可视化 SMPL 模型渲染结果 |
| `--thres2d` | float | 0.3 | 2D 关键点置信度阈值 |
| `--start` | int | 0 | 起始帧 |
| `--end` | int | 100000 | 结束帧 |
| `--smooth3d` | int | 0 | 3D 关键点平滑窗口大小 (0=不平滑) |
| `--MAX_REPRO_ERROR` | int | 50 | 重投影误差阈值 (像素) |
| `--skel` | flag | False | 是否强制重新计算骨架 |
| `--write_smpl_full` | flag | False | 额外写出完整 pose 参数 |
| `--write_vertices` | flag | False | 额外写出模型顶点 |
| `--verbose` | flag | False | 打印详细优化日志 |

---

## 2. 输入数据格式

### 2.1 目录结构

```
${data}/
├── intri.yml              # 相机内参 (OpenCV YAML 格式)
├── extri.yml              # 相机外参 (OpenCV YAML 格式)
├── images/
│   ├── 1/                 # 相机 1 的图片
│   │   ├── 000000.jpg
│   │   ├── 000001.jpg
│   │   └── ...
│   ├── 2/                 # 相机 2 的图片
│   ├── 3/
│   └── 4/
└── annots/
    ├── 1/                 # 相机 1 的 2D 标注
    │   ├── 000000.json
    │   ├── 000001.json
    │   └── ...
    ├── 2/
    ├── 3/
    └── 4/
```

### 2.2 相机参数文件

**intri.yml** -- 内参 (OpenCV YAML 格式):
```yaml
%YAML:1.0
---
names:
  - "1"
  - "2"
  - "3"
  - "4"
K_1: !!opencv-matrix         # 3x3 内参矩阵
  rows: 3
  cols: 3
  dt: d
  data: [fx, 0, cx, 0, fy, cy, 0, 0, 1]
dist_1: !!opencv-matrix      # 1x5 畸变系数
  rows: 1
  cols: 5
  dt: d
  data: [k1, k2, p1, p2, k3]
# K_2, dist_2, ... 同理
```

**extri.yml** -- 外参 (OpenCV YAML 格式):
```yaml
%YAML:1.0
---
names:
  - "1"
  - "2"
  - "3"
  - "4"
R_1: !!opencv-matrix         # 3x1 旋转向量 (Rodrigues)
  rows: 3
  cols: 1
  dt: d
  data: [rx, ry, rz]
Rot_1: !!opencv-matrix       # 3x3 旋转矩阵
  rows: 3
  cols: 3
  dt: d
  data: [...]
T_1: !!opencv-matrix         # 3x1 平移向量
  rows: 3
  cols: 1
  dt: d
  data: [tx, ty, tz]
# R_2, Rot_2, T_2, ... 同理
```

读取逻辑 (`easymocap/mytools/camera_utils.py:read_camera`):
- 使用 `cv2.FileStorage` 读取 OpenCV YAML
- 从 Rodrigues 向量 `R_{cam}` 还原旋转矩阵 `Rot_{cam}`
- 计算投影矩阵 `P = K @ [R | T]`，存入 `cameras[cam]['P']`

### 2.3 2D 标注文件 (annots)

每帧每个相机一个 JSON 文件，格式：

```json
{
    "filename": "images/1/000000.jpg",
    "height": 1080,
    "width": 1920,
    "annots": [
        {
            "personID": 0,
            "bbox": [x1, y1, x2, y2, confidence],   // 5 个值
            "keypoints": [                            // Nx3, N=25 for body25
                [x, y, confidence],
                ...
            ]
        }
    ]
}
```

读取逻辑 (`easymocap/mytools/file_utils.py:read_annot`):
- `personID` 被重命名为 `id`
- `keypoints` 转为 `np.array`，形状 `(nJoints, 3)`
- 负置信度被置零
- 按 `id` 排序

---

## 3. 整体流程

```
┌─────────────────────────────────────────────────────────────────────┐
│  main (__main__)                                                    │
│  ┌─────────────────────────┐   ┌─────────────────────────────────┐ │
│  │ 1. 参数解析 + 数据集创建  │   │ 2. 阶段一: 三角化 (mv1pmf_skel)│ │
│  │    load_parser()         │──>│    遍历每帧:                    │ │
│  │    parse_parser()        │   │    - 读取多视角 2D 关键点        │ │
│  │    MV1PMF(...)           │   │    - SVD 三角化 → 3D 关键点     │ │
│  │                          │   │    - 检查重投影误差              │ │
│  └─────────────────────────┘   │    - 写出 keypoints3d JSON       │ │
│                                └────────────────┬────────────────┘ │
│                                                  │                  │
│                                ┌────────────────▼────────────────┐ │
│                                │ 3. 阶段二: SMPL 拟合            │ │
│                                │    (mv1pmf_smpl)                │ │
│                                │    - 加载 3D 关键点 + 2D 关键点  │ │
│                                │    - 加载 SMPL 模型             │ │
│                                │    - 优化 shape → pose → 2D    │ │
│                                │    - 写出 SMPL 参数 JSON        │ │
│                                │    - 渲染可视化                  │ │
│                                └─────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```

### 3.1 入口 (`apps/demo/mv1p.py:__main__`)

**代码位置**: `apps/demo/mv1p.py:118-141`

```python
parser = load_parser()          # 定义命令行参数
parser.add_argument('--skel')   # 额外的 --skel 参数
args = parse_parser(parser)     # 解析参数, 自动发现相机子目录

# 创建多视角单人数据集
dataset = MV1PMF(args.path, annot_root=args.annot, cams=args.sub,
    out=args.out, config=CONFIG[args.body], kpts_type=args.body,
    undis=args.undis, no_img=False, verbose=args.verbose)

# 阶段一: 三角化 (如果 keypoints3d 目录不存在或指定 --skel)
if args.skel or not os.path.exists(skel_path):
    mv1pmf_skel(dataset, check_repro=True, args=args)

# 阶段二: SMPL 拟合 (始终执行)
mv1pmf_smpl(dataset, args)
```

执行流程:
1. `parse_parser()` 自动发现 `images/` 下的子目录作为相机列表 (如 `['1','2','3','4']`)，保存实验配置到 `exp.yml`
2. 构造 `MV1PMF` 数据集对象，读取相机参数、建立图像/标注列表
3. 先执行三角化得到 3D 骨架，再执行 SMPL 拟合

---

### 3.2 数据集初始化 (`MV1PMF` / `MVBase`)

**代码位置**: `easymocap/dataset/mv1pmf.py:14-75`, `easymocap/dataset/base.py:372-637`

`MV1PMF` 继承自 `MVBase`，初始化时:

1. **扫描文件列表**: 对每个相机 `cam`，收集 `images/{cam}/` 和 `annots/{cam}/` 下的文件列表
2. **确定帧数**: `nFrames = min(各相机图片数量)`
3. **读取相机参数** (`MVBase.read_camera`):
   - 从 `intri.yml` + `extri.yml` 读取每个相机的 K, R, T, dist
   - 计算投影矩阵 `P = K @ [R | T]`，堆叠为 `Pall`，形状 `(nViews, 3, 4)`
   - 计算各相机对的基础矩阵 `Fall`
4. **创建 FileWriter**: 负责后续的结果写入和可视化

`__getitem__(index)` 获取第 `index` 帧数据时:
- 读取所有视角的图片和 2D 标注
- 如果 `undis=True`，对图片做 `cv2.undistort`，对关键点做 `Undistort.points`
- `select_person` 按 `pid=0` 过滤出单人标注
- 返回 `images` (list of ndarray) 和 `annots` (dict: `bbox` (nViews,5), `keypoints` (nViews,25,3))

---

### 3.3 阶段一: 三角化 (`mv1pmf_skel`)

**代码位置**: `apps/demo/mv1p.py:44-69`

对每一帧执行:

```
for nf in range(start, end):
    1. images, annots = dataset[nf]
       # annots['keypoints']: (nViews, 25, 3)  -- 各视角的 2D 关键点
       # annots['bbox']:      (nViews, 5)      -- 各视角的边界框

    2. check_keypoints(annots['keypoints'], min_conf=0.3)
       # 将置信度 < 0.3 的关键点置零

    3. keypoints3d, kpts_repro = simple_recon_person(annots['keypoints'], dataset.Pall)
       # 三角化: 多视角 2D → 3D

    4. check_repro_error(keypoints3d, kpts_repro, annots['keypoints'], P, MAX_REPRO_ERROR=50)
       # 重投影误差 > 50 像素的关键点置零, 重新三角化

    5. kp3ds.append(keypoints3d)

    6. [可选] vis_det  → 保存到 {out}/detec/{:06d}.jpg
    7. [可选] vis_repro → 保存到 {out}/repro/{:06d}.jpg
```

最终:
- 如果 `smooth3d > 0`，对 3D 关键点序列进行滑动窗口平滑
- 遍历写出 `{out}/keypoints3d/{:06d}.json`
- 如果 `vis_repro`，用 ffmpeg 合成 `{out}/repro.mp4`

#### 3.3.1 三角化算法 (`simple_recon_person`)

**代码位置**: `easymocap/mytools/reconstruction.py:93-100`

```python
def simple_recon_person(keypoints_use, Puse):
    out = batch_triangulate(keypoints_use, Puse)  # SVD 三角化
    kpts_repro = projectN3(out, Puse)              # 重投影到各视角
    return out, kpts_repro
```

**`batch_triangulate`** (`reconstruction.py:52-90`):
- 输入: `keypoints_` `(nViews, nJoints, 3)`, `Pall` `(nViews, 3, 4)`
- 对每个关节点，构造超定方程组 `A x = 0` (DLT 方法):
  - `A[2i]   = conf * (u * P[2] - P[0])`
  - `A[2i+1] = conf * (v * P[2] - P[1])`
- SVD 分解取最小奇异值对应的解
- 仅对 **至少 2 个视角有有效检测** 的关节点求解
- 输出: `(nJoints, 4)` -- `[x, y, z, conf_mean]`

**`projectN3`** (`reconstruction.py:17-29`):
- 将 3D 点投影回每个视角: `kp2d = P @ [x,y,z,1]^T`，然后做齐次坐标归一化

#### 3.3.2 重投影误差检查 (`check_repro_error`)

**代码位置**: `apps/demo/mv1p.py:33-42`

- 计算每个视角每个关节点的重投影误差 (像素距离)
- 对误差超过 `MAX_REPRO_ERROR` (默认 50 像素) 的关键点，将其置信度设为 0
- 用更新后的 2D 关键点重新三角化

---

### 3.4 阶段二: SMPL 拟合 (`mv1pmf_smpl`)

**代码位置**: `apps/demo/mv1p.py:71-116`

#### 3.4.1 数据加载

```python
# 1. 重新遍历数据集，收集 2D 关键点和 bbox
for nf in range(start, end):
    images, annots = dataset[nf]    # no_img=True, 不读图片
    keypoints2d.append(annots['keypoints'])   # (nViews, 25, 3)
    bboxes.append(annots['bbox'])             # (nViews, 5)

# 2. 从三角化结果加载 3D 关键点
kp3ds = dataset.read_skeleton(start, end)     # (nFrames, 25, 4)

# 3. 堆叠为 numpy 数组
keypoints2d = np.stack(keypoints2d)  # (nFrames, nViews, 25, 3)
bboxes = np.stack(bboxes)            # (nFrames, nViews, 5)
```

#### 3.4.2 SMPL 模型加载

**代码位置**: `easymocap/smplmodel/body_param.py:32-64`

```python
body_model = load_model(gender='neutral', model_type='smpl')
```

- 从 `data/smplx/smpl/` 加载 SMPL 模型参数
- 加载关节回归器 `J_regressor_body25.npy` (将 SMPL 的 6890 个顶点回归到 Body25 的 25 个关节点)
- 模型放到 CUDA 设备上

#### 3.4.3 优化过程 (`smpl_from_keypoints3d2d`)

**代码位置**: `easymocap/pipeline/basic.py:55-78`

分三个阶段进行优化:

**阶段 A: Shape 优化 (`optimizeShape`)**

**代码位置**: `easymocap/pyfitting/optimize_simple.py:17-82`

- 目标: 用 3D 关键点的骨长约束拟合 `shapes` (10 维)
- 优化变量: `shapes` (1, 10)
- 损失函数:
  - `s3d`: 模型骨长与实测骨长的差异 (加权 1.0)
  - `reg_shapes`: shape 参数的 L2 正则 (加权 5e-3)
- 优化器: L-BFGS (strong_wolfe line search)
- 使用 `body15` 的前 1-14 段肢体 (不含 nose, neck)

**阶段 B: 3D Pose 优化 (`optimizePose3D`)**

**代码位置**: `easymocap/pyfitting/optimize_simple.py:305-349`

分两步:

1. **全局 RT 优化** (`OPT_R=True, OPT_T=True`):
   - 优化变量: `Rh` (nFrames, 3), `Th` (nFrames, 3)
   - 损失: 3D 关键点距离 + 平滑 + 正则

2. **Body Pose 优化** (`OPT_POSE=True`):
   - 优化变量: `Rh`, `Th`, `poses` (nFrames, 72)
   - 损失函数 (各权重来自 `load_weight_pose`):
     - `k3d` (1.0): 模型关节与三角化关节的 3D 距离
     - `smooth_body` (5.0): 相邻帧关节位置平滑
     - `smooth_poses` (1.0): 相邻帧 pose 参数平滑
     - `smooth_Rh` (1.0): 相邻帧全局旋转平滑
     - `reg_poses` (1e-3): pose 参数 L2 正则
     - `reg_poses_zero` (1e-2): 特定关节 pose 归零正则
   - 优化器: L-BFGS

**阶段 C: 2D Pose 优化 (`optimizePose2D`)**

**代码位置**: `easymocap/pyfitting/optimize_simple.py:351-401`

- 优化变量: `Rh`, `Th`, `poses`
- 额外损失:
  - `k2d` (1e-4): 模型关节投影到各视角与 2D 检测的距离 (在 bbox 归一化坐标下)
- 使用多视角 2D 关键点联合约束

**所有优化阶段的通用流程** (`_optimizeSMPL`, `optimize_simple.py:244-303`):
```
1. 准备参数 (deepcopy, 零化 root pose, 插值空白帧)
2. 前向计算: kpts_est = body_model(poses, shapes, Rh, Th)
3. 计算各项损失 → 加权求和
4. L-BFGS 优化 (FittingMonitor 控制收敛: ftol=1e-4)
5. 后处理 (插值空白帧, tensor → numpy)
```

#### 3.4.4 结果写出与可视化

```python
for nf in range(start, end):
    # 1. 写出 SMPL 参数
    dataset.write_smpl(param, nf)
    #    → {out}/smpl/{:06d}.json

    # 2. [可选] 写出完整 poses (--write_smpl_full)
    #    → {out}/smpl_full/{:06d}.json

    # 3. [可选] 写出顶点 (--write_vertices)
    #    → {out}/vertices/{:06d}.json

    # 4. [可选] SMPL 渲染可视化 (--vis_smpl)
    #    body_model(return_verts=True) → 6890 个顶点
    #    pyrender 渲染到各视角图片上
    #    → {out}/smpl/{:06d}.jpg

    # 5. [可选] SMPL 重投影可视化 (--vis_repro)
    #    body_model(return_verts=False) → 关节点
    #    projectN3 投影到各视角
    #    → {out}/repro_smpl/{:06d}.jpg
```

---

## 4. 输出结果格式

### 4.1 输出目录结构

```
${out}/
├── exp.yml                # 实验参数快照
├── keypoints3d/           # 三角化的 3D 关键点
│   ├── 000000.json
│   ├── 000001.json
│   └── ...
├── smpl/                  # SMPL 参数
│   ├── 000000.json
│   ├── 000001.json
│   └── ...
├── detec/                 # [--vis_det] 2D 检测可视化
│   ├── 000000.jpg         # 多视角拼接图
│   └── ...
├── repro/                 # [--vis_repro] 三角化重投影可视化
│   ├── 000000.jpg
│   └── ...
├── repro.mp4              # [--vis_repro] 重投影可视化视频
├── smpl/                  # [--vis_smpl] SMPL 渲染可视化 (与参数同目录)
│   ├── 000000.jpg
│   └── ...
├── smpl.mp4               # [--vis_smpl] SMPL 渲染视频
├── repro_smpl/            # [--vis_repro + smpl 阶段] SMPL 关节重投影
│   ├── 000000.jpg
│   └── ...
└── repro_smpl.mp4         # SMPL 关节重投影视频
```

### 4.2 keypoints3d JSON 格式

每帧一个文件 (`{out}/keypoints3d/{:06d}.json`):

```json
[
    {
        "id": 0,
        "keypoints3d": [
            [x, y, z, confidence],   // 关节 0 (Nose)
            [x, y, z, confidence],   // 关节 1 (Neck)
            ...                      // 共 25 个关节 (Body25)
        ]
    }
]
```

- 形状: `(25, 4)` -- 每个关节 `[x, y, z, conf]`
- `conf` 为各视角置信度的加权平均值
- 坐标系: 世界坐标系 (由相机外参定义)

### 4.3 SMPL 参数 JSON 格式

每帧一个文件 (`{out}/smpl/{:06d}.json`):

```json
[
    {
        "id": 0,
        "Rh": [[rx, ry, rz]],         // (1, 3) 全局旋转 (axis-angle)
        "Th": [[tx, ty, tz]],         // (1, 3) 全局平移
        "poses": [[p0, p1, ..., p71]],  // (1, 72) 身体姿态参数 (24个关节 x 3, axis-angle)
        "shapes": [[s0, s1, ..., s9]]   // (1, 10) 体型参数 (PCA 系数)
    }
]
```

参数说明:
- `Rh`: 全局旋转，axis-angle 表示，3 维向量
- `Th`: 全局平移，世界坐标系下的位置
- `poses`: SMPL body pose，共 72 维 = 24 个关节 x 3 (axis-angle)。前 3 维是 root pose (全零，因为 root 旋转由 `Rh` 控制)
- `shapes`: SMPL shape 参数 (beta)，10 维 PCA 系数，所有帧共享

### 4.4 可视化图片

- **detec**: 各视角的 2D 检测框 + 关键点，拼接成一张图
- **repro**: 三角化 3D 关键点重投影回各视角，拼接成一张图
- **smpl**: SMPL 模型通过 pyrender 渲染叠加到各视角图片上 (需要 display 环境)
- **repro_smpl**: SMPL 模型关节点投影到各视角的可视化

所有拼接图使用 `merge()` 函数将多视角图像排列成网格。

---

## 5. 运行日志分析

以样例数据 `debug/test` 运行 3 帧为例:

```
  Demo code for multiple views and one person:
    - Input : debug/test => 1, 2, 3, 4        # 4 个相机
    - Output: debug/test/output_doc
    - Body  : smpl=>neutral, body25            # SMPL neutral, Body25 关键点

triangulation: 100%|██████████| 3/3             # 阶段一: 三角化
dump: 100%|██████████| 3/3                      # 写出 keypoints3d
Composing video: .../repro.mp4                  # ffmpeg 合成重投影视频

loading: 100%|██████████| 3/3                   # 阶段二: 加载数据
-> [Optimize global RT  ]:   1.4s               # Shape + RT 优化
-> [Optimize 3D Pose/3 frames]:  17.6s          # 3D Pose 优化 (L-BFGS)
-> [Optimize 2D Pose/3 frames]:   5.3s          # 2D Pose 优化 (L-BFGS)
render: ...                                     # 渲染 SMPL (需要 display)
```

---

## 6. 关键代码文件索引

| 文件 | 功能 |
|------|------|
| `apps/demo/mv1p.py` | 主入口: 三角化 + SMPL 拟合 |
| `easymocap/mytools/cmd_loader.py` | 命令行参数定义与解析 |
| `easymocap/dataset/mv1pmf.py` | `MV1PMF` 多视角单人数据集 |
| `easymocap/dataset/base.py` | `MVBase` 基类: 相机读取、数据加载、文件写入 |
| `easymocap/dataset/config.py` | Body25 等关键点配置 (关节数、骨架连接) |
| `easymocap/mytools/reconstruction.py` | 三角化 (`batch_triangulate`) 和重投影 (`projectN3`) |
| `easymocap/mytools/camera_utils.py` | 相机参数读写、畸变矫正 |
| `easymocap/mytools/file_utils.py` | JSON 读写、标注读取、结果写出 |
| `easymocap/mytools/writer.py` | `FileWriter`: 结果写出和可视化 |
| `easymocap/smplmodel/body_param.py` | SMPL 模型加载、参数工具函数 |
| `easymocap/smplmodel/body_model.py` | `SMPLlayer`: PyTorch SMPL 前向计算 |
| `easymocap/pipeline/basic.py` | `smpl_from_keypoints3d2d`: 多阶段优化编排 |
| `easymocap/pipeline/weight.py` | 各模型类型的默认损失权重 |
| `easymocap/pipeline/config.py` | `Config`: 优化控制参数 |
| `easymocap/pyfitting/optimize_simple.py` | `optimizeShape`, `optimizePose3D`, `optimizePose2D` |
| `easymocap/pyfitting/lossfactory.py` | 各种损失函数 (3D, 2D, 平滑, 正则) |
| `easymocap/visualize/renderer.py` | pyrender 渲染器 |

---

## 7. 数据流总结

```
输入:
  images/{cam}/*.jpg           各视角图片
  annots/{cam}/*.json          各视角 2D 关键点 (25 joints, Body25)
  intri.yml + extri.yml        相机参数

         │
         ▼
  ┌──────────────────────┐
  │  Phase 1: 三角化      │
  │  多视角 2D → 3D       │
  │  (SVD/DLT)           │
  └──────────┬───────────┘
             │
             ▼
  keypoints3d/{:06d}.json     3D 关键点 (25, 4)

         │
         ▼
  ┌──────────────────────────────────────────────┐
  │  Phase 2: SMPL 拟合                           │
  │  Step A: optimizeShape (骨长 → shapes)        │
  │  Step B: optimizePose3D (3D kpts → Rh,Th,poses)│
  │  Step C: optimizePose2D (2D kpts → 微调)      │
  └──────────┬───────────────────────────────────┘
             │
             ▼
  smpl/{:06d}.json            SMPL 参数: Rh(1,3), Th(1,3), poses(1,72), shapes(1,10)

输出可视化:
  detec/{:06d}.jpg            2D 检测可视化
  repro/{:06d}.jpg            三角化重投影可视化
  repro.mp4                   重投影视频
  smpl/{:06d}.jpg             SMPL 渲染可视化
  smpl.mp4                    SMPL 渲染视频
  repro_smpl/{:06d}.jpg       SMPL 关节重投影可视化
  repro_smpl.mp4              SMPL 关节重投影视频
```

---

### Doc Mvmp (`doc/mvmp.md`)

# EasyMocap - mvmp

This code aims to solve the problem of reconstructing multiple persons from multiple calibrated cameras. The released code is an easy-to-use version. See [Advanced](#Advanced) for more details.

## 0. Preparation

Prepare your calibrated and synchronized system by yourself.

You can download our dataset [here](https://zjueducn-my.sharepoint.com/:u:/g/personal/s_q_zju_edu_cn/EZFGgpK2Y6RBkPbGvny_PC0BIS08qJvxGYEHYopjhHX_TQ?e=LY3pgm).

```bash
├── intri.yml
├── extri.yml
├── annots
│   ├── 0
│   ├── 1
│   ├── 2
│   ├── 3
│   ├── 4
│   ├── 5
│   ├── 6
│   └── 7
└── videos
    ├── 0.mp4
    ├── 1.mp4
    ├── 2.mp4
    ├── 3.mp4
    ├── 4.mp4
    ├── 5.mp4
    ├── 6.mp4
    └── 7.mp4
```

Extract the images from videos:
```bash
data=/path/to/data
python3 scripts/preprocess/extract_video.py ${data} --no2d
```

## 1. Reconstucting human pose
This step will reconstruct the human pose in each frame.
```bash
python3 apps/demo/mvmp.py ${data} --out ${data}/output --annot annots --cfg config/exp/mvmp1f.yml --undis --vis_det --vis_repro
```

## 2. Recovering SMPL body model
First we should tract the human pose in each frame. This step will track and interpolate missing frames.
```bash
python3 apps/demo/auto_track.py ${data}/output ${data}/output-track --track3d
```

Then we can fit SMPL model to the tracked keyponts:

```bash
python3 apps/demo/smpl_from_keypoints.py ${data} --skel ${data}/output-track/keypoints3d --out ${data}/output-track/smpl --verbose --opts smooth_poses 1e1
```

To visualize the results, see [visualization tutorial](./doc/realtime_visualization.md)


## Advanced

For more complicated scenes, our lab has a real-time version of this algorithm, which can perform 3D reconstruction and tracking simultaneously.

If you want to use this part for commercial queries, please contact [Xiaowei Zhou](mailto:xwzhou@zju.edu.cn).



https://user-images.githubusercontent.com/22812405/123629197-968c0080-d846-11eb-8417-4e6d3a65466d.mp4

---

### Doc Realtime Visualization (`doc/realtime_visualization.md`)

# EasyMoCap -> Real-time Visualization

We are the first one to release a real-time visualization tool for both skeletons and SMPL/SMPL+H/SMPL-X/MANO models.

## Install

Please install `EasyMocap` first. This part requires `Open3D==0.9.0`:

```bash
python3 -m pip install open3d==0.9.0
```

## Open the server
Before any visualization, you should run a server:

```bash
# quick start:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml
# If you want to specify the host and port:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml host <your_ip_address> port <set_a_port>
```

This step will open the visualization window:



You can alternate the viewpoints free. Press `a` to automatic rotate the scene.  The configuration file `config/vis/o3d_scene.yml` defines the scene and other properties. In the default setting, we define the xyz-axis in the origin, the bounding box of the scene and a chessboard in the ground.

## Send the data

If you are success to open the server, you can visualize your 3D data anywhere. We provide an example code:

```bash
python3 apps/vis/vis_client.py --path <path/to/your/keypoints3d> --host <previous_ip_address> --port <previous_port>
```

Take the `zju-ls-feng` results as example, you can show the skeleton in the main window:



## Embed this feature to your code

To add this visualization to your other code, you can follow these steps:

```bash
# 1. import the base client
from easymocap.socket.base_client import BaseSocketClient
# 2. set the ip address and port
client = BaseSocketClient(host, port)
# 3. send the data
client.send(data)
```

The format of data is:
```python
data = [
    {
        'id': 0,
        'keypoints3d': numpy.ndarray # (nJoints, 4) , (x, y, z, c) for each joint
    },
    {
        'id': 1,
        'keypoints3d': numpy.ndarray # (nJoints, 4)
    }
]
```

## Define your scene

In the configuration file, we main define the `body_model` and `scene`. You can replace them for your data.

## Examples

To understand our code, we provide lots of results for visualization.

First download the data [here](https://zjueducn-my.sharepoint.com/:u:/g/personal/s_q_zju_edu_cn/EQO5cILlYS1BgTT1ufDv2N4Bj41cvhstgmw_tMyi-6smWA?e=wSS9FB) and set the path to `vis`:

```bash
vis=/path/to/vis/data
```

### 1. Skeletons

Basic skeletons:

```bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml write True out ${vis}/output/skel-body25 camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smpl/keypoints3d
```

<div align="center">
    <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/assets/vis3d/skel-body25.gif" width="80%">
</div>

Body+Face+Hand:
```bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_total.yml write True out ${vis}/output/skel-total camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smplx/keypoints3d
```

<div align="center">
    <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/assets/vis3d/skel-total.gif" width="80%">
</div>

Multiple Person:

```bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene.yml write True out ${vis}/output/skel-multi camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/multi/keypoints3d --step 4
```

<div align="center">
    <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/assets/vis3d/skel-multi.gif" width="80%">
</div>

### 2. Mesh

SMPL:
```bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_smpl.yml write True out ${vis}/output/mesh-smpl camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smpl/smpl --smpl
```

<div align="center">
    <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/assets/vis3d/mesh-smpl.gif" width="80%">
</div>

SMPLX:
```bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_smplx.yml write True out ${vis}/output/mesh-smplx camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/smplx/smpl --smpl
```

<div align="center">
    <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/assets/vis3d/mesh-smplx.gif" width="80%">
</div>

MANO:
```bash
# Start the server:
python3 apps/vis/vis_server.py --cfg config/vis3d/o3d_scene_manol.yml write True out ${vis}/output/mesh-manol camera.cz 3. camera.cy 0.5
# Send the keypoints:
python3 apps/vis/vis_client.py --path ${vis}/manol/smpl --smpl
```

<div align="center">
    <img src="https://raw.githubusercontent.com/chingswy/Dataset-Demo/main/assets/vis3d/mesh-manol.gif" width="80%">
</div>

## Advanced

### 1. Camera Setting

Try to modify these keys to control the location and rotation of the cameras.
```yml
camera:
  phi: 0
  theta: -30
  cx: 0.
  cy: 0.
  cz: 6.
```

### 2. Scene Setting

We provide some useful mesh in `easymocap.visualize.o3dwrapper`. If you want to add your own 3D mesh, add it to the key `scene`.

### 3. Body Model

At present, we just allow use one type of body model in the scene for fast visualization. So you must set the body model before you run the `apps/vis/vis_server.py`. If you want to use different models in a scene, you can implement it and pull a request.

---

### Doc Todo (`doc/todo.md`)

# TODO

This part is coming soon, please stay tuned.

---

### Doc Tutorial New Task (`doc/tutorial_new_task.md`)

# Add new tasks

## 0. Prepare the data and dataset

## 1. Add new loss functions

## 2. Add new optimization

## 3. Write your own main function

## 4. Evaluation for the new tasks

---

--- METRICS ---
- Files Extracted: 11
- Estimated Token Budget: ~9418 tokens
- Canonical Reference: https://codewiki.google/github.com/zju3dv/EasyMocap