Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis

Project | Paper

This is the official PyTorch implementation of "Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis".

<a href="https://andrewsonga.github.io">Chonghyuk Song</a>, <a href="https://gengshan-y.github.io">Gengshan Yang</a>, <a href="https://dunbar12138.github.io">Kangle Deng</a>, <a href="https://www.cs.cmu.edu/~junyanz/">Jun-Yan Zhu</a>, <a href="https://www.cs.cmu.edu/~deva/">Deva Ramanan</a> Carnegie Mellon University ICCV 2023

https://user-images.githubusercontent.com/20153928/232976346-581c6080-6451-4306-bbf2-e9a34aea1599.mp4

Given a long video of deformable objects captured by a handheld RGBD sensor, Total-Recon renders the scene from novel camera trajectories derived from in-scene motion of actors: (1) egocentric cameras that simulate the point-of-view of a target actor (such as the pet) and (2) 3rd-person (or pet) cameras that follow the actor from behind. Our method also enables (3) 3D video filters that attach virtual 3D assets to the actor. Total-Recon achieves this by reconstructing the geometry, appearance, and root-body and articulated motion of each deformable object in the scene as well as the background.

Timeline

We plan to release our code in the following 4 stages:

Inference and Evaluation code for 4 select sequences of our stereo RGBD dataset
Raw data and pre-optimized models for all sequences of our dataset
Training code (per-object pretraining and joint-finetuning)
Data preprocessing code for user-provided RGBD videos

Previous Bugs

Before a recent commit there was a bug in the code that set the default $\lambda = 1$ (the interpolation factor of the EMA filter for updating object bounds and near-far plane: new state = (1 - $\lambda$) $\times$ signal + $\lambda$ $\times$ old state). This prevents these bounds from being updated at all during training and may result in failed reconstruction. The default value has now been corrected to 0.0 and the code now correctly updates the bounds and near-far plane during training. Please pull the latest version of the codebase for a bug-free experience.

Getting Started

Dependencies

(1) Clone repo (including submodules):

git clone https://github.com/andrewsonga/Total-Recon.git --recursive

# This step is REQUIRED for all subsequent steps!
cd Total-Recon

(2) Install conda env:

conda env create -f misc/totalrecon-cu113.yml
conda activate totalrecon-cu113

(3) Install submodules:

pip install -e third_party/pytorch3d
pip install -e third_party/kmeans_pytorch
python -m pip install detectron2 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html

(4) Install ffmpeg:

apt-get install ffmpeg

(5) Download the pre-trained VCN optical flow model for data preprocessing (instructions are taken from BANMo):

mkdir lasr_vcn
wget https://www.dropbox.com/s/bgsodsnnbxdoza3/vcn_rob.pth -O ./lasr_vcn/vcn_rob.pth

Data

The following steps (1) ~ (4) for downloading, preprocessing, and formatting RGBD sequences only pertains to Total-Recon's dataset. <ins>To apply Total-Recon on your own RGBD videos, please follow the instructions</ins> here.

(1) Download and untar the raw data:

bash download_rawdata.sh

# untar raw data
tar -xzvf totalrecon_rawdata.tar.gz

(2) Save the raw data under raw/:

src_dir=totalrecon_rawdata
bash place_rawdata.sh $src_dir

###############################################################
# argv[1]: The directory inside Total-Recon/ where the downloaded raw data is stored

(3) Preprocess raw data (takes around a couple of hours per sequence):

Multi-actor sequences:

# e.g.
prefix=humandog-stereo000; gpu=0
bash preprocess/preprocess_rawdata_stereo_maskcamgiven_multiactor.sh $prefix $gpu

###############################################################
# argv[1]: prefix of the preprocessed data folders under "database/DAVIS/JPEGImages/" (minus suffices such as "-leftcam", "-rightcam", "-human", "-animal", "-bkgd", and "-uncropped")
# argv[2]: gpu number (0, 1, 2, ...)

Uni-actor sequences:

# e.g.
prefix=cat2-stereo000; ishuman='n'; gpu=0
bash preprocess/preprocess_rawdata_stereo_maskcamgiven_uniactor.sh $prefix $ishuman $gpu

###############################################################
# argv[1]: prefix of the preprocessed data folders under "database/DAVIS/JPEGImages/" (minus suffices such as "-leftcam", "-rightcam", and "-bkgd")
# argv[2]: human or not, where `y` denotes human and  `n` denotes quadreped
# argv[3]: gpu number (0, 1, 2, ...)

(4) [NOT REQUIRED FOR INFERENCE] Format preprocessed data for training:

Multi-actor sequences:

# e.g.
prefix=humandog-stereo000; gpu=0
bash preprocess/format_processeddata_stereo_multiactor.sh $prefix $gpu

###############################################################
# argv[1]: prefix of the preprocessed data folders under "database/DAVIS/JPEGImages/" (minus suffices such as "-leftcam", "-rightcam", "-human", "-animal", "-bkgd", and "-uncropped")
# argv[2]: gpu number (0, 1, 2, ...)

Uni-actor sequences:

# e.g.
prefix=cat2-stereo000; gpu=0
bash preprocess/format_processeddata_stereo_uniactor.sh $prefix $gpu

###############################################################
# argv[1]: prefix of the preprocessed data folders under "database/DAVIS/JPEGImages/" (minus suffices such as "-leftcam", "-rightcam", and "-bkgd")
# argv[2]: gpu number (0, 1, 2, ...)

Pre-optimized Models

(1) Download the pre-optimized models and untar them.

bash download_models.sh

tar -xzvf totalrecon_models.tar.gz

(2) Appropriately relocate the pre-optimized models:

# Place the pre-optimized models under logdir/
# argv[1]: The directory inside Total-Recon where the downloaded pre-optimized models are stored

src_dir=totalrecon_models
bash place_models.sh $src_dir

3D Assets

To run the 3D video filter and to be able to visualize flying embodied-view cameras, purchase and download 3D models in .obj format for 1) the unicorn horn, and 2) the Canon camera.

Rename the .obj file for the camera mesh to camera.obj, then place the file camera.obj and unzipped folder UnicornHorn_OBJ inside mesh_material.

Training

To train Total-Recon on our provided dataset, run per-object pretraining and joint-finetuning as follows:

# change appropriately (e.g. "humancat-stereo000" or "cat2-stereo000")
prefix=humandog-stereo000 
gpus=0,1,2,3              # gpu ids for training
addr=10001                # master port for torch.distributed

bash train_$prefix.sh $gpus $addr

To train Total-Recon on your own videos, run one of the following commands:

# for multi-actor sequences
prefix=humancat-mono000
gpus=0,1,2,3
addr=10001

bash train_multiactor.sh $gpus $addr $prefix

# for uni-actor sequences
prefix=human2-mono000
gpus=0,1,2,3
addr=10001
use_human=""              # "" (for human actors) / "no" (for animal actors)

bash train_uniactor.sh $gpus $addr $prefix "$use_human"

Inference

Mesh and Root-body Pose Extraction

Before inference or evaluation can be done, extract the object-level meshes and root-body poses from the trained model. This only needs to be done once per model:

# argv[1]: gpu number (0, 1, 2, ...)
# argv[2]: folder name of the trained model inside logdir/

seqname=humandog-stereo000-leftcam-jointft    # (appropriately rename `seqname`)
bash extract_fgbg.sh $gpu $seqname

Left-Right Camera Registration

Before inference or evaluation can be done, copy the left-right camera registration data from the raw data directory to the trained model directory:

prefix=humandog-stereo000                     # (appropriately rename `prefix`)
seqname=$prefix-leftcam-jointft               # directory name of trained model

# for uniactor sequences
cp raw/$prefix-leftcam/normrefcam2secondcam.npy logdir/$seqname/   

# for multiactor sequences     
cp raw/$prefix-human-leftcam/normrefcam2secondcam.npy logdir/$seqname/

Egocentric View Synthesis

https://user-images.githubusercontent.com/20153928/234135753-610bc744-789e-4174-9b75-f7c979376506.mp4

(takes around a few hours) The rendered videos will be saved as nvs-fpsview-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_fps.sh $gpu $seqname "$add_args"

<details><summary>per-sequence arguments <code>(add_args)</code></summary>

Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args="--fg_obj_index 1 --asset_obj_index 1 --fg_normalbase_vertex_index 96800 --fg_downdir_vertex_index 1874 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --asset_offset_z -0.05 --scale_fps 0.50"

Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 150324 --fg_downdir_vertex_index 150506 --asset_scale 0.003 --firstpersoncam_offset_y 0.05 --firstpersoncam_offsetabt_xaxis 25 --firstpersoncam_offsetabt_yaxis 15 --firstpersoncam_offsetabt_zaxis 5 --fix_frame 50 --scale_fps 0.75"

Cat1 (v1)

seqname=cat1-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 204713 --fg_downdir_vertex_index 204830 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_yaxis -20 --firstpersoncam_offsetabt_zaxis 10 --asset_offset_z -0.05 --scale_fps 0.75"

Cat1 (v2)

seqname=cat1-stereo001-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 34175 --fg_downdir_vertex_index 6043 --asset_scale 0.003 --firstpersoncam_offset_z 0.13 --firstpersoncam_offsetabt_xaxis 35 --firstpersoncam_offsetabt_yaxis -20 --firstpersoncam_offsetabt_zaxis -15 --scale_fps 0.75"

Cat2 (v1)

seqname=cat2-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 338844 --fg_downdir_vertex_index 166318 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 10 --firstpersoncam_offsetabt_yaxis 10 --asset_offset_z -0.05 --scale_fps 0.75"

Cat2 (v2)

seqname=cat2-stereo001-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 308732 --fg_downdir_vertex_index 309449 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 20 --firstpersoncam_offsetabt_yaxis 20 --firstpersoncam_offsetabt_zaxis -20 --scale_fps 0.75"

Cat3

seqname=cat3-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 105919 --fg_downdir_vertex_index 246367 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 20 --firstpersoncam_offsetabt_zaxis 10 --asset_offset_z -0.05 --scale_fps 0.75"

Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 159244 --fg_downdir_vertex_index 93456 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 35 --firstpersoncam_offsetabt_yaxis 30 --firstpersoncam_offsetabt_zaxis 20 --asset_offset_z -0.05 --scale_fps 0.75"

Dog1 (v2)

seqname=dog1-stereo001-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 227642 --fg_downdir_vertex_index 117789 --asset_scale 0.003 --firstpersoncam_offset_z 0.035 --firstpersoncam_offsetabt_xaxis 45 --scale_fps 0.75"

Human 1

seqname=human1-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 161978 --fg_downdir_vertex_index 37496 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 10 --firstpersoncam_offsetabt_yaxis 10 --asset_offset_z -0.05 --scale_fps 0.75"

Human 2

seqname=human2-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 114756 --fg_downdir_vertex_index 114499 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --firstpersoncam_offsetabt_yaxis 20 --asset_offset_z -0.05 --scale_fps 0.75"

</details>

3rd-Person-Follow (3rd-Pet-Follow) View Synthesis

https://user-images.githubusercontent.com/20153928/234136211-02241af4-9e7b-486e-a2d3-36c21f384ecc.mp4

(takes around a few hours) The rendered videos will be saved as nvs-tpsview-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_tps.sh $gpu $seqname $add_args

<details><summary>per-sequence arguments <code>(add_args)</code></summary>

Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args="--fg_obj_index 1 --asset_obj_index 1 --thirdpersoncam_offset_y 0.25 --thirdpersoncam_offset_z -0.80 --asset_scale 0.003 --scale_tps 0.70"

Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args="--fg_obj_index 0 --asset_obj_index 0 --thirdpersoncam_fgmeshcenter_elevate_y 1.00 --thirdpersoncam_offset_x -0.05 --thirdpersoncam_offset_y 0.15