MOTSFusion

MOTSFusion: Track to Reconstruct and Reconstruct to Track

Introduction

This repository contains the corresponding source code for the paper "Track to Reconstruct and Reconstruct to Track" arXiv PrePrint.

News

This paper has been accepted as both a conference paper at ICRA 2020, as well as being accepted as a journal paper in Robotics and Automation Letters (RA-L)!!!

Requirements

The code was tested on:

• CUDA 9, cuDNN 7
• Tensorflow 1.13
• Python 3.6

Note: For some external code that is required to run in the precompute script, you need different requirements (see references). Please refer to the corresponding repositories to obtain the requirements for these.

Instructions

At first, download the datasets in section References (Stereo image pairs) as well as the detections and adapt the config files in ./configs according to your desired setup. The file "config_default" will run 2D as well as 3D tracking on the KITTI MOTS validation set. Next, download our pretrained segmentation network and extract it into './external/BB2SegNet'. Before running the main script, run:

python precompute.py -config ./configs/config_default

After this, all necessary information such as segmentations, optical flow, disparity and the corresponding pointcloud should be computed. Now you can run the tracker using:

python main.py -config ./configs/config_default

After the tracker has completed all sequences, results will be evaluated automatically.

Results

We release our results on the MOTS test set cars pedestrians

As well as the detections and segmentations (test-RRC, test-TRCNN, train/val-RRC, train/val-TRCNN).

References

• RRC Detections: https://github.com/JunaidCS032/MOTBeyondPixels - can be downloaded along with other features in section 'Running the demo script'
• Optical flow/disparity: https://github.com/lmb-freiburg/netdef_models
• Camera pose: https://github.com/raulmur/ORB_SLAM2
• MOTS dataset and TrackRCNN detections & segmentations: https://www.vision.rwth-aachen.de/page/mots
• MOT dataset: http://www.cvlibs.net/datasets/kitti/eval_tracking.php

Citation

@article{luiten2020track,
  title={Track to reconstruct and reconstruct to track},
  author={Luiten, Jonathon and Fischer, Tobias and Leibe, Bastian},
  journal={IEEE Robotics and Automation Letters},
  volume={5},
  number={2},
  pages={1803--1810},
  year={2020},
  publisher={IEEE}
}

License

MIT License