RGBD semantic segmentation

A paper list of RGBD semantic segmentation.

*Last updated: 2023/10/07

Update log

2020/May - update all of recent papers and make some diagram about history of RGBD semantic segmentation.
2020/July - update some recent papers (CVPR2020) of RGBD semantic segmentation.
2020/August - update some recent papers (ECCV2020) of RGBD semantic segmentation.
2020/October - update some recent papers (CVPR2020, WACV2020) of RGBD semantic segmentation.
2020/November - update some recent papers (ECCV2020, arXiv), the links of papers and codes for RGBD semantic segmentation.
2020/December - update some recent papers (PAMI, PRL, arXiv, ACCV) of RGBD semantic segmentation.
2021/February - update some recent papers (TMM, NeurIPS, arXiv) of RGBD semantic segmentation.
2021/April - update some recent papers (CVPR2021, ICRA2021, IEEE SPL, arXiv) of RGBD semantic segmentation.
2021/July - update some recent papers (CVPR2021, ICME2021, arXiv) of RGBD semantic segmentation.
2021/August - update some recent papers (IJCV, ICCV2021, IEEE SPL, arXiv) of RGBD semantic segmentation.
2022/January - update some recent papers (TITS, PR, IEEE SPL, arXiv) of RGBD semantic segmentation.
2022/March - update benchmark results on Cityscapes and ScanNet datasets.
2022/April - update some recent papers (CVPR, BMVC, IEEE TMM, arXiv) of RGBD semantic segmentation.
2022/May - update some recent papers of RGBD semantic segmentation.
2022/July - update some recent papers of RGBD semantic segmentation.
2023/January - update some recent papers of RGBD semantic segmentation.
2023/October - update some recent papers of RGBD semantic segmentation.

Table of Contents

• Datasets
• Metrics
• Performance tables
• Paper list

Datasets

The papers related to datasets used mainly in natural/color image segmentation are as follows.

• [NYUDv2] The NYU-Depth V2 dataset consists of 1449 RGB-D images showing interior scenes, which all labels are usually mapped to 40 classes. The standard training and test set contain 795 and 654 images, respectively.
• [SUN RGB-D] The SUN RGB-D dataset contains 10,335 RGBD images with semantic labels organized in 37 categories. The 5,285 images are used for training, and 5050 images are used for testing.
• [2D-3D-S] Stanford-2D-3D-Semantic dataset contains 70496 RGB and depth images as well as 2D annotation with 13 object categories. Areas 1, 2, 3, 4, and 6 are utilized as the training and Area 5 is used as the testing set.
• [Cityscapes] Cityscapes contains a diverse set of stereo video sequences recorded in street scenes from 50 different cities, with high quality pixel-level annotations of 5 000 frames in addition to a larger set of 20 000 weakly annotated frames.
• [ScanNet] ScanNet is an RGB-D video dataset containing 2.5 million views in more than 1500 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations.

Metrics

The papers related to metrics used mainly in RGBD semantic segmentation are as follows.

• [PixAcc] Pixel accuracy
• [mAcc] Mean accuracy
• [mIoU] Mean intersection over union
• [f.w.IOU] Frequency weighted IOU

Performance tables

Speed is related to the hardware spec (e.g. CPU, GPU, RAM, etc), so it is hard to make an equal comparison. We select four indexes namely PixAcc, mAcc, mIoU, and f.w.IOU to make comparison. The closer the segmentation result is to the ground truth, the higher the above four indexes are.

NYUDv2