Awesome-Open-Vocabulary-Semantic-Segmentation

If you find this project helpful, please consider giving it a star ⭐.

Contents

<!-- vim-markdown-toc GitLab -->

• Open-Vocabulary Semantic Segmentation(mainly updated by @liuqinying)

  • Fully-supervised Methods
  • Weakly-supervised Methods
  • Training-free Methods
  • Others

• Zero-shot Semantic Segmentation (mainly updated by @liuqinying)

• Referring-Image-Segmentation (mainly updated by @ghost-000)

  • Fully-Supervised Methods
  • Weakly-Supervised Methods

• Open-Vocabulary Object Detection (mainly updated by @tbh3223)

• Universal Semantic Segmentation

• Other Related Work

• Related Survey

<!-- vim-markdown-toc -->

Open-Vocabulary Semantic Segmentation

Fully-Supervised Open-Vocabulary Semantic Segmentation

The model is trained on fully-supervised semantic segmentation datasets with pixel-level annotations (e.g., COCO Stuff dataset).

1. <span id = "1001">[LSeg]</span> | ICLR'22 | Language-driven Semantic Segmentation | [pdf] | [code]
2. <span id = "1002">[OpenSeg]</span> | ECCV'22 | Scaling Open-vocabulary Image Segmentation with Image-level Labels | [pdf] | [code]
3. <span id = "1003">[Xu et al.]</span> | ECCV'22 | A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model | [pdf] | [code]
4. <span id = "1003">[SegCLIP]</span> | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
5. <span id = "1003">[MaskCLIP]</span> | ICML'23 | Open-Vocabulary Universal Image Segmentation with MaskCLIP | [pdf] | [code]
6. <span id = "1003">[OVSeg]</span> | CVPR'23 | Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP | [pdf] | [code]
7. <span id = "1003">[X-Decoder]</span> | CVPR'23 | Generalized Decoding for Pixel, Image, and Language | [pdf] | [code]
8. <span id = "1003">[SAN]</span> | CVPR'23(Highlight) | Side Adapter Network for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
9. <span id = "1003">[SAN]</span> | TAPMI'23 | SAN: Side Adapter Network for Open-vocabulary Semantic Segmentation | [pdf] | [code]
10. <span id = "1003">[ODISE]</span> | CVPR'23 | Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models | [pdf] | [code]
11. <span id = "1003">[FreeSeg]</span> | CVPR'23 | FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation | [pdf] | [code]
12. <span id = "1003">[OpenSeeD]</span> | ICCV'23 | A Simple Framework for Open-Vocabulary Segmentation and Detection | [pdf] | [code]
13. <span id = "1003">[GKC]</span> | ICCV'23 | Global Knowledge Calibration for Fast Open-Vocabulary Segmentation | [pdf]
14. <span id = "1003">[OPSNet]</span> | ICCV'23 | Open-vocabulary Panoptic Segmentation with Embedding Modulation | [pdf] | [code]
15. <span id = "2005">[MasQCLIP]</span> | ICCV'23 | MasQCLIP for Open-Vocabulary Universal Image Segmentation | [pdf]
16. <span id = "2005">[DeOP]</span> | ICCV'23 | Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network | [pdf] | [code]
17. <span id = "2005">[Li et al.]</span> | ICCV'23 | Open-vocabulary Object Segmentation with Diffusion Models | [pdf] | [code]
18. <span id = "1003">[HIPIE]</span> | NeurIPS'23 | Hierarchical Open-vocabulary Universal Image Segmentation | [pdf] | [code]
19. <span id = "1003">[FC-CLIP]</span> | NeurIPS'23 | Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP | [pdf] | [code]
20. <span id = "1003">[MAFT]</span> | NeurIPS'23 | Learning Mask-aware CLIP Representations for Zero-Shot Segmentation | [pdf] | [code]
21. <span id = "1003">[ADA]</span> | NeurIPS'23 | Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation | [pdf]
22. <span id = "1003">[Dao et al]</span> | TMM | Class Enhancement Losses with Pseudo Labels for Open-Vocabulary Semantic Segmentation | [pdf]
23. <span id = "1003">[SELF-SEG]</span> | Arixv'23.12 | Self-Guided Open-Vocabulary Semantic Segmentation | [pdf]
24. <span id = "1003">[OpenSD]</span> | Arixv'23.12 | OpenSD: Unified Open-Vocabulary Segmentation and Detection | [pdf] | [code]
25. <span id = "1003">[RENOVATE]</span> | Arixv'24.03 | Renovating Names in Open-Vocabulary Segmentation Benchmarks | [pdf]
26. <span id = "1003">[DreamCLIP]</span> | ECCV'24 | DreamLIP: Language-Image Pre-training with Long Captions | [pdf] | [code]
27. <span id = "1003">[CAT-Seg]</span> | CVPR'24 | CAT-Seg : Cost Aggregation for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
28. <span id = "1003">[SED]</span> | CVPR'24 | SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
29. <span id = "1003">[SCAN]</span> | CVPR'24 | Open-Vocabulary Segmentation with Semantic-Assisted Calibration | [pdf] | [code]
30. <span id = "1003">[OpenTrans]</span> | CVPR'24 | Transferable and Principled Efficiency for Open-Vocabulary Segmentation | [pdf] | [code])
31. <span id = "1003">[H-CLIP]</span> | Arixv'24.05 | Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation | [pdf]
32. <span id = "1003">[OpenDAS]</span> | Arixv'24.05 | OpenDAS: Domain Adaptation for Open-Vocabulary Segmentation | [pdf]
33. <span id = "1003">[USE]</span> | CVPR'24 | USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation | [pdf]
34. <span id = "1003">[EBSeg]</span> | CVPR'24 | Open-Vocabulary Semantic Segmentation with Image Embedding Balancing | [pdf] | [code])
35. <span id = "1003">[MAFT+]</span> | ECCV'24 | Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation | [pdf] | [code])

Weakly-Supervised Open-Vocabulary Semantic Segmentation

[text-supervised/language-supervised] The model is trained on weakly supervised datasets with only image-level annotations/captions (e.g., CC12M dataset).

1. <span id = "2001">[GroupViT]</span> | CVPR'22 | GroupViT: Semantic Segmentation Emerges from Text Supervision | [pdf] | [code]
2. <span id = "2002">[ViL-Seg]</span> | ECCV'22 | Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding | [pdf]
3. <span id = "2003">[MaskCLIP+]</span> | ECCV'22(Oral) | Extract Free Dense Labels from CLIP | [pdf] | [code]
4. <span id = "2006">[ViewCo]</span> | ICLR'23 | Viewco: Discovering Text-supervised Segmentation Masks via Multi-view Semantic Consistency | [pdf]
5. <span id = "2004">[SegCLIP]</span> | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
6. <span id = "2005">[CLIP-S4]</span> | CVPR'23 | CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation | [pdf]
7. <span id = "2005">[PACL]</span> | CVPR'23 | Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning | [pdf]
8. <span id = "2005">[OVSegmentor]</span> | CVPR'23 | Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision | [pdf] | [code]
9. <span id = "2005">[SimSeg]</span> | CVPR'23 | A Simple Framework for Text-Supervised Semantic Segmentation | [pdf] | [code]
10. <span id = "2005">[TCL]</span> | CVPR'23 | Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs | [pdf] | [code]
11. <span id = "2005">[SimCon]</span> | Arxiv'23.02 | SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation | [pdf]
12. <span id = "2005">[Zhang et al.]</span> |