Awesome-Controllable-T2I-Diffusion-Models

<!-- # <p align=center>`awesome gan-inversion`</p> -->

<br /> <p align="center"> <h1 align="center">Awesome Controllable T2I Diffusion Models</h1> </p> <br />

We are focusing on how to Control text-to-image diffusion models with Novel Conditions.

For more detailed information, please refer to our survey paper: Controllable Generation with Text-to-Image Diffusion Models: A Survey

<p align="center"> <img src="assets/count.png" alt="img" width="49%" /> <img src="assets/controllable_generation.png" alt="img" width="49%" /> </p>

💖 Citation

If you find value in our survey paper or curated collection, please consider citing our work and starring our repo to support us.

@article{cao2024controllable,
  title={Controllable Generation with Text-to-Image Diffusion Models: A Survey},
  author={Pu Cao and Feng Zhou and Qing Song and Lu Yang},
  journal={arXiv preprint arXiv:2403.04279},
  year={2024}
}

🎁 How to contribute to this repository?

Since the following content is generated based on our database, please provide the following information in the issue to help us fill in the database to add new papers (please do not submit a PR directly).

1. Paper title
2. arXiv ID (if any)
3. Publication status (if any)

🌈 Contents

• Generation with Specific Condition
  • Personalization
    • Subject-Driven Generation
    • Person-Driven Generation
    • Style-Driven Generation
    • Interaction-Driven Generation
    • Image-Driven Generation
    • Distribution-Driven Generation
  • Spatial Control
  • Advanced Text-Conditioned Generation
  • In-Context Generation
  • Brain-Guided Generation
  • Sound-Guided Generation
  • Text Rendering
• Generation with Multiple Conditions
  • Joint Training
  • Continual Learning
  • Weight Fusion
  • Attention-based Integration
  • Guidance Composition
• Universal Controllable Generation
  • Universal Conditional Score Prediction
  • Universal Condition-Guided Score Estimation

<!-- start -->

🚀Generation with Specific Condition

🍇Personalization

🍉Subject-Driven Generation

PartCraft: Crafting Creative Objects by Parts.<br> Kam Woh Ng, Xiatian Zhu, Yi-Zhe Song, Tao Xiang.<br> ECCV 2024. [PDF]

ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance.<br> Jiannan Huang, Jun Hao Liew, Hanshu Yan, Yuyang Yin, Yao Zhao, Yunchao Wei.<br> arXiv 2024. [PDF]

Personalized Residuals for Concept-Driven Text-to-Image Generation.<br> Cusuh Ham, Matthew Fisher, James Hays, Nicholas Kolkin, Yuchen Liu, Richard Zhang, Tobias Hinz.<br> arXiv 2024. [PDF]

Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance.<br> Kelvin C. K. Chan, Yang Zhao, Xuhui Jia, Ming-Hsuan Yang, Huisheng Wang.<br> arXiv 2024. [PDF]

MMTryon: Multi-Modal Multi-Reference Control for High-Quality Fashion Generation.<br> Xujie Zhang, Ente Lin, Xiu Li, Yuxuan Luo, Michael Kampffmeyer, Xin Dong, Xiaodan Liang.<br> arXiv 2024. [PDF]

Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting.<br> Weili Zeng, Yichao Yan, Qi Zhu, Zhuo Chen, Pengzhi Chu, Weiming Zhao, Xiaokang Yang.<br> arXiv 2024. [PDF]

CAT: Contrastive Adapter Training for Personalized Image Generation.<br> Jae Wan Park, Sang Hyun Park, Jun Young Koh, Junha Lee, Min Song.<br> arXiv 2024. [PDF]

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation.<br> Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, Xiao Yang.<br> arXiv 2024. [PDF]

U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation.<br> You Wu, Kean Liu, Xiaoyue Mi, Fan Tang, Juan Cao, Jintao Li.<br> arXiv 2024. [PDF]

Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation.<br> Yutong He, Alexander Robey, Naoki Murata, Yiding Jiang, Joshua Williams, George J. Pappas, Hamed Hassani, Yuki Mitsufuji, Ruslan Salakhutdinov, J. Zico Kolter.<br> arXiv 2024. [PDF]

Attention Calibration for Disentangled Text-to-Image Personalization.<br> Yanbing Zhang, Mengping Yang, Qin Zhou, Zhe Wang.<br> arXiv 2024. [PDF]

Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization.<br> Jimyeong Kim, Jungwon Park, Wonjong Rhee.<br> arXiv 2024. [PDF]

MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration.<br> Zhichao Wei, Qingkun Su, Long Qin, Weizhi Wang.<br> arXiv 2024. [PDF]

Generative Active Learning for Image Synthesis Personalization.<br> Xulu Zhang, Wengyu Zhang, Xiao-Yong Wei, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, Qing Li.<br> arXiv 2024. [PDF]

Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization.<br> Yeji Song, Jimyeong Kim, Wonhark Park, Wonsik Shin, Wonjong Rhee, Nojun Kwak.<br> arXiv 2024. [PDF]

Tuning-Free Image Customization with Image and Text Guidance.<br> Pengzhi Li, Qiang Nie, Ying Chen, Xi Jiang, Kai Wu, Yuhuan Lin, Yong Liu, Jinlong Peng, Chengjie Wang, Feng Zheng.<br> arXiv 2024. [PDF]

Fast Personalized Text-to-Image Syntheses With Attention Injection.<br> Yuxuan Zhang, Yiren Song, Jinpeng Yu, Han Pan, Zhongliang Jing.<br> arXiv 2024. [PDF]

OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models.<br> Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, Wenhan Luo.<br> arXiv 2024. [PDF]

StableGarment: Garment-Centric Generation via Stable Diffusion.<br> Rui Wang, Hailong Guo, Jiaming Liu, Huaxia Li, Haibo Zhao, Xu Tang, Yao Hu, Hao Tang, Peipei Li.<br> arXiv 2024. [PDF]

Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation.<br> Likun Li, Haoqi Zeng, Changpeng Yang, Haozhe Jia, Di Xu.<br> arXiv 2024. [PDF]

FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation.<br> Pengchong Qiao, Lei Shang, Chang Liu, Baigui Sun, Xiangyang Ji, Jie Chen.<br> arXiv 2024. [PDF]

RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization.<br> Mengqi Huang, Zhendong Mao, Mingcong Liu, Qian He, Yongdong Zhang.<br> arXiv 2024. [PDF]

DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models.<br> Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, Pin-Yu Chen.<br> arXiv 2024. [PDF]

Direct Consistency Optimization for Compositional Text-to-Image Personalization.<br> Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, Jinwoo Shin.<br> arXiv 2024. [PDF]

ComFusion: Personalized Subject Generation in Multiple Specific Scenes From Single Image.<br> Yan Hong, Jianfu Zhang.<br> arXiv 2024. [PDF]

Visual Concept-driven Image Generation with Text-to-Image Diffusion Model.<br> Tanzila Rahman, Shweta Mahajan, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Leonid Sigal.<br> arXiv 2024. [PDF]

Textual Localization: Decomposing Multi-concept Images for Subject-Driven Text-to-Image Generation.<br> Junjie Shentu, Matthew Watson, Noura Al Moubayed.<br> arXiv 2024. [PDF]

DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization.<br> Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, Seunggyu Chang.<br> CVPR 2024. [PDF]

SeFi-IDE: Semantic-Fidelity Identity Embedding for Personalized Diffusion-Based Generation.<br> Yang Li, Songlin Yang, Wei Wang, Jing Dong.<br> arXiv 2024. [PDF]

Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization.<br> Henglei Lv, Jiayu Xiao, Liang Li, Qingming Huang.<br> arXiv 2024. [PDF]

Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding.<br> Jianxiang Lu, Cong Xie, Hui Guo.<br> arXiv 2024. [PDF]

BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models.<br> Senthil Purushwalkam, Akash Gokul, Shafiq Joty, Nikhil Naik.<br> arXiv 2024. [PDF]

PALP: Prompt Aligned Personalization of Text-to-Image Models.<br> Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, Ariel Shamir.<br> arXiv 2024. [PDF]

Cross Initialization for Personalized Text-to-Image Generation.<br> Lianyu Pang, Jian Yin, Haoran Xie, Qiping Wang, Qing Li, Xudong Mao.<br> CVPR 2024. [PDF]

DreamTuner: Single Image is Enough for Subject-Driven Generation.<br> Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, Qian He.<br> arXiv 2023. [PDF]

Decoupled Textual Embeddings for Customized Image Generation.<br> Yufei Cai, Yuxiang Wei, Zhilong Ji, Jinfeng Bai, Hu Han, Wangmeng Zuo.<br> arXiv 2023. [PDF]

Compositional Inversion for Stable Diffusion Models.<br> Xulu Zhang, Xiao-Yong Wei, Jinlin Wu, Tianyi Zhang, Zhaoxiang Zhang, Zhen Lei, Qing Li.<br> AAAI 2024. [PDF]

Customization Assistant for Text-to-image Generation.<br> Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun.<br> CVPR 2024. [PDF]

VideoBooth: Diffusion-based Video Generation with Image Prompts.<br> Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, Ziwei Liu.<br> arXiv 2023. [PDF]

HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models.<br> Zhonghao Wang, Wei Wei, Yang Zhao, Zhisheng Xiao, Mark Hasegawa-Johnson, Humphrey Shi, Tingbo Hou.<br> arXiv 2023. [PDF]

VideoAssembler: Identity-Consistent Video Generation with Reference Entities using Diffusion Model.<br> Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Zuxuan Wu, Hang Xu, Yu-Gang Jiang.<br> arXiv 2023.