<div align="center"> <h2> LLMs Meet Multimodal Generation and Editing: A Survey </h2> <a href='https://arxiv.org/abs/2405.19334'><img src='https://img.shields.io/badge/ArXiv-2405.19334-red'></a> </div>

🤗 Introduction

• This repository contains a curated list of LLMs meet multimodal generation. Modalities consist of visual (including image, video and 3D) and audio (including sound, speech and music). <p align="center"> <img src="assets/fig.jpg" width=300"">

</p>

• We welcome any contributions and suggestions to our repository or the addition of your own work. Feel free to make a pull request or leave your comments!!

📋 Contents

• 🤗 Introduction
• 📋 Contents
• 💘 Tips
• 📍 Multimodal Generation
  • Image Generation
    • 🔅 LLM-based
    • Non-LLM-based (Clip/T5)
    • Datasets
  • Video Generation
    • 🔅 LLM-based
    • Non-LLM-based
    • Datasets
  • 3D Generation
    • 🔅 LLM-based
    • Non-LLM-based (Clip/T5)
    • Datasets
  • Audio Generation
    • 🔅 LLM-based
    • Non-LLM-based
    • Datasets
  • Generation with Multiple Modalities
    • 🔅 LLM-based
    • Non-LLM-based
• 📍 Multimodal Editing
  • Image Editing
    • 🔅 LLM-based
    • Non-LLM-based (Clip/T5)
  • Video Editing
    • 🔅 LLM-based
    • Non-LLM-based (Clip/T5)
  • 3D Editing
    • 🔅 LLM-based
    • Non-LLM-based (Clip/T5)
  • Audio Editing
    • 🔅 LLM-based
    • Non-LLM-based (Clip/T5)
• 📍 Multimodal Agents
• 📍 Multimodal Understanding with LLMs
  • Image Understanding
  • Video Understanding
  • 3D Understanding
  • Audio Understanding
• 📍 Multimodal LLM Safety
  • Attack
  • Defense and Detect
  • Alignment
  • Datasets
  • 3D, Video and Audio Safety
• 📍 Related Surveys
  • LLM
  • Vision
• 👨‍💻 Team
• 😉 Citation
• ⭐️ Star History

💘 Tips

• ✅ Paper searching via catatogue: directly clicking the content of the catatogue to select the area of your research and browse related papers.
• ✅ Paper searching via author name: Free feel to search papers of a specific author via ctrl + F and then type the author name. The dropdown list of authors will automatically expand when searching.
• ✅ Paper searching via tag: You can also search the related papers via the following tags: customization, iteractive, human motion generation tokenizer. (More tags are ongoing)

📍 Multimodal Generation

Image Generation

🔅 LLM-based

• An Image is Worth 32 Tokens for Reconstruction and Generation (11 June 2024)<details><summary>Qihang Yu, Mark Weber, Xueqing Deng et al.</summary> Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, Liang-Chieh Chen</details></details>

• TRINS: Towards Multimodal Language Models that Can Read (10 June 2024)<details><summary>[CVPR 2024] Ruiyi Zhang, Yanzhe Zhang, Jian Chen et al.</summary> Ruiyi Zhang, Yanzhe Zhang, Jian Chen, Yufan Zhou, Jiuxiang Gu, Changyou Chen, Tong Sun</details></details>

<!-- + **Genie: Generative Interactive Environments** (26 Feb 2024)<details><summary>Jake Bruce, Michael Dennis, Ashley Edwards, et al.</summary> Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, Tim Rocktäschel</details> [](https://arxiv.org/abs/2402.15391v1) -->

• Chameleon: Mixed-Modal Early-Fusion Foundation Models (16 May 2024)<details><summary>Chameleon Team</details>

• Graphic Design with Large Multimodal Model (22 Apr 2024)<details><summary>Yutao Cheng, Zhao Zhang, Maoke Yang, et al.</summary> Yutao Cheng, Zhao Zhang, Maoke Yang, Hui Nie, Chunyuan Li, Xinglong Wu, and Jie Shao</details>

• PMG : Personalized Multimodal Generation with Large Language Models (7 Apr 2024)<details><summary>Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, et al.</summary>Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, Xi Xiao</details>

• MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control (19 Mar 2024)<details><summary>Enshen Zhou, Yiran Qin, Zhenfei Yin, et al.</summary>Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, Jing Shao</details>

• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (8 Mar 2024)<details><summary>Xiwei Hu, Rui Wang, Yixiao Fang, et al.</summary> Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, Gang Yu</details>

• StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis (30 Jan 2024)<details><summary>Zecheng Tang, Chenfei Wu, Zekai Zhang, et al.</summary>Zecheng Tang, Chenfei Wu, Zekai Zhang, Mingheng Ni, Shengming Yin, Yu Liu, Zhengyuan Yang, Lijuan Wang, Zicheng Liu, Juntao Li, Nan Duan</details> tokenizer

• DiffusionGPT: LLM-Driven Text-to-Image Generation System (18 Jan 2024)<details><summary>Jie Qin, Jie Wu, Weifeng Chen, et al.</summary> Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, Shilei Wen</details>

• StarVector: Generating Scalable Vector Graphics Code from Images (17 Dec 2023)<details><summary>Juan A. Rodriguez, Shubham Agarwal, Issam H. Laradji, et al.</summary> Juan A. Rodriguez, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, Marco Pedersoli</details>

• VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation (14 Dec 2023)<details><summary>Jinguo Zhu, Xiaohan Ding, Yixiao Ge, et al.</summary> Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, Ying Shan</details>

• StoryGPT-V: Large Language Models as Consistent Story Visualizers (13 Dec 2023)<details><summary>Xiaoqian Shen, Mohamed Elhoseiny</summary> Xiaoqian Shen, Mohamed