Awesome-Multimodal-Large-Language-Models

Our MLLM works

🔥🔥🔥 A Survey on Multimodal Large Language Models
Project Page [This Page] | Paper

The first comprehensive survey for Multimodal Large Language Models (MLLMs). :sparkles: </div>

Welcome to add WeChat ID (wmd_ustc) to join our MLLM communication group! :star2: </div>

---

🔥🔥🔥 VITA: Towards Open-Source Interactive Omni Multimodal LLM

<p align="center"> <img src="./images/vita.png" width="80%" height="80%"> </p>

<font size=7><div align='center' > [🍎 Project Page] [📖 arXiv Paper] [🌼 GitHub] </div></font>

[2024.08.12] We are announcing VITA, the first-ever open-source Multimodal LLM that can process Video, Image, Text, and Audio, and meanwhile has an advanced multimodal interactive experience. 🌟

<b>Omni Multimodal Understanding</b>. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. ✨

<b>Non-awakening Interaction</b>. VITA can be activated and respond to user audio questions in the environment without the need for a wake-up word or button. ✨

<b>Audio Interrupt Interaction</b>. VITA is able to simultaneously track and filter external queries in real-time. This allows users to interrupt the model's generation at any time with new questions, and VITA will respond to the new query accordingly. ✨

---

🔥🔥🔥 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

<p align="center"> <img src="./images/videomme.jpg" width="80%" height="80%"> </p>

<font size=7><div align='center' > [🍎 Project Page] [📖 arXiv Paper] [📊 Dataset][🏆 Leaderboard] </div></font>

[2024.06.03] We are very proud to launch Video-MME, the first-ever comprehensive evaluation benchmark of MLLMs in Video Analysis! 🌟

It applies to both <b>image MLLMs</b>, i.e., generalizing to multiple images, and <b>video MLLMs</b>. Our leaderboard involes SOTA models like Gemini 1.5 Pro, GPT-4o, GPT-4V, LLaVA-NeXT-Video, InternVL-Chat-V1.5, and Qwen-VL-Max. 🌟

It includes both <b>short- (< 2min)</b>, <b>medium- (4min~15min)</b>, and <b>long-term (30min~60min)</b> videos, ranging from <b>11 seconds to 1 hour</b>. ✨

<b>All data are newly collected and annotated by humans, not from any existing video dataset</b>. ✨

---

🔥🔥🔥 MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Project Page [Leaderboards] | Paper | :black_nib: Citation

A comprehensive evaluation benchmark for MLLMs. Now the leaderboards include 50+ advanced models, such as Qwen-VL-Max, Gemini Pro, and GPT-4V. :sparkles:

If you want to add your model in our leaderboards, please feel free to email bradyfu24@gmail.com. We will update the leaderboards in time. :sparkles:

<details><summary>Download MME :star2::star2: </summary>

The benchmark dataset is collected by Xiamen University for academic research only. You can email yongdongluo@stu.xmu.edu.cn to obtain the dataset, according to the following requirement.

Requirement: A real-name system is encouraged for better academic communication. Your email suffix needs to match your affiliation, such as xx@stu.xmu.edu.cn and Xiamen University. Otherwise, you need to explain why. Please include the information bellow when sending your application email.

Name: (tell us who you are.)
Affiliation: (the name/url of your university or company)
Job Title: (e.g., professor, PhD, and researcher)
Email: (your email address)
How to use: (only for non-commercial use)

</details>

---

<br> 📑 If you find our projects helpful to your research, please consider citing: <br>

@article{fu2023mme,
  title={MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models},
  author={Fu, Chaoyou and Chen, Peixian and Shen, Yunhang and Qin, Yulei and Zhang, Mengdan and Lin, Xu and Yang, Jinrui and Zheng, Xiawu and Li, Ke and Sun, Xing and others},
  journal={arXiv preprint arXiv:2306.13394},
  year={2023}
}

@article{fu2024vita,
  title={VITA: Towards Open-Source Interactive Omni Multimodal LLM},
  author={Fu, Chaoyou and Lin, Haojia and Long, Zuwei and Shen, Yunhang and Zhao, Meng and Zhang, Yifan and Wang, Xiong and Yin, Di and Ma, Long and Zheng, Xiawu and He, Ran and Ji, Rongrong and Wu, Yunsheng and Shan, Caifeng and Sun, Xing},
  journal={arXiv preprint arXiv:2408.05211},
  year={2024}
}

@article{fu2024video,
  title={Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis},
  author={Fu, Chaoyou and Dai, Yuhan and Luo, Yondong and Li, Lei and Ren, Shuhuai and Zhang, Renrui and Wang, Zihan and Zhou, Chenyu and Shen, Yunhang and Zhang, Mengdan and others},
  journal={arXiv preprint arXiv:2405.21075},
  year={2024}
}

@article{yin2023survey,
  title={A survey on multimodal large language models},
  author={Yin, Shukang and Fu, Chaoyou and Zhao, Sirui and Li, Ke and Sun, Xing and Xu, Tong and Chen, Enhong},
  journal={arXiv preprint arXiv:2306.13549},
  year={2023}
}

---

<font size=5><center><b> Table of Contents </b> </center></font>

• Awesome Papers
  • Multimodal Instruction Tuning
  • Multimodal Hallucination
  • Multimodal In-Context Learning
  • Multimodal Chain-of-Thought
  • LLM-Aided Visual Reasoning
  • Foundation Models
  • Evaluation
  • Multimodal RLHF
  • Others
• Awesome Datasets
  • Datasets of Pre-Training for Alignment
  • Datasets of Multimodal Instruction Tuning
  • Datasets of In-Context Learning
  • Datasets of Multimodal Chain-of-Thought
  • Datasets of Multimodal RLHF
  • Benchmarks for Evaluation
  • Others

---

Awesome Papers

Multimodal Instruction Tuning

Title	Venue	Date	Code	Demo
<br> mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models <br>	arXiv	2024-08-09	Github	-
<br> VITA: Towards Open-Source Interactive Omni Multimodal LLM <br>	arXiv	2024-08-09	Github	-
<br> LLaVA-OneVision: Easy Visual Task Transfer <br>	arXiv	2024-08-06	Github	Demo
<br> MiniCPM-V: A GPT-4V Level MLLM on Your Phone <br>	arXiv	2024-08-03	Github	Demo
VILA^2: VILA Augmented VILA	arXiv	2024-07-24	-	-
SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models	arXiv	2024-07-22	-	-
EVLM: An Efficient Vision-Language Model for Visual Understanding	arXiv	2024-07-19	-	-
<br> InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output <br>	arXiv	2024-07-03	Github	Demo
<br> OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding <br>	arXiv	2024-06-27	Github	Local Demo
<br> Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs <br>	arXiv	2024-06-24	Github	Local Demo
<br> Long Context Transfer from Language to Vision <br>	arXiv	2024-06-24	Github	Local Demo
<br> Unveiling Encoder-Free Vision-Language Models <br>	arXiv	2024-06-17	Github	Local Demo
<br> Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models <br>	arXiv	2024-06-12	Github	-
<br> VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs <br>	arXiv	2024-06-11	Github	Local Demo
<br> Parrot: Multilingual Visual Instruction Tuning <br>	arXiv	2024-06-04	Github	-
<br> Ovis: Structural Embedding Alignment for Multimodal Large Language Model <br>	arXiv	2024-05-31	Github	-
<br> Matryoshka Query Transformer for Large Vision-Language Models <br>	arXiv	2024-05-29	Github	Demo
<br> ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models <br>	arXiv	2024-05-24	Github	-
<br> Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models <br>	arXiv	2024-05-24	Github	Demo
<br> [**Libra: Building Decoupled