<p align="center" width="100%"> <img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_banner.png" width="80%" height="80%"> </p> <div> <div align="center"> <a href='https://scholar.google.com/citations?user=KuYlJCIAAAAJ&hl=en' target='_blank'>张一元^1,2*</a>&emsp; <a href='https://kxgong.github.io/' target='_blank'>龚凯雄^1,2*</a>&emsp; <a href='http://kpzhang93.github.io/' target='_blank'>张凯鹏^2,†</a>&emsp; </br> <a href='http://www.ee.cuhk.edu.hk/~hsli/' target='_blank'>李浩昇^1,2</a>&emsp; <a href='https://mmlab.siat.ac.cn/yuqiao/index.html' target='_blank'>乔宇²</a>&emsp; <a href='https://wlouyang.github.io/' target='_blank'>欧阳万里²</a>&emsp; <a href='http://people.eecs.berkeley.edu/~xyyue/' target='_blank'>岳翔宇^1,†,‡</a> </div> <div> <div align="center"> ¹ <a href='http://mmlab.ie.cuhk.edu.hk/' target='_blank'>香港中文大学多媒体实验室</a>&emsp; </br> ² <a href='https://github.com/OpenGVLab' target='_blank'>上海人工智能实验室OpenGVLab </a></br> ^* 同等贡献&emsp; ^† 通讯作者&emsp; ^‡ 项目负责人&emsp; </div>

---

<a href="https://twitter.com/_akhaliq/status/1682248055637041152"><img src="https://yellow-cdn.veclightyear.com/835a84d5/774051ba-d803-4782-9004-d0a46755b8a0.png" width="25" height="25"></a> <a href="https://www.youtube.com/watch?v=V8L8xbsTyls&ab_channel=CSBoard"><img src="https://yellow-cdn.veclightyear.com/835a84d5/174f679c-4932-4c70-82d4-0d9a4d665536.png" width="25" height="25"></a> <a href='https://huggingface.co/kxgong/Meta-Transformer'> <img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\huggingface.png" width="25" height="25"> </a> <a href='https://open.spotify.com/episode/6JJxcy2zMtTwr4jXPQEXjh'> <img src="https://yellow-cdn.veclightyear.com/835a84d5/2bdcdabd-4a80-494e-9e65-76b71f341b6d.svg" width="20" height="20"></a>

结合大语言模型的Meta-Transformer ✨✨✨

我们很高兴呈现OneLLM，它将Meta-Transformer框架与多模态大语言模型相结合，实现了多模态联合训练🚀，支持更多模态包括fMRI、深度图和法线图🚀，并在25个基准测试中展示了非常令人印象深刻的性能🚀🚀🚀。

🔥🔥 代码、预训练模型和数据集可在OneLLM公开获取。

🔥🔥 项目网站在OneLLM。

🌟 单一基础模型支持广泛应用

作为基础模型，Meta-Transformer可以处理来自12种模态的数据，这决定了它可以支持广泛的应用。如图所示，Meta-Transformer可为下游任务提供服务，包括股票分析📈、天气预报☀️ ☔ ☁️ ❄️ ⛄ ⚡、遥感📡、自动驾驶🚗、社交网络🌍、语音识别🔉等。

<p align="center" width="100%"> <img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_application.png" width="100%" height="100%"> </p>

表1：Meta-Transformer能够处理多达12种模态，包括自然语言<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\text.jpg" width="15" height="15">、RGB图像<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\img.jpg" width="15" height="15">、点云<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\pcd.jpg" width="15" height="15">、音频<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\audio.jpg" width="15" height="15">、视频<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\video.jpg" width="15" height="15">、表格数据<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\table.jpg" width="15" height="15">、图<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\graph.jpg" width="15" height="15">、时间序列数据<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\time.jpg" width="15" height="15">、高光谱图像<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\hyper.jpg" width="15" height="15">、IMU<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\imu.jpg" width="15" height="15">、医学图像<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\xray.jpg" width="15" height="15">和红外图像<img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\icons\infrared.jpg" width="15" height="15">。

<p align="left"> <img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_cmp.png" width=100%> </p>

🚩🚩🚩 共享编码器、非配对数据、更多模态

<div> <img class="image" src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_teaser.png" width="52%" height="100%"> <img class="image" src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_exp.png" width="45.2%" height="100%"> </div>

本仓库旨在探索Transformer在多模态学习中的潜力和可扩展性。我们利用Transformer处理可变长度序列的优势，提出了遵循元方案的"数据到序列"标记化，然后将其应用于12种模态，包括文本、图像、点云、音频、视频、红外、高光谱、X射线、表格、图、时间序列和惯性测量单元（IMU）数据。

<p align="left"> <img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_data2seq.png" width=100%> </p>

获得标记序列后，我们采用模态共享编码器来提取不同模态的表示。通过特定任务的头部，Meta-Transformer可以处理不同模态的各种任务，如分类、检测和分割。

<p align="left"> <img src="https://raw.githubusercontent.com/invictus717/MetaTransformer/master/assets\Meta-Transformer_framework.png" width=100%> </p> # 🌟 新闻

• 2023.8.17: 发布直接从多模态获取嵌入的代码。我们将进一步发布利用Meta-Transformer进行以人为中心的视觉任务的代码。
• 2023.8.2: 🎉🎉🎉 已发布用于图像、点云、图形、表格、时间序列、X射线、高光谱、LiDAR数据的Meta-Transformer实现。我们还发布了一个非常强大的自动驾驶基础模型 🚀🚀🚀。
• 2023.7.22: 已发布我们Meta-Transformer的预训练权重和使用演示。全面的文档和图像模态的实现正在进行中，将很快发布。敬请期待更多精彩更新！⌛⌛⌛
• 2023.7.21: 论文发布在arxiv，代码将逐步发布。
• 2023.7.8: Github仓库初始化。

🔓 模型库

<summary> 开源模态无关模型 </summary> <br> <div>

模型	预训练	规模	参数量	下载	国内下载源
Meta-Transformer-B16	LAION-2B	Base	85M	ckpt	ckpt
Meta-Transformer-L14	LAION-2B	Large	302M	ckpt	ckpt

</div>

• 预训练编码器使用演示

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

🕙 待办事项

• [ x ] 带有大型语言模型的Meta-Transformer。
• [ x ] 使用Meta-Transformer进行多模态联合训练。
• [ x ] 支持更多模态和更多任务。

联系方式

🚀🚀🚀 我们希望将这个仓库打造成一个强大的基础，用于各种模态的主流AI感知任务。您的贡献可以在这个努力中发挥重要作用，我们热烈欢迎您参与我们的项目！

如需联系我们，请随时发送电子邮件至yiyuanzhang.ai@gmail.com、kaixionggong@gmail.com、zhangkaipeng@pjlab.org.cn或xyyue@ie.cuhk.edu.hk！ <br></br>

 

引用

如果代码和论文对您的研究有帮助，请引用：

@article{zhang2023meta,
  title={Meta-transformer: A unified framework for multimodal learning},
  author={Zhang, Yiyuan and Gong, Kaixiong and Zhang, Kaipeng and Li, Hongsheng and Qiao, Yu and Ouyang, Wanli and Yue, Xiangyu},
  journal={arXiv preprint arXiv:2307.10802},
  year={2023}
}

许可证

本项目基于Apache 2.0许可证发布。

致谢

本代码基于优秀的开源项目开发，包括MMClassification、MMDetection、MMsegmentation、OpenPoints、Time-Series-Library、Graphomer、SpectralFormer和ViT-Adapter。