DeepSeek-VL

<div align="center"> <img src="https://yellow-cdn.veclightyear.com/835a84d5/4fd454d9-01db-458b-85dd-1b81706afc78.svg" width="60%" alt="DeepSeek LLM" /> </div> <hr> <div align="center"> <a href="https://www.deepseek.com/" target="_blank"> <img alt="主页" src="https://yellow-cdn.veclightyear.com/835a84d5/16572eae-5c7b-4766-b1fd-b2c4ba4fa44b.svg" /> </a> <a href="https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B" target="_blank"> <img alt="聊天" src="https://img.shields.io/badge/🤖%20聊天-DeepSeek%20VL-536af5?color=536af5&logoColor=white" /> </a> <a href="https://huggingface.co/deepseek-ai" target="_blank"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white" /> </a> </div> <div align="center"> <a href="https://discord.gg/Tc7c45Zzu5" target="_blank"> <img alt="Discord" src="https://img.shields.io/badge/Discord-DeepSeek%20AI-7289da?logo=discord&logoColor=white&color=7289da" /> </a> <a href="images/qr.jpeg" target="_blank"> <img alt="微信" src="https://img.shields.io/badge/微信-DeepSeek%20AI-brightgreen?logo=wechat&logoColor=white" /> </a> <a href="https://twitter.com/deepseek_ai" target="_blank"> <img alt="关注Twitter" src="https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x&logoColor=white" /> </a> </div> <div align="center"> <a href="LICENSE-CODE"> <img alt="代码许可" src="https://img.shields.io/badge/代码许可-MIT-f5de53?&color=f5de53"> </a> <a href="LICENSE-MODEL"> <img alt="模型许可" src="https://img.shields.io/badge/模型许可-模型协议-f5de53?&color=f5de53"> </a> </div> <p align="center"> <a href="#3-model-downloads"><b>📥 模型下载</b></a> | <a href="#4-quick-start"><b>⚡ 快速开始</b></a> | <a href="#5-license"><b>📜 许可证</b></a> | <a href="#6-citation"><b>📖 引用</b></a> <br> <a href="https://arxiv.org/abs/2403.05525"><b>📄 论文链接</b></a> | <a href="https://huggingface.co/papers/2403.05525"><b>🤗 Huggingface论文链接</b></a> | <a href="https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B"><b>👁️ 演示</b></a> </p>

1. 简介

介绍DeepSeek-VL，这是一个为实际视觉和语言理解应用设计的开源视觉语言（VL）模型。DeepSeek-VL具有通用的多模态理解能力，能够处理逻辑图表、网页、公式识别、科学文献、自然图像，以及复杂场景中的具身智能。

DeepSeek-VL：迈向真实世界的视觉语言理解

卢浩宇*、刘文*、张博**、王冰轩、董凯、刘博、孙敬翔、任卓声、李卓书、杨浩、孙耀锋、邓成奇、徐瀚威、谢震达、阮冲（*共同第一作者，**项目负责人）

2. 发布

<details> <summary>✅ <b>2024-03-14</b>：DeepSeek-VL-7B的演示在<a href="https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B">Hugging Face</a>上可用。</summary> <br>请在<a href="https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B">https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B</a>查看DeepSeek-VL-7B的gradio演示。亲身体验其功能！ </details> <details> <summary>✅ <b>2024-03-13</b>：支持DeepSeek-VL gradio演示。 </details> <details> <summary>✅ <b>2024-03-11</b>：DeepSeek-VL系列发布，包括<code>DeepSeek-VL-7B-base</code>、<code>DeepSeek-VL-7B-chat</code>、<code>DeepSeek-VL-1.3B-base</code>和<code>DeepSeek-VL-1.3B-chat</code>。</summary> <br>此次发布包括DeepSeek-VL系列中为各种应用定制的多样化模型。这些模型有两种规模：7B和1.3B参数，每种规模都提供基础和聊天变体，以满足不同的需求和集成场景。 </details>

3. 模型下载

我们向公众发布了DeepSeek-VL系列，包括1.3B-base、1.3B-chat、7b-base和7b-chat模型，以支持学术和商业社区内更广泛、更多样化的研究。请注意，本模型的使用受许可证部分中概述的条款约束。根据这些条款，允许商业使用。

Huggingface

模型	序列长度	下载链接
DeepSeek-VL-1.3B-base	4096	🤗 Hugging Face
DeepSeek-VL-1.3B-chat	4096	🤗 Hugging Face
DeepSeek-VL-7B-base	4096	🤗 Hugging Face
DeepSeek-VL-7B-chat	4096	🤗 Hugging Face

4. 快速入门

安装

在Python 3.8或更高版本的环境中，运行以下命令安装必要的依赖项：

pip install -e .

简单推理示例

import torch
from transformers import AutoModelForCausalLM

from deepseek_vl.models import VLChatProcessor, MultiModalityCausalLM
from deepseek_vl.utils.io import load_pil_images


# 指定模型路径
model_path = "deepseek-ai/deepseek-vl-7b-chat"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

## 单图对话示例
conversation = [
    {
        "role": "User",
        "content": "<image_placeholder>描述这张图片的每个阶段。",
        "images": ["./images/training_pipelines.jpg"],
    },
    {"role": "Assistant", "content": ""},
]

## 多图（或上下文学习）对话示例
# conversation = [
#     {
#         "role": "User",
#         "content": "<image_placeholder>前景中的狗没有穿任何东西，"
#                    "<image_placeholder>一只戴着圣诞帽的狗，"
#                    "<image_placeholder>一只穿着巫师服装的狗，以及"
#                    "<image_placeholder>这只狗穿着什么？",
#         "images": [
#             "images/dog_a.png",
#             "images/dog_b.png",
#             "images/dog_c.png",
#             "images/dog_d.png",
#         ],
#     },
#     {"role": "Assistant", "content": ""}
# ]

# 加载图片并准备输入
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True
).to(vl_gpt.device)

# 运行图像编码器获取图像嵌入
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# 运行模型获取响应
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]}", answer)

命令行聊天

python cli_chat.py --model_path "deepseek-ai/deepseek-vl-7b-chat"

# 或本地路径
python cli_chat.py --model_path "本地模型路径"

Gradio演示

pip install -e .[gradio]

python deepseek_vl/serve/app_deepseek.py

玩得开心！

5. 许可证

此代码仓库采用MIT许可证。DeepSeek-VL Base/Chat模型的使用受DeepSeek模型许可证约束。DeepSeek-VL系列（包括Base和Chat）支持商业使用。

6. 引用

@misc{lu2024deepseekvl,
      title={DeepSeek-VL: Towards Real-World Vision-Language Understanding},
      author={Haoyu Lu and Wen Liu and Bo Zhang and Bingxuan Wang and Kai Dong and Bo Liu and Jingxiang Sun and Tongzheng Ren and Zhuoshu Li and Hao Yang and Yaofeng Sun and Chengqi Deng and Hanwei Xu and Zhenda Xie and Chong Ruan},
      year={2024},
      eprint={2403.05525},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}