sglang

PyPI - Downloads

</div>

| 博客 | 论文 | Slack |

SGLang是一个快速的大型语言模型和视觉语言模型服务框架。通过联合设计后端运行时和前端语言，它使您与模型的交互更快速、更可控。

核心特性包括：

快速后端运行时：通过RadixAttention进行前缀缓存、跳跃前向约束解码、连续批处理、令牌注意力（分页注意力）、张量并行、flashinfer内核和量化（AWQ/FP8/GPTQ/Marlin）实现高效服务。
灵活的前端语言：通过链式生成调用、高级提示、控制流、多模态、并行性和外部交互，轻松编程LLM应用程序。

新闻

[2024/07] 🔥 使用SGLang运行时实现更快的Llama3服务（相比TensorRT-LLM和vLLM）（博客）。
[2024/04] SGLang被官方**LLaVA-NeXT（视频）**发布使用（博客）。
[2024/02] SGLang通过压缩有限状态机实现3倍更快的JSON解码（博客）。

[2024/01] SGLang通过RadixAttention提供高达5倍更快的推理（博客）。
[2024/01] SGLang为官方LLaVA v1.6发布演示提供服务支持（使用方法）。

</details>

安装

方法1：使用pip

pip install --upgrade pip
pip install "sglang[all]"

# 安装FlashInfer CUDA内核
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

方法2：从源代码安装

# 使用最新的发布分支
git clone -b v0.2.13 https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

# 安装FlashInfer CUDA内核
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

方法3：使用docker

Docker镜像可在Docker Hub上找到，名为lmsysorg/sglang，基于Dockerfile构建。将下面的<secret>替换为您的huggingface hub token。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

方法4：使用docker compose

如果您计划将其作为服务运行，推荐使用此方法。更好的方法是使用k8s-sglang-service.yaml。

将compose.yml复制到本地机器
在终端中执行命令docker compose up -d。

通用说明

FlashInfer目前是SGLang必须安装的依赖之一。如果您使用的是sm80以下的NVIDIA GPU设备，如T4，暂时无法使用SGLang。我们预计很快会解决这个问题，请保持关注。如果您在sm80+设备（如A100、L40S、H100）上遇到任何与FlashInfer相关的问题，可以考虑使用Triton的内核，方法是添加--disable-flashinfer --disable-flashinfer-sampling，并提出一个issue。
如果您只需要使用OpenAI后端，可以通过pip install "sglang[openai]"来避免安装其他依赖。

后端：SGLang运行时（SRT）

SGLang运行时（SRT）是一个高效的服务引擎。

快速开始

启动服务器

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000

发送请求

curl http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "从前有一个",
    "sampling_params": {
      "max_new_tokens": 16,
      "temperature": 0
    }
  }'

在这里了解更多关于参数格式的信息。

兼容OpenAI的API

此外，服务器支持兼容OpenAI的API。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

它支持流式传输、视觉，以及OpenAI API参考中指定的Chat/Completions/Models/Batch端点的大多数功能。

附加服务器参数

添加--tp 2以启用多GPU张量并行。如果报错"peer access is not supported between these two devices"，请在服务器启动命令中添加--enable-p2p-check。

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --tp 2

添加--dp 2以启用多GPU数据并行。它也可以与张量并行一起使用。如果有足够的内存，数据并行对吞吐量更有利。

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --dp 2 --tp 2

如果在服务过程中看到内存不足错误，请尝试通过设置较小的--mem-fraction-static值来减少KV缓存池的内存使用。默认值为0.9。

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --mem-fraction-static 0.7

有关调整超参数以获得更好性能的信息，请参阅hyperparameter_tuning.md。
如果在处理长提示时出现内存不足错误，请尝试设置较小的分块预填充大小。

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --chunked-prefill-size 4096

添加--nnodes 2以在多个节点上运行张量并行。如果您有两个节点，每个节点有两个GPU，想要运行TP=4，让sgl-dev-0作为第一个节点的主机名，50000作为可用端口。

# 节点0
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 0

# 节点1
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1

如果模型在Hugging Face tokenizer中没有模板，您可以指定一个自定义聊天模板。
要启用实验性的torch.compile支持，您可以添加--enable-torch-compile。它可以加速小批量大小下的小型模型。
要启用fp8量化，您可以在fp16检查点上添加--quantization fp8，或者直接加载fp8检查点而不指定任何参数。

支持的模型

Llama / Llama 2 / Llama 3 / Llama 3.1
Mistral / Mixtral / Mistral NeMo
Gemma / Gemma 2
Qwen / Qwen 2 / Qwen 2 MoE
DeepSeek / DeepSeek 2
LLaVA 1.5 / 1.6
- python -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000
- python -m sglang.launch_server --model-path liuhaotian/llava-v1.6-vicuna-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000
- python -m sglang.launch_server --model-path liuhaotian/llava-v1.6-34b --tokenizer-path liuhaotian/llava-v1.6-34b-tokenizer --port 30000
LLaVA-NeXT-Video
- 参见 examples/usage/llava_video
Yi-VL
- 参见 srt_example_yi_vl.py
StableLM
Command-R
DBRX
Grok
ChatGLM
InternLM 2

支持新模型的说明在这里。

使用ModelScope中的模型

要使用ModelScope中的模型，请设置环境变量SGLANG_USE_MODELSCOPE。

export SGLANG_USE_MODELSCOPE=true

启动Qwen2-7B-Instruct服务器

SGLANG_USE_MODELSCOPE=true python -m sglang.launch_server --model-path qwen/Qwen2-7B-Instruct --port 30000

运行Llama 3.1 405B

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

性能基准测试

通过运行以下命令对单个静态批次进行基准测试，无需启动服务器。参数与launch_server.py相同。请注意，这不是动态批处理服务器，因此对于真实服务器可以处理的批次大小，它可能会耗尽内存。真实服务器会将预填充截断成几个批次，而这个单元测试不会。对于准确的大批量测试，请考虑使用sglang.bench_serving。
```
python -m sglang.bench_latency --model-path meta-llama/Meta-Llama-3-8B-Instruct --batch 32 --input-len 256 --output-len 32
```
对在线服务进行基准测试。首先启动一个服务器，然后运行以下命令。
```
python3 -m sglang.bench_serving --backend sglang --num-prompt 10
```

前端：结构化生成语言（SGLang）

前端语言可以与本地模型或API模型一起使用。它是OpenAI API的替代方案。对于复杂的提示工作流程，你可能会发现它更容易使用。

快速开始

下面的例子展示了如何使用sglang来回答一个多轮问题。

使用本地模型

首先，使用以下命令启动服务器

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000

然后，连接到服务器并回答一个多轮问题。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

使用OpenAI模型

设置OpenAI API密钥

export OPENAI_API_KEY=sk-******

然后，回答一个多轮问题。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

语言特性

首先，导入sglang。

[object Object],[object Object],[object Object],[object Object]

sglang提供了一些简单的原语，如gen、select、fork、image。你可以在由sgl.function装饰的函数中实现你的提示流程。然后，你可以用run或run_batch来调用这个函数。系统将为你管理状态、聊天模板、并行性和批处理。

以下示例的完整代码可以在readme_examples.py中找到

控制流

你可以在函数体内使用任何Python代码，包括控制流、嵌套函数调用和外部库。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

并行性

使用fork来启动并行提示。由于sgl.gen是非阻塞的，下面的for循环会并行发出两个生成调用。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

多模态

使用sgl.image来传递图像作为输入。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

另请参见srt_example_llava.py。

约束解码

使用regex来指定一个正则表达式作为解码约束。这只支持本地模型。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

JSON解码

使用regex来用正则表达式指定一个JSON模式。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

另请参阅json_decode.py以获取使用Pydantic模型指定格式的附加示例。

批处理

使用run_batch来执行连续批处理的一批请求。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

流式传输

添加stream=True以启用流式传输。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

角色

在使用聊天模型时，使用sgl.system、sgl.user和sgl.assistant来设置角色。您还可以使用开始和结束标记定义更复杂的角色提示。

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

技巧和实现细节

sgl.gen中的choices参数是通过计算所有选项的标准化令牌长度对数概率并选择概率最高的一个来实现的。
sgl.gen中的regex参数是通过自回归解码与logit偏置掩码结合来实现的，根据正则表达式设置的约束。它与temperature=0和temperature!=0兼容。