lorax学习资料汇总 - 支持千级微调模型的多LoRA推理服务器

lorax简介

lorax(LoRA eXchange)是一个多LoRA推理服务框架,允许用户在单个GPU上部署数千个微调模型,大幅降低服务成本而不影响吞吐量和延迟。它由Predibase公司开发并开源,采用Apache 2.0许可证。

lorax logo

主要特性

lorax具有以下主要特性:

🚅 动态适配器加载:可以在请求中包含来自HuggingFace、Predibase或任何文件系统的微调LoRA适配器,它会即时加载而不阻塞并发请求。
🏋️‍♀️ 异构连续批处理:将不同适配器的请求打包到同一批次中,使延迟和吞吐量与并发适配器数量几乎保持恒定。
🧁 适配器交换调度:在GPU和CPU内存之间异步预取和卸载适配器,调度请求批处理以优化系统的总吞吐量。
👬 优化推理:包括张量并行、预编译CUDA内核、量化、token流等高吞吐量和低延迟优化。
🚢 生产就绪:预构建的Docker镜像、Kubernetes的Helm图表、Prometheus指标和Open Telemetry分布式追踪。支持多轮对话的OpenAI兼容API。
🤯 免费商用:Apache 2.0许可证。

快速开始

安装要求

Nvidia GPU (Ampere及以上)
CUDA 11.8+兼容设备驱动
Linux操作系统
Docker

启动lorax服务器

安装 nvidia-container-toolkit
执行以下命令:

sudo systemctl daemon-reload
sudo systemctl restart docker

model=mistralai/Mistral-7B-Instruct-v0.1
volume=$PWD/data

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/predibase/lorax:main --model-id $model

更多详细教程请参考Docker快速入门指南。

使用方法

通过REST API调用

基础LLM调用示例:

curl 127.0.0.1:8080/generate \
    -X POST \
    -d '{
        "inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]",
        "parameters": {
            "max_new_tokens": 64
        }
    }' \
    -H 'Content-Type: application/json'

LoRA适配器调用示例:

curl 127.0.0.1:8080/generate \
    -X POST \
    -d '{
        "inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]",
        "parameters": {
            "max_new_tokens": 64,
            "adapter_id": "vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k"
        }
    }' \
    -H 'Content-Type: application/json'

详细API文档请参考REST API参考。

通过Python客户端调用

安装:

pip install lorax-client

使用示例:

from lorax import Client

client = Client("http://127.0.0.1:8080")

# 调用基础LLM
prompt = "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]"
print(client.generate(prompt, max_new_tokens=64).generated_text)

# 调用LoRA适配器
adapter_id = "vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k"
print(client.generate(prompt, max_new_tokens=64, adapter_id=adapter_id).generated_text)

更多详情请参考Python客户端参考。

通过OpenAI兼容API进行对话

lorax支持通过OpenAI兼容的API进行多轮对话,同时支持动态加载适配器。只需将任意适配器指定为model参数即可。

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://127.0.0.1:8080/v1",
)

resp = client.chat.completions.create(
    model="alignment-handbook/zephyr-7b-dpo-lora",
    messages=[
        {
            "role": "system",
            "content": "You are a friendly chatbot who always responds in the style of a pirate",
        },
        {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
    ],
    max_tokens=100,
)
print("Response:", resp.choices[0].message.content)