Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Meta-Llama-3.1-8B-Instruct-FP8-dynamic项目介绍

项目概述

Meta-Llama-3.1-8B-Instruct-FP8-dynamic是一个强大的文本生成模型，能够在多个语言中进行文本输入和输出。该模型针对文本生成任务进行了优化，适用于商业和研究用途，特别是助理类聊天应用。然而，该模型不适用于违反法律法规的用途。

模型优化

该模型通过将Meta-Llama-3.1-8B-Instruct的权重和激活进行FP8量化，从而实现优化。这种量化技术能够将每个参数的比特数从16减少到8，从而大幅减少硬盘空间和GPU内存需求。

量化仅应用于transformers块内的线性操作符的权重和激活上。采用对称的每通道量化，即通过线性缩放每个输出维度来映射量化后的FP8权重和激活表示。此外，还在逐个token的动态基础上对激活进行量化。量化过程使用了LLM Compressor工具。

部署

该模型可以通过vLLM软件后端高效部署。下列Python示例展示了如何加载和使用该模型：

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic"

sampling_params = SamplingParams(temperature=0.6, top_p=0.9, max_tokens=256)

tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompts = tokenizer.apply_chat_template(messages, tokenize=False)

llm = LLM(model=model_id)

outputs = llm.generate(prompts, sampling_params)

generated_text = outputs[0].outputs[0].text
print(generated_text)

vLLM还支持与OpenAI兼容的服务。详细信息请参见文档。

创建

这个模型的创建过程使用了LLM Compressor与UltraChat校准样本。具体代码片段如下：

import torch

from transformers import AutoTokenizer

from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
from llmcompressor.transformers.compression.helpers import (  # noqa
    calculate_offload_device_map,
    custom_offload_device_map,
)

recipe = """
quant_stage:
    quant_modifiers:
        QuantizationModifier:
            ignore: ["lm_head"]
            config_groups:
                group_0:
                    weights:
                        num_bits: 8
                        type: float
                        strategy: channel
                        dynamic: false
                        symmetric: true
                    input_activations:
                        num_bits: 8
                        type: float
                        strategy: token
                        dynamic: true
                        symmetric: true
                    targets: ["Linear"]
"""

model_stub = "meta-llama/Meta-Llama-3.1-8B-Instruct"
model_name = model_stub.split("/")[-1]

device_map = calculate_offload_device_map(
    model_stub, reserve_for_hessians=False, num_gpus=1, torch_dtype="auto"
)

model = SparseAutoModelForCausalLM.from_pretrained(
    model_stub, torch_dtype="auto", device_map=device_map
)

output_dir = f"./{model_name}-FP8-dynamic"

oneshot(
    model=model,
    recipe=recipe,
    output_dir=output_dir,
    save_compressed=True,
    tokenizer=AutoTokenizer.from_pretrained(model_stub),
)