FasterTransformer4CodeFuse

| English |

</div>

简介

提供高性能模型推理，主要支持蚂蚁集团的CodeFuse模型。

与原始FT相比，本仓库具有以下特点：

:white_check_mark: CodeFuse模型的Int8量化
:white_check_mark: 提示词结尾不需要完整单词
:white_check_mark: Python API
:white_check_mark: Python API的流式输出
:white_check_mark: 更高的模型加载速度
:white_check_mark: 一些bug修复

性能

批处理大小：1

<table> <tr> <td colspan="3">模型</td> <td colspan="4">CodeFuse 13B</td> </tr> <tr> <td colspan="3">测量指标</td> <td colspan="4">延迟（毫秒）</td> </tr> <tr> <td colspan="3">GPU</td> <td colspan="2">单个A100</td> <td colspan="2">2 * A100 张量并行</td> </tr> <tr> <td colspan="3">数据类型</td> <td>fp16</td> <td>int8</td> <td>fp16</td> <td>int8</td> </tr> <tr> <td rowspan="4">输入/输出长度</td> <td>16</td> <td>8</td> <td>160</td> <td>195</td> <td>238</td> <td>84</td> </tr> <tr> <td>64</td> <td>32</td> <td>608</td> <td>369</td> <td>373</td> <td>295</td> </tr> <tr> <td>256</td> <td>128</td> <td>2650</td> <td>1530</td> <td>1492</td> <td>1130</td> </tr> <tr> <td>1024</td> <td>512</td> <td>10776</td> <td>7054</td> <td>6786</td> <td>5415</td> </tr> <tr> <td colspan="3">每秒生成的令牌数</td> <td>48</td> <td>75</td> <td>77</td> <td>98</td> </tr> </table>

快速开始

我们在容器环境中运行：nvcr.io/nvidia/pytorch:22.09-py3。

1. 安装依赖

pip install --no-cache-dir pybind11==2.6.2 transformers accelerate sentencepiece

echo "export pybind11_DIR=/opt/conda/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/" >> ~/.bashrc
export pybind11_DIR=/opt/conda/lib/python3.8/site-packages/pybind11/share/cmake/pybind11/

2. 构建

mkdir build ; cd build
export TORCH_PYTHON_LIBRARIES=/opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so
cmake -DCMAKE_BUILD_TYPE=Release -DSM="80;75" -DBUILD_PYT=ON -DSPARSITY_SUPPORT=OFF -DMEASURE_BUILD_TIME=ON \
      -DBUILD_CUTLASS_MIXED_GEMM=ON -DBUILD_MULTI_GPU=ON -DBUILD_TRT=OFF \
      -DENABLE_FP8=OFF -DBUILD_PYBIND=ON -DTORCH_PYTHON_LIBRARIES=${TORCH_PYTHON_LIBRARIES} ..
make -j"$(grep -c ^processor /proc/cpuinfo)"

3. 运行

您可以使用examples/pytorch/codefuse/huggingface_convert.py脚本将检查点文件从HuggingFace转换为FasterTransformer。

export MODEL_NAME=codefuse
export TENSOR_PARA_SIZE=2

python ../examples/pytorch/codefuse/huggingface_convert.py \
       -o ../models/${MODEL_NAME}/fastertransformer \
       -i ../models/${MODEL_NAME}/transformers \
       -infer_gpu_num ${TENSOR_PARA_SIZE} \
       -processes 20 \
       -weight_data_type fp16 \
       -model_name gptneox

您可以使用examples/pytorch/codefuse/quant_and_save.py脚本将fp16或fp32的FasterTransformer检查点文件转换为int8文件和尺度，获得更高的模型加载速度和更小的检查点文件。

export MODEL_NAME=codefuse
export TENSOR_PARA_SIZE=2

python ../examples/pytorch/codefuse/quant_and_save.py \
       --in_dir ../models/${MODEL_NAME}/fastertransformer/${TENSOR_PARA_SIZE}-gpu \
       --out_dir ../models/${MODEL_NAME}/fastertransformer/${TENSOR_PARA_SIZE}-gpu_int8 \
       --lib_path ../build/lib/libth_common.so \
       --tensor_para_size ${TENSOR_PARA_SIZE} \
       --use_gptj_residual \
       --data_type fp16

您可以使用examples/pytorch/codefuse/codefuse_example.py运行模型推理。

export MODEL_NAME=codefuse

# fp16 1gpu
python ../examples/pytorch/codefuse/codefuse_example.py \
       --ckpt_path ../models/${MODEL_NAME}/fastertransformer/1-gpu \
       --tokenizer_path ../models/${MODEL_NAME}/transformers

# int8 1gpu
python ../examples/pytorch/codefuse/codefuse_example.py \
       --ckpt_path ../models/${MODEL_NAME}/fastertransformer/1-gpu_int8 \
       --tokenizer_path ../models/${MODEL_NAME}/transformers \
       --int8_mode 1 \
       --enable_int8_weights 1

# fp16 2gpus
torchrun --nproc_per_node 2 ../examples/pytorch/codefuse/codefuse_example.py \
         --world_size 2 \
         --ckpt_path ../models/${MODEL_NAME}/fastertransformer/2-gpu \
         --tokenizer_path ../models/${MODEL_NAME}/transformers

# int8 2gpus
torchrun --nproc_per_node 2 ../examples/pytorch/codefuse/codefuse_example.py \
         --world_size 2 \
         --ckpt_path ../models/${MODEL_NAME}/fastertransformer/2-gpu_int8 \
         --tokenizer_path ../models/${MODEL_NAME}/transformers \
         --int8_mode 1 \
         --enable_int8_weights 1