Welcome to MindSpore Transformers (MindFormers)

I. Introduction

The goal of the MindSpore Transformers suite is to build a comprehensive development kit for large model training, fine-tuning, evaluation, inference, and deployment: It provides mainstream pre-trained Transformer models and state-of-the-art downstream task applications in the industry, covering rich parallel features. It aims to help users easily achieve large model training and innovative research and development.

Based on MindSpore's built-in parallel technology and modular design, the MindSpore Transformers suite has the following features:

Seamless transition from single card to large-scale cluster training with one line of code;
Flexible and easy-to-use personalized parallel configuration;
Automatic topology awareness, efficiently integrating data parallel and model parallel strategies;
One-click start of single-card/multi-card training, fine-tuning, evaluation, and inference processes for any task;
Supports users to configure any module in a modular way, such as optimizers, learning strategies, network assembly, etc.;
Provides advanced and easy-to-use interfaces such as Trainer, pipeline, and AutoClass;
Provides pre-set SOTA weight automatic download and loading functions;
Supports seamless migration and deployment in artificial intelligence computing centers;

If you have any suggestions for MindSpore Transformers, please contact us through issues, and we will handle them promptly.

The currently supported model list is as follows:

Model	Task (task name)	Model (model name)
LLama2	text_generation	llama2_7b <br>llama2_13b <br>llama2_7b_lora <br>llama2_13b_lora <br>llama2_70b
GLM2	text_generation	glm2_6b<br>glm2_6b_lora
CodeGeex2	text_generation	codegeex2_6b
LLama	text_generation	llama_7b <br>llama_13b <br>llama_7b_lora
GLM	text_generation	glm_6b<br>glm_6b_lora
Bloom	text_generation	bloom_560m<br>bloom_7.1b <br>
GPT2	text_generation	gpt2_small <br>gpt2_13b <br>
PanGuAlpha	text_generation	pangualpha_2_6_b<br>pangualpha_13b
BLIP2	contrastive_language_image_pretrain<br> zero_shot_image_classification	blip2_stage1_vit_g
CLIP	contrastive_language_image_pretrain<br> zero_shot_image_classification	clip_vit_b_32<br>clip_vit_b_16 <br>clip_vit_l_14<br>clip_vit_l_14@336
BERT	masked_language_modeling<br>text_classification <br>token_classification <br>question_answering	bert_base_uncased <br>txtcls_bert_base_uncased<br>txtcls_bert_base_uncased_mnli <br>tokcls_bert_base_chinese<br>tokcls_bert_base_chinese_cluener <br>qa_bert_base_uncased<br>qa_bert_base_chinese_uncased
T5	translation	t5_small
sam	segment_anything	sam_vit_b <br>sam_vit_l <br>sam_vit_h
MAE	masked_image_modeling	mae_vit_base_p16
VIT	image_classification	vit_base_p16
Swin	image_classification	swin_base_p4w7

The currently supported model list in research is as follows:

Model	Task (task name)	Model (model name)
skywork	text_generation	skywork_13b
Baichuan2	text_generation	baichuan2_7b <br>baichuan2_13b <br>baichuan2_7b_lora <br>baichuan2_13b_lora
Baichuan	text_generation	baichuan_7b <br>baichuan_13b
Qwen	text_generation	qwen_7b
Wizardcoder	text_generation	wizardcoder_15b
Internlm	text_generation	Internlm_7b
ziya	text_generation	ziya_13b

II. mindformers Installation

Method 1: Linux source code compilation installation

Source code compilation installation is supported. Users can execute the following commands to install the package

git clone -b dev https://gitee.com/mindspore/mindformers.git
cd mindformers
bash build.sh

Method 2: Docker image

Docker download command

docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers0.8.0_mindspore2.2.0:aarch_20231025

Create container

# --device is used to control the NPU card number and range of the specified container
# -v is used to map directories outside the container
# --name is used to customize the container name
docker run -it -u root \
--ipc=host \
--network host \
--device=/dev/davinci0 \
--device=/dev/davinci1 \
--device=/dev/davinci2 \
--device=/dev/davinci3 \
--device=/dev/davinci4 \
--device=/dev/davinci5 \
--device=/dev/davinci6 \
--device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /etc/localtime:/etc/localtime \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /var/log/npu/:/usr/slog \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
--name {请手动输入容器名称} \
swr.cn-central-221.ovaijisuan.com/mindformers/mindformers0.8.0_mindspore2.2.0:aarch_20231025 \
/bin/bash

三、版本匹配关系

版本对应关系	MindFormers	MindPet	MindSpore	Python	芯片	备注
版本号	dev	1.0.2	2.2	3.9	Ascend 910A/B	开发分支(非稳定版本)
版本号	0.8	1.0.2	2.2	3.9	Ascend 910A/B	发布版本分支

四、快速使用

MindFormers套件提供两种使用和开发方式，为开发者提供灵活且简洁的使用方式和高阶开发接口。

方式一：使用已有脚本启动

用户可以直接克隆整个仓库，按照以下步骤即可运行套件中已支持的任意configs模型任务配置文件，方便用户快速使用和开发：

准备工作
- 步骤1：克隆mindformers
```
git clone -b dev https://gitee.com/mindspore/mindformers.git
cd mindformers
```
- 步骤2：准备相应任务的数据集，请参考docs目录下各模型的README.md文档准备相应数据集
- 步骤3：修改配置文件configs/{model_name}/run_{model_name}_***.yaml中数据集路径
- 步骤4：如果要使用分布式训练，则需提前生成RANK_TABLE_FILE
```
# 不包含8本身，生成0~7卡的hccl json文件
python mindformers/tools/hccl_tools.py --device_num [0,8)
```
单卡启动：统一接口启动，根据模型CONFIG完成任意模型的单卡训练、微调、评估、推理流程

# 训练启动，run_status支持train、finetuen、eval、predict四个关键字，以分别完成模型训练、评估、推理功能，默认使用配置文件中的run_mode
python run_mindformer.py --config {CONFIG_PATH} --run_mode {train/finetune/eval/predict}

多卡启动：scripts脚本启动，根据模型CONFIG完成任意模型的单卡/多卡训练、微调、评估、推理流程

# 8卡分布式运行，DEVICE_RANGE = [0,8)，不包含8本身
cd scripts
bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_MODE

常用参数说明

RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件
CONFIG_PATH: 为configs文件夹下面的{model_name}/run_*.yaml配置文件
DEVICE_ID: 为设备卡，范围为0~7
DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身
RUN_MODE: 为任务运行状态，支持关键字train\finetune\eval\predict\export

方式二：调用API启动

详细高阶API使用教程请参考：MindFormers大模型使用教程

准备工作
- 步骤1：安装mindformers
具体安装请参考第二章。
- 步骤2：准备数据
准备相应任务的数据集，请参考docs目录下各模型的README.md文档准备相应数据集。

Trainer快速入门

用户可以通过以上方式安装mindformers库，然后利用Trainer高阶接口执行模型任务的训练、微调、评估、推理功能。

Trainer训练/微调启动

用户可使用Trainer.train或者Trainer.finetune接口完成模型的训练/微调/断点续训。

import mindspore; mindspore.set_context(mode=0, device_id=0)
from mindformers import Trainer

cls_trainer = Trainer(task='image_classification', # 已支持的任务名
                      model='vit_base_p16', # 已支持的模型名
                      train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式
                      eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式
# 示例1：开启训练复现流程
cls_trainer.train()
# 示例2：加载集成的mae权重，开启微调流程
cls_trainer.finetune(finetune_checkpoint='mae_vit_base_p16')
# 示例3：开启断点续训功能
cls_trainer.train(train_checkpoint=True, resume_training=True)

Trainer评估启动

用户可使用Trainer.evaluate接口完成模型的评估流程。

import mindspore; mindspore.set_context(mode=0, device_id=0)
from mindformers import Trainer

cls_trainer = Trainer(task='image_classification', # 已支持的任务名
                      model='vit_base_p16', # 已支持的模型名
                      eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式
# 示例1：开启评估已集成模型权重的复现流程
cls_trainer.evaluate()
# 示例2：开启评估训练得到的最后一个权重
cls_trainer.evaluate(eval_checkpoint=True)
# 示例3：开启评估指定的模型权重
cls_trainer.evaluate(eval_checkpoint='./output/checkpoint/rank_0/mindformers.ckpt')

结果打印示例(已集成的vit_base_p16模型权重评估分数)：
Top1 Accuracy=0.8317

Trainer推理启动

用户可使用Trainer.predict接口完成模型的推理流程。

import mindspore; mindspore.set_context(mode=0, device_id=0)
from mindformers import Trainer

cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16') # 已支持的模型名 input_data = './cat.png' # 一张猫的图片

示例1：指定输入的数据完成模型推理

predict_result_d = cls_trainer.predict(input_data=input_data)

示例2：开启推理（自动加载训练得到的最后一个权重）

predict_result_b = cls_trainer.predict(input_data=input_data, predict_checkpoint=True)

示例3：加载指定的权重以完成推理

predict_result_c = cls_trainer.predict(input_data=input_data, predict_checkpoint='./output/checkpoint/rank_0/mindformers.ckpt') print(predict_result_d)


```text
结果打印示例（已集成的vit_base_p16模型权重推理结果）：
{'label': 'cat', score: 0.99}

pipeline 快速入门

MindFormers套件为用户提供了已集成模型的pipeline推理接口，方便用户体验大模型推理服务。

pipeline 使用

# 以gpt2 small为例
import mindspore; mindspore.set_context(mode=0, device_id=0)
from mindformers.pipeline import pipeline

pipeline_task = pipeline(task="text_generation", model="gpt2")
pipeline_result = pipeline_task("An increasing sequence: one,", do_sample=False, max_length=20)
print(pipeline_result)

结果打印示例（已集成的gpt2模型权重推理结果）：
[{'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight,']}]

AutoClass 快速入门

MindFormers套件为用户提供了高阶AutoClass类，包含AutoConfig、AutoModel、AutoProcessor、AutoTokenizer四类，方便开发者进行调用。

AutoConfig 获取已支持的任意模型配置

from mindformers import AutoConfig

# 获取gpt2的模型配置
gpt2_config = AutoConfig.from_pretrained('gpt2')
# 获取vit_base_p16的模型配置
vit_base_p16_config = AutoConfig.from_pretrained('vit_base_p16')

AutoModel 获取已支持的网络模型

from mindformers import AutoModel

# 利用from_pretrained功能实现模型的实例化（默认加载对应权重）
gpt2 = AutoModel.from_pretrained('gpt2')
# 利用from_config功能实现模型的实例化（默认加载对应权重）
gpt2_config = AutoConfig.from_pretrained('gpt2')
gpt2 = AutoModel.from_config(gpt2_config)
# 利用save_pretrained功能保存模型对应配置
gpt2.save_pretrained('./gpt2', save_name='gpt2')

AutoProcessor 获取已支持的预处理方法

from mindformers import AutoProcessor

# 通过模型名关键字获取对应模型预处理过程（实例化gpt2的预处理过程，通常用于Trainer/pipeline推理入参）
gpt2_processor_a = AutoProcessor.from_pretrained('gpt2')
# 通过yaml文件获取相应的预处理过程
gpt2_processor_b = AutoProcessor.from_pretrained('configs/gpt2/run_gpt2.yaml')

AutoTokenizer 获取已支持的tokenizer方法

from mindformers import AutoTokenizer
# 通过模型名关键字获取对应模型预处理过程（实例化gpt2的tokenizer，通常用于Trainer/pipeline推理入参）
gpt2_tokenizer = AutoTokenizer.from_pretrained('gpt2')

五、贡献

欢迎参与社区贡献，可参考MindSpore贡献要求贡献者Wiki。

六、许可证

Apache 2.0许可证