camel

Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model —— 一个中文低资源的llama+lora方案

| English | 中文 | NOTE&FAQ(Please take a look before using)

camel

This is the repo for the Chinese-Vicuna project, which aims to build and share instruction-following Chinese LLaMA model tuning methods which can be trained on a single Nvidia RTX-2080TI, multi-round chatbot which can be trained on a single Nvidia RTX-3090 with the context len 2048.

Why is it called Vicuna: In view of the successful development of alpaca models such as llama,alpaca,guanaco，We want to train a Chinese small alpaca like Vicuna, small but strong enough !

The advantages of our solution are high parameter efficiency, graphics card friendliness, and easy deployment:

Llama-7B instruction tuning is possible on a 2080Ti (11G) (7b-instruct)
Llama-13B instruction tuning is possible on a 3090 (24G) (13b-instruct)
Llama 7B can be fine-tuned on 3090 even for conversations of 2048 length; Use 50,000 pieces of data to get good results (chatv1)
Llama 7B fine-tuning example on medical and legal domains
Support qlora-4bit which can train Llama 13B on 2080Ti.
Easily deployable on 2080Ti/3090, support multiple-gpu inference, which can reduce VRAM more.

The repo contains:

code for finetune the model
code for generation based on trained model
code for run on CPU (fp16 or int4 is support, in purely C++)
tools to download/convert/quantify original facebook llama.ckpt

This is our instruction demo (with beam-size=4, so you will see 4 process output in the meantime):

https://user-images.githubusercontent.com/72137647/228496412-60043912-f491-430b-848a-599e6edfa5ef.mp4

This is our multi-turn instruction demo (with beam-size=4, so you will see 4 process output in the meantime):

https://user-images.githubusercontent.com/72137647/229739363-1b48f3a9-02a1-46ab-81ee-8c62dc1399b2.mp4

NOTICE!

Before asking questions, take a look at this FAQ first! In the FAQ, you can find how to solve problems may be encountered when installing and using this project.

What‘s New

June, 12, 2023: Release Chinese-Vicuna-4bit andChinese-Vicuna-4bit-11600 which can be continue-finetuned
June, 1, 2023: support for 4bit training + inference, providing a multi-GPU inference interface (NOTICE THAT the environment is different from the original 8bit! Also provides test_tokenizers.py to further check EOS token)
May 17, 2023: Llama 7B fine-tuning example on legal domains, The performance is in here
May 10, 2023: Released chatv1 which have better conversational ability. The performance is in here
May 10, 2023: Released instruct_chat_50k.jsonl which is composed of 30k Chinese sharegpt dataset and 20k alpaca-instruction-Chinese-dataset
April 11, 2023: Released our continuous-finetune on the vertical corpus of Chinese medical quizzes Chinese-Vicuna-medical.Provides examples of vertical corpus training
April 4, 2023: Add performance for 13B, which trains on a single 3090.
April 1, 2023: Add better support for multi-turn chat in chat.py ( Now support 4 generation mode in stream mode/typewriter style: beam search, greedy, sample, beam sample ; We also add cancel button for regeneration )
March 29, 2023: Add more detailed test samples. performance
March 29, 2023: Added breakpoint retraining interface to support continued training of other datasets from our checkpoint
March 29, 2023: Released our new 13B-based lora model
March 28, 2023: Released our model on huggingface
March 27, 2023: Released checkpoint-final for training 3 epochs on belle+guanaco
March 27, 2023: Added multi-round interactive dialog script with alpaca-lora-serve service
March 29, 2023: Added gradio typewriter-like output with beam search, better user interaction support.
March 26, 2023: Provides a quantitative approach
March 24, 2023: Released checkpoint-8000 for training about 1.5 epochs on belle+guanaco（100w data）
March 23, 2023: Released checkpoint-4000 with 50w data training
March 23, 2023: Deploy the code for fine-tuning and inferencing in colab
March 23, 2023: Provides code that can be used for inference in pure c++

Vicuna

what's new
what is the meaning
try on colab
performance
- Checkpoint-4000(Facico/Chinese-Vicuna-lora-7b-0.75epoch-belle-and-guanaco)
- Checkpoint-8000(Facico/Chinese-Vicuna-lora-7b-1.5epoch-belle-and-guanaco)
- Checkpoint-final(Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco) and it is used for multiple rounds of dialogue
What we need?
- code、data、Large Language Model、LORA model、Device
How to use
- Installing、Multi-gpu training、Single-gpu training、Inference and use gradio to generate a web page(Streaming mode+beam search)、 multi-round interaction and use gradio to generate a web page(Streaming mode+beam search)、Streaming mode base on alpaca-lora-serve
inference on CPU with pure C++
More tools，for more details, see tool readme
- ways for faster weight download ( 8MB/s )download_llama.sh
- convert tools between the original facebook checkpoint and huggingface formatconvert_llama.py
- a quantitative approach that requires less than 4G graphics memory for inference
Possible problems encountered
todo
citation

Overview

LLaMA paper: https://arxiv.org/abs/2302.13971v1
Self-Instruct paper: https://arxiv.org/abs/2212.10560
data generation: https://github.com/LianjiaTech/BELLE and https://guanaco-model.github.io/
the first work: https://github.com/tatsu-lab/stanford_alpaca

We currently select the combination of BELLE and Guanaco data as our main training dataset. We will train on multi-turn instruction data.

What is the meaning?

Similar to the explosion of the stable diffusion model, platforms like civitai have emerged, consisting of a base model + various LORA models in an open source community.

The repo hopes to help you to train these LORA models.

what is LORA?: Simply, it's a plugin used to help adapt large models to your dataset, technical details can be found inLoRA: Low-Rank Adaptation of Large Language Models。Its advantage is that the finetune is very fast when it comes to getting a small model, about 30M, and the key is that it supports Plug and Play. As can be expected, this is a very suitable architecture for open source ecology.

Here, we will help you train through a very low-demand environment, with only one 2080 (11G) to achieve certain results.

Try on colab

colab link	Descriptions
	Use the specified lora model to inference and generate a webui
	Finetune with the data we collected
	Provide a webui for multi-round dialogue interaction

Performance

7B performance can be seen here, 13B is here (which train on a single 3090). There are also people reporting that our method works on 33B and gains good result.

history-checkpoint performance can be seen here

<details> <summary><b>checkpoint-4000(7B)</b></summary>

Human	Chinese-Vicuna
生成一份python的排序代码	A: def sort_list(lst):<br/> lst.sort()<br/> return lst
自然语言处理是什么	自然语言处理是一种计算机科学领域，旨在使计算机能够理解和处理自然语言。
能给我讲一段笑话吗	为什么猫吃鱼？因为它们喜欢游泳。
将下面文字翻译成英文。猫喜欢吃鱼，所以它们会游泳	Cats like to eat fish, so they will swim.
233和122的和等于多少	233和122的和等于355。
华为是一家怎样的公司	华为是一家全球范围内最大的移动通信设备制造商之一，以及一家全球范围内最大的移动通信设备制造商之一。华为公司的业务涵盖包括移动通信设备、互联网设备、智能家居设备等。
我想和女朋友在北京约会，能推荐几个约会的地方吗	北京有很多约会的地方，可以推荐以下几个约会的地方: <br/><br/>1. 北京艺术博物馆<br/>2. 颐和园<br/>3. 北京故宫博物馆<br/>4. 北京天安门广场<br/>5. 颐和园风景区