VGen

VGen is an open-source video synthesis codebase developed by the Tongyi Lab of Alibaba Group, featuring state-of-the-art video generative models. This repository includes implementations of the following methods:

VGen can produce high-quality videos from the input text, images, desired motion, desired subjects, and even the feedback signals provided. It also offers a variety of commonly used video generation tools such as visualization, sampling, training, inference, join training using images and videos, acceleration, and more.

🔥News!!!

[2024.06] We release the code and models of InstructVideo. InstructVideo enables the LoRA fine-tuning and inference in VGen. Feel free to use LoRA fine-tuning for other tasks.
[2024.04] We release the models of DreamVideo and ModelScopeT2V V1.5!!! ModelScopeT2V V1.5 is further fine-tuned on ModelScopeT2V for 365k iterations with more data.
[2024.04] We release the code and models of TF-T2V!
[2024.04] We release the code and models of VideoLCM!
[2024.03] We release the training and inference code of DreamVideo!
[2024.03] We release the code and model of HiGen!!
[2024.01] The gradio demo of I2VGen-XL has been completed in HuggingFace, thanks to our colleague @Wenmeng Zhou and @AK for the support, and welcome to try it out.
[2024.01] We support running the gradio app locally, thanks to our colleague @Wenmeng Zhou for the support and @AK for the suggestion, and welcome to have a try.
[2024.01] Thanks @Chenxi for supporting the running of i2vgen-xl on . Feel free to give it a try.
[2024.01] The gradio demo of I2VGen-XL has been completed in Modelscope, and welcome to try it out.
[2023.12] We have open-sourced the code and models for DreamTalk, which can produce high-quality talking head videos across diverse speaking styles using diffusion models.
[2023.12] We release TF-T2V that can scale up existing video generation techniques using text-free videos, significantly enhancing the performance of both Modelscope-T2V and VideoComposer at the same time.
[2023.12] We updated the codebase to support higher versions of xformer (0.0.22), torch2.0+, and removed the dependency on flash_attn.
[2023.12] We release InstructVideo that can accept human feedback signals to improve VLDM
[2023.12] We release the diffusion based expressive talking head generation DreamTalk
[2023.12] We release the high-efficiency video generation method VideoLCM
[2023.12] We release the code and model of I2VGen-XL and the ModelScope T2V
[2023.12] We release the T2V method HiGen and customizing T2V method DreamVideo.
[2023.12] We write an introduction document for VGen and compare I2VGen-XL with SVD.
[2023.11] We release a high-quality I2VGen-XL model, please refer to the Webpage

TODO

Release the technical papers and webpage of I2VGen-XL
Release the code and pretrained models that can generate 1280x720 videos
Release the code and models of DreamTalk that can generate expressive talking head
Release the code and pretrained models of HumanDiff
Release models optimized specifically for the human body and faces
Updated version can fully maintain the ID and capture large and accurate motions simultaneously
Release other methods and the corresponding models

Preparation

The main features of VGen are as follows:

Expandability, allowing for easy management of your own experiments.
Completeness, encompassing all common components for video generation.
Excellent performance, featuring powerful pre-trained models in multiple tasks.

Installation

conda create -n vgen python=3.8
conda activate vgen
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

You also need to ensure that your system has installed the ffmpeg command. If it is not installed, you can install it using the following command:

sudo apt-get update && apt-get install ffmpeg libsm6 libxext6  -y

Datasets

We have provided a demo dataset that includes images and videos, along with their lists in data.

Please note that the demo images used here are for testing purposes and were not included in the training.

Clone the code

git clone https://github.com/ali-vilab/VGen.git
cd VGen

Getting Started with VGen

(1) Train your text-to-video model

Executing the following command to enable distributed training is as easy as that.

python train_net.py --cfg configs/t2v_train.yaml

In the t2v_train.yaml configuration file, you can specify the data, adjust the video-to-image ratio using frame_lens, and validate your ideas with different Diffusion settings, and so on.

Before the training, you can download any of our open-source models for initialization. Our codebase supports custom initialization and grad_scale settings, all of which are included in the Pretrain item in yaml file.
During the training, you can view the saved models and intermediate inference results in the workspace/experiments/t2v_traindirectory.

After the training is completed, you can perform inference on the model using the following command.

python inference.py --cfg configs/t2v_infer.yaml

Then you can find the videos you generated in the workspace/experiments/test_img_01 directory. For specific configurations such as data, models, seed, etc., please refer to the t2v_infer.yaml file.

If you want to directly load our previously open-sourced Modelscope T2V model, please refer to this link.

(2) Run the I2VGen-XL model

(i) Download model and test data:

!pip install modelscope
from modelscope.hub.snapshot_download import snapshot_download
model_dir = snapshot_download('damo/I2VGen-XL', cache_dir='models/', revision='v1.0.0')

or you can also download it through HuggingFace (https://huggingface.co/damo-vilab/i2vgen-xl):

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/damo-vilab/i2vgen-xl

(ii) Run the following command:

python inference.py --cfg configs/i2vgen_xl_infer.yaml

or you can run:

python inference.py --cfg configs/i2vgen_xl_infer.yaml  test_list_path data/test_list_for_i2vgen.txt test_model models/i2vgen_xl_00854500.pth

The test_list_path represents the input image path and its corresponding caption. Please refer to the specific format and suggestions within demo file data/test_list_for_i2vgen.txt. test_model is the path for loading the model. In a few minutes, you can retrieve the high-definition video you wish to create from the workspace/experiments/test_list_for_i2vgen directory. At present, we find that the current model performs inadequately on anime images and images with a black background due to the lack of relevant training data. We are consistently working to optimize it.

(iii) Run the gradio app locally:

python gradio_app.py

(iv) Run the model on ModelScope and HuggingFace:

<span style="color:red">Due to the compression of our video quality in GIF format, please click 'HRER' below to view the original video.</span>

<center> <table> <center> <tr> <td ><center> <image height="260" src="https://img.alicdn.com/imgextra/i1/O1CN01CCEq7K1ZeLpNQqrWu_!!6000000003219-0-tps-1280-720.jpg"></image> </center></td> <td ><center>  <image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01hIQcvG1spmQMLqBo0_!!6000000005816-1-tps-1280-704.gif"></image> </center></td> </tr> <tr> <td ><center> <p>Input Image</p> </center></td> <td ><center> <p>Click <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/442125067544.mp4">HERE</a> to view the generated video.</p> </center></td> </tr> <tr> <td ><center> <image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01ZXY7UN23K8q4oQ3uG_!!6000000007236-2-tps-1280-720.png"></image> </center></td> <td ><center>  <image height="260" src="https://img.alicdn.com/imgextra/i1/O1CN01iaSiiv1aJZURUEY53_!!6000000003309-1-tps-1280-704.gif"></image> </center></td> </tr> <tr> <td ><center> <p>Input Image</p> </center></td> <td ><center> <p>Click <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441385957074.mp4">HERE</a> to view the generated video.</p> </center></td> </tr> <tr> <td ><center> <image height="260" src="https://img.alicdn.com/imgextra/i3/O1CN01NHpVGl1oat4H54Hjf_!!6000000005242-2-tps-1280-720.png"></image> </center></td> <td ><center>   <image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01DgLj1T240jfpzKoaQ_!!6000000007329-1-tps-1280-704.gif"></image> </center></td> </tr> <tr> <td ><center> <p>Input Image</p> </center></td> <td ><center> <p>Click <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/442102706767.mp4">HERE</a> to view the generated video.</p> </center></td> </tr> <tr> <td ><center> <image height="260"