VSGAN-tensorrt-docker

Repository to use super resolution models and video frame interpolation models and also trying to speed them up with TensorRT. This repository contains the fastest inference code that you can find, at least I am trying to archive that. Not all codes can use TensorRT due to various reasons, but I try to add that if it works. Further model architectures are planned to be added later on.

Table of contents

<!--ts-->

• Usage
• Usage example
• Deduplicated inference
• Shot Boundary Detection
• vs-mlrt (C++ TRT)
  • multi-gpu
• ddfi
• VFR (variable refresh rate)
• Color transfer
• Benchmarks
• License

<!--te-->

---

Currently working networks:

• Rife4 with HolyWu/vs-rife and hzwer/Practical-RIFE (rife4.0 rife4.1 rife4.2 rife4.3 rife4.4 rife4.5 rife4.6 rife4.7.1 rife4.8.1 rife4.9.2 rife4.10.1 rife4.11.1 rife4.12.2 rife4.12 lite rife4.13.2 rife4.13 lite rife4.14 rife4.14 lite rife4.15 rife4.17 rife4.18 rife4.19-beta rife4.20 rife4.21 rife4.22) and v2 with WolframRhodium

• RealCUGAN with bilibili/ailab

• GMFupSS with 98mxr/GMFupSS

• GMFSS_union with HolyWu version

• GMFSS_Fortuna and GMFSS_Fortuna_union with 98mxr/GMFSS_Fortuna

• SRVGGNetCompact with xinntao/Real-ESRGAN

• Model based shot boundary detection with rwightman/pytorch-image-models, snap-research/EfficientFormer (EfficientFormerV2), wentaozhu/AutoShot, abhijay9/ShiftTolerant-LPIPS and photosynthesis-team/piq

Also used:

• TensorRT C++ inference and python script usage with AmusementClub/vs-mlrt
• ddfi with Mr-Z-2697/ddfi-rife (auto dedup-duplication, not an arch)
• custom ffmpeg with styler00dollar/ffmpeg-static-arch-docker
• lsmash with AkarinVS/L-SMASH-Works
• bestsource with vapoursynth/bestsource
• trt precision check and upscale frame skip with mafiosnik777/enhancr

Model	Rife	GMFupSS	GMFSS_union	GMFSS_Fortuna / GMFSS_Fortuna_union
CUDA	yes (4.0-4.12)	yes	yes (vanilla / wgan)	yes (base / union)
TensorRT	yes (4.0-4.22, skipped some lite models)	-	-	-

Further stuff that can use TensorRT via mlrt with onnx is for example Real-ESRGAN / SRVGGNetCompact, SAFMN, DPIR, Waifu2x, real-cugan, apisr, AnimeJaNai, ModernSpanimation and AniScale. Onnx files can be found here.

Some important things:

• If you are on Windows, install all the latest updates first, otherwise wsl won't work properly. 21H2 minimum.
• Do not use webm video, webm is often broken. It can work, but don't complain about broken output afterwards. I would suggest to render webm into mp4 or mkv.
• Only use ffmpeg to determine if video is variable framerate (vfr) or not. Other programs do not seem reliable.
• Processing vfr video is dangerous, but you can try to use fpsnum and fpsden. Either use these params or render the input video into constant framerate (crf).
• Colabs have a weak cpu, you should try x264 with --opencl. (A100 does not support NVENC and such)

<div id='usage'/>

Usage

Get CUDA and latest Nvidia drivers. After that, follow the following steps:

WARNING FOR WINDOWS USERS: Docker Desktop 4.17.1 is broken. I confirmed that 4.25.0 should work. Older tested versions are 4.16.3 or 4.17.0. I would recommend to use 4.25.0. 4.17.1 results in Docker not starting which is mentioned in this issue.

ANOTHER WARNING FOR PEOPLE WITHOUT AVX512: Instead of using styler00dollar/vsgan_tensorrt:latest, which I build with my 7950x and thus with all AVX, use styler00dollar/vsgan_tensorrt:latest_no_avx512 in compose.yaml to avoid Illegal instruction (core dumped) which is mentioned in this issue.

AND AS A FINAL INFO, Error opening input file pipe: IS NOT A REAL ERROR MESSAGE. That means invalid data got piped into ffmpeg and can be piped error messages for example. To see the actual error messages and what got piped, you can use vspipe -c y4m inference.py -.

Quickstart:

# if you have Windows, install Docker Desktop https://www.docker.com/products/docker-desktop/
# if you encounter issues, install one of the following versions:
# 4.16.3: https://desktop.docker.com/win/main/amd64/96739/Docker%20Desktop%20Installer.exe
# 4.17.0: https://desktop.docker.com/win/main/amd64/99724/Docker%20Desktop%20Installer.exe

# if you have Arch, install the following dependencies
yay -S docker nvidia-docker nvidia-container-toolkit docker-compose docker-buildx

# run the docker with docker-compose
# you need to be inside the vsgan folder with cli before running the following step, git clone repo and cd into it
# go into the vsgan folder, inside that folder should be compose.yaml, run this command
# you can adjust folder mounts in the yaml file
docker-compose run --rm vsgan_tensorrt

There are now multiple containers to choose from, if you don't want the default, then edit compose.yaml and set a different tag image: styler00dollar/vsgan_tensorrt:x prior to running docker-compose run --rm vsgan_tensorrt.

• latest: Default docker with everything. Trying to keep everything updated and fixed.
• latest_no_avx512 is for cpus without avx512 support, otherwise it just crashes if you try to run avx512 binaries on cpus without such support. Use this if your cpu does not support all instruction sets.
• minimal: Bare minimum to run ffmpeg, mlrt and a few video readers.

docker image	compressed download	extracted container	short description
styler00dollar/vsgan_tensorrt:latest	9gb	17gb	default latest with trt10.3
styler00dollar/vsgan_tensorrt:latest_no_avx512 (placeholder, need to build and upload later)	?	?	trt10.3 without avx512
styler00dollar/vsgan_tensorrt:trt9.3	8gb	15gb	trt9.3 use bfdb96a with this docker
styler00dollar/vsgan_tensorrt:trt9.3_no_avx512	8gb	15gb	trt9.3 without avx512 use bfdb96a with this docker
styler00dollar/vsgan_tensorrt:minimal	4gb	8gb	trt8.6 + ffmpeg + mlrt + ffms2 + lsmash + bestsource

Piping usage:

# you can use it in various ways, ffmpeg example
vspipe -c y4m inference.py - | ffmpeg -i pipe: example.mkv -y

# example without vspipe
ffmpeg -f vapoursynth -i inference.py example.mkv -y

# example with ffmpeg trt plugin + nvenc
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -vf scale_npp=1280:720,format_cuda=rgbpf32le,tensorrt=my_engine.engine,format_cuda=nv12 -c:v hevc_nvenc -preset lossless output.mkv -y
# example with ffmpeg trt plugin + hwdownload (cpu encoding)
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -vf format_cuda=rgbpf32le,tensorrt=my_engine.engine,format_cuda=nv12,hwdownload,format=nv12 -vcodec ffv1 output.mkv -y

If docker does not want to start, try this before you use docker:

# fixing docker errors
sudo systemctl start docker
sudo chmod 666 /var/run/docker.sock

Linux docker autostart:

sudo systemctl enable --now docker

The following stuff is for people who want to run things from scratch. Manual ways of downloading the docker image:

# Download prebuild image from dockerhub (recommended)
docker pull styler00dollar/vsgan_tensorrt:latest

# if you have `unauthorized: authentication required` problems, download the docker with
git clone https://github.com/NotGlop/docker-drag
cd docker-drag
python docker_pull.py styler00dollar/vsgan_tensorrt:latest
docker load -i