DI-engine

<div align="center"> <a href="https://di-engine-docs.readthedocs.io/en/latest/"><img width="1000px" height="auto" src="https://github.com/opendilab/DI-engine-docs/blob/main/source/images/head_image.png"></a> </div>

---

<div align="center"> <a href="https://hellogithub.com/repository/175c1e13739c4e429d0abf2b32ec583d" target="_blank"> <img src="https://api.hellogithub.com/v1/widgets/recommend.svg?rid=175c1e13739c4e429d0abf2b32ec583d&claim_uid=cExIpHuMKdTQ6BW" alt="Featured｜HelloGitHub" style="width: 250px; height: 54px;" width="250" height="54" /> </a> </div> <br>

Updated on 2024.06.27 DI-engine-v0.5.2

Introduction to DI-engine

Documentation | 中文文档 | Tutorials | Feature | Task & Middleware | TreeTensor | Roadmap

DI-engine is a generalized decision intelligence engine for PyTorch and JAX.

It provides python-first and asynchronous-native task and middleware abstractions, and modularly integrates several of the most important decision-making concepts: Env, Policy and Model. Based on the above mechanisms, DI-engine supports various deep reinforcement learning algorithms with superior performance, high efficiency, well-organized documentation and unittest:

• Most basic DRL algorithms: such as DQN, Rainbow, PPO, TD3, SAC, R2D2, IMPALA
• Multi-agent RL algorithms: such as QMIX, WQMIX, MAPPO, HAPPO, ACE
• Imitation learning algorithms (BC/IRL/GAIL): such as GAIL, SQIL, Guided Cost Learning, Implicit BC
• Offline RL algorithms: BCQ, CQL, TD3BC, Decision Transformer, EDAC, Diffuser, Decision Diffuser, SO2
• Model-based RL algorithms: SVG, STEVE, MBPO, DDPPO, DreamerV3
• Exploration algorithms: HER, RND, ICM, NGU
• LLM + RL Algorithms: PPO-max, DPO, PromptPG
• Other algorithms: such as PER, PLR, PCGrad
• MCTS + RL algorithms: AlphaZero, MuZero, please refer to LightZero
• Generative Model + RL algorithms: Diffusion-QL, QGPO, SRPO, please refer to GenerativeRL

DI-engine aims to standardize different Decision Intelligence environments and applications, supporting both academic research and prototype applications. Various training pipelines and customized decision AI applications are also supported:

<details open> <summary>(Click to Collapse)</summary>

• Traditional academic environments

  • DI-zoo: various decision intelligence demonstrations and benchmark environments with DI-engine.

• Tutorial courses

  • PPOxFamily: PPO x Family DRL Tutorial Course

• Real world decision AI applications

  • DI-star: Decision AI in StarCraftII
  • PsyDI: Towards a Multi-Modal and Interactive Chatbot for Psychological Assessments
  • DI-drive: Auto-driving platform
  • DI-sheep: Decision AI in 3 Tiles Game
  • DI-smartcross: Decision AI in Traffic Light Control
  • DI-bioseq: Decision AI in Biological Sequence Prediction and Searching
  • DI-1024: Deep Reinforcement Learning + 1024 Game

• Research paper

  • InterFuser: [CoRL 2022] Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer
  • ACE: [AAAI 2023] ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency
  • GoBigger: [ICLR 2023] Multi-Agent Decision Intelligence Environment
  • DOS: [CVPR 2023] ReasonNet: End-to-End Driving with Temporal and Global Reasoning
  • LightZero: [NeurIPS 2023 Spotlight] A lightweight and efficient MCTS/AlphaZero/MuZero algorithm toolkit
  • SO2: [AAAI 2024] A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  • LMDrive: [CVPR 2024] LMDrive: Closed-Loop End-to-End Driving with Large Language Models
  • SmartRefine: [CVPR 2024] SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction
  • ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze
  • UniZero: Generalized and Efficient Planning with Scalable Latent World Models

• Docs and Tutorials

  • DI-engine-docs: Tutorials, best practice and the API reference.
  • awesome-model-based-RL: A curated list of awesome Model-Based RL resources
  • awesome-exploration-RL: A curated list of awesome exploration RL resources
  • awesome-decision-transformer: A curated list of Decision Transformer resources
  • awesome-RLHF: A curated list of reinforcement learning with human feedback resources
  • awesome-multi-modal-reinforcement-learning: A curated list of Multi-Modal Reinforcement Learning resources
  • awesome-diffusion-model-in-rl: A curated list of Diffusion Model in RL resources
  • awesome-ui-agents: A curated list of of awesome UI agents resources, encompassing Web, App, OS, and beyond
  • awesome-AI-based-protein-design: a collection of research papers for AI-based protein design
  • awesome-end-to-end-autonomous-driving: A curated list of awesome End-to-End Autonomous Driving resources
  • awesome-driving-behavior-prediction: A collection of research papers for Driving Behavior Prediction
  </details>

On the low-level end, DI-engine comes with a set of highly re-usable modules, including RL optimization functions, PyTorch utilities and auxiliary tools.

BTW, DI-engine also has some special system optimization and design for efficient and robust large-scale RL training:

<details close> <summary>(Click for Details)</summary>

• treevalue: Tree-nested data structure
• DI-treetensor: Tree-nested PyTorch tensor Lib
• DI-toolkit: A simple toolkit package for decision intelligence
• DI-orchestrator: RL Kubernetes Custom Resource and Operator Lib
• DI-hpc: RL HPC OP Lib
• DI-store: RL Object Store

</details>

Have fun with exploration and exploitation.

Outline

• Introduction to DI-engine
• Outline
• Installation
• Quick Start
• Feature
  • Algorithm Versatility
  • Environment Versatility
  • General Data Container: TreeTensor
• Feedback and Contribution
• Supporters
  • ↳ Stargazers
  • ↳ Forkers
• Citation
• License

Installation

You can simply install DI-engine from PyPI with the following command:

pip install DI-engine

If you use Anaconda or Miniconda, you can install DI-engine from conda-forge through the following command:

conda install -c opendilab di-engine

For more information about installation, you can refer to installation.

And our dockerhub repo can be found here，we prepare base image and env image with common RL environments.

<details close> <summary>(Click for Details)</summary>

• base: opendilab/ding:nightly
• rpc: opendilab/ding:nightly-rpc
• atari: opendilab/ding:nightly-atari
• mujoco: opendilab/ding:nightly-mujoco
• dmc: opendilab/ding:nightly-dmc2gym
• metaworld: opendilab/ding:nightly-metaworld
• smac: opendilab/ding:nightly-smac
• grf: opendilab/ding:nightly-grf
• cityflow: opendilab/ding:nightly-cityflow
• evogym: opendilab/ding:nightly-evogym
• d4rl: opendilab/ding:nightly-d4rl

</details>

The detailed documentation are hosted on doc | 中文文档.

Quick Start

3 Minutes Kickoff

3 Minutes Kickoff (colab)

DI-engine Huggingface Kickoff (colab)

How to migrate a new RL Env | 如何迁移一个新的强化学习环境

[How to customize the neural network