Awesome Model-Based Reinforcement Learning

This is a collection of research papers for model-based reinforcement learning (mbrl). And the repository will be continuously updated to track the frontier of model-based rl.

Welcome to follow and star!

<pre name="code" class="html"> <font color="red">[2024.05.20] <b>New: We update the ICML 2024 submissions of model-based rl!</b></font> [2023.11.29] We update the ICLR 2024 submissions of model-based rl. [2023.09.29] We update the NeurIPS 2023 paper list of model-based rl. [2023.06.15] We update the ICML 2023 paper list of model-based rl. [2023.02.05] We update the ICLR 2023 paper list of model-based rl. [2022.11.03] We update the NeurIPS 2022 paper list of model-based rl. [2022.07.06] We update the ICML 2022 paper list of model-based rl. [2022.02.13] We update the ICLR 2022 paper list of model-based rl. [2021.12.28] We release the awesome model-based rl. </pre>

Table of Contents

• Awesome Model-Based Reinforcement Learning
  • Table of Contents
  • A Taxonomy of Model-Based RL Algorithms
  • Papers
    • Classic Model-Based RL Papers
    • ICML 2024
    • ICLR 2024
    • NeurIPS 2023
    • ICML 2023
    • ICLR 2023
    • NeurIPS 2022
    • ICML 2022
    • ICLR 2022
    • NeurIPS 2021
    • ICLR 2021
    • ICML 2021
    • Other
  • Tutorial
  • Codebase
  • Contributing
  • License

A Taxonomy of Model-Based RL Algorithms

We’ll start this section with a disclaimer: it’s really quite hard to draw an accurate, all-encompassing taxonomy of algorithms in the Model-Based RL space, because the modularity of algorithms is not well-represented by a tree structure. So we will publish a series of related blogs to explain more Model-Based RL algorithms.

<p align="center"> <img style="border-radius: 0.3125em; box-shadow: 0 2px 4px 0 rgba(34,36,38,.12),0 2px 10px 0 rgba(34,36,38,.08);" src="./assets/mbrl-taxonomy.png"> <br> <em style="display: inline-block;">A non-exhaustive, but useful taxonomy of algorithms in modern Model-Based RL.</em> </p>

We simply divide Model-Based RL into two categories: Learn the Model and Given the Model.

• Learn the Model mainly focuses on how to build the environment model.

• Given the Model cares about how to utilize the learned model.

And we give some examples as shown in the figure above. There are links to algorithms in taxonomy.

[1] World Models: Ha and Schmidhuber, 2018
[2] I2A (Imagination-Augmented Agents): Weber et al, 2017
[3] MBMF (Model-Based RL with Model-Free Fine-Tuning): Nagabandi et al, 2017
[4] MBVE (Model-Based Value Expansion): Feinberg et al, 2018
[5] ExIt (Expert Iteration): Anthony et al, 2017
[6] AlphaZero: Silver et al, 2017
[7] POPLIN (Model-Based Policy Planning): Wang et al, 2019
[8] M2AC (Masked Model-based Actor-Critic): Pan et al, 2020

Papers

format:
- [title](paper link) [links]
  - author1, author2, and author3
  - Key: key problems and insights
  - OpenReview: optional
  - ExpEnv: experiment environments

Classic Model-Based RL Papers

<details open> <summary>Toggle</summary>

• Dyna, an integrated architecture for learning, planning, and reacting

  • Richard S. Sutton. ACM 1991
  • Key: dyna architecture
  • ExpEnv: None

• PILCO: A Model-Based and Data-Efficient Approach to Policy Search

  • Marc Peter Deisenroth, Carl Edward Rasmussen. ICML 2011
  • Key: probabilistic dynamics model
  • ExpEnv: cart-pole system, robotic unicycle

• Learning Complex Neural Network Policies with Trajectory Optimization

  • Sergey Levine, Vladlen Koltun. ICML 2014
  • Key: guided policy search
  • ExpEnv: mujoco

• Learning Continuous Control Policies by Stochastic Value Gradients

  • Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, Tom Erez. NIPS 2015
  • Key: backpropagation through paths, gradient on real trajectory
  • ExpEnv: mujoco

• Value Prediction Network

  • Junhyuk Oh, Satinder Singh, Honglak Lee. NIPS 2017
  • Key: value-prediction model <!-- VE? -->
  • ExpEnv: collect domain, atari

• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

  • Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee. NIPS 2018
  • Key: ensemble model and Qnet, value expansion
  • ExpEnv: mujoco, roboschool

• Recurrent World Models Facilitate Policy Evolution

  • David Ha, Jürgen Schmidhuber. NIPS 2018
  • Key: vae(representation), rnn(predictive model)
  • ExpEnv: car racing, vizdoom

• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

  • Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine. NIPS 2018
  • Key: probabilistic ensembles with trajectory sampling
  • ExpEnv: cartpole, mujoco

• When to Trust Your Model: Model-Based Policy Optimization

  • Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine. NeurIPS 2019
  • Key: ensemble model, sac, k-branched rollout
  • ExpEnv: mujoco

• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

  • Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, Tengyu Ma. ICLR 2019
  • Key: Discrepancy Bounds Design, ME-TRPO with multi-step, Entropy regularization
  • ExpEnv: mujoco

• Model-Ensemble Trust-Region Policy Optimization

  • Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel. ICLR 2018
  • Key: ensemble model, TRPO
  <!-- - OpenReview: 7, 7, 6 -->
  • ExpEnv: mujoco

• Dream to Control: Learning Behaviors by Latent Imagination

  • Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi. ICLR 2019
  • Key: DreamerV1, latent space imagination
  • ExpEnv: deepmind control suite, atari, deepmind lab

• Exploring Model-based Planning with Policy Networks

  • Tingwu Wang, Jimmy Ba. ICLR 2020
  • Key: model-based policy planning in action space and parameter space
  • ExpEnv: mujoco

• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

  • Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver. Nature 2020
  • Key: MCTS, value equivalence
  • ExpEnv: chess, shogi, go, atari

</details>

ICML 2024

<details open> <summary>Toggle</summary>

• HarmonyDream: Task Harmonization Inside World Models

  • Haoyu Ma, Jialong Wu, Ningya Feng, Chenjun Xiao, Dong Li, Jianye Hao, Jianmin Wang, Mingsheng Long
  • Key: observation modeling and reward modeling analysis in world models
  • ExpEnv: meta-world, rlbench, deepmind control suite, atari 100k

• 3D-VLA: A 3D Vision-Language-Action Generative World Model

  • Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan
  • Key: unify 3D perception, reasoning, and action with a generative world model; create a large-scale 3D embodied instruction tuning dataset
  • ExpEnv: rlbench, calvin

• CompeteAI: Understanding the Competition Behaviors in Large Language Model-based Agents

  • Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, Xing Xie
  • Key: propose a competitive framework for LLM-based agents; build a simulated competitive environment
  • ExpEnv: a virtual town with only restaurants and customers

• Model-based Reinforcement Learning for Parameterized Action Spaces

  • Renhao Zhang, Haotian Fu, Yilin Miao, George Konidaris
  • Key: discrete-continuous hybrid action space, dynamics model with parameterized actions, MPC with parameterized actions
  • ExpEnv: platform, goal, hard goal, catch point, hard move

• Learning Latent Dynamic Robust Representations for World Models

  • Ruixiang Sun, Hongyu Zang, Xin Li, Riashat Islam
  • Key: modified Dreamer architecture, hybrid-recurrent state space model
  • ExpEnv: deepmind control suite, distracted deepmind control suite, mani-skill2

• AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors

  • Yucen Wang, Shenghua Wan, Le Gan, Shuai Feng, De-Chuan Zhan
  • Key: implicit action generator, action-conditioned separated world models
  • ExpEnv: deepmind control suite

• Hieros: Hierarchical Imagination on Structured State Space Sequence World Models

  • Paul Mattes, Rainer Schlosser, Ralf Herbrich
  • Key: state-space models, multilayered hierarchical imagination, S5 based world model
  • ExpEnv: atari 100k

• Improving Token-Based World Models with Parallel Observation Prediction

  • Lior Cohen, Kaixin Wang, Bingyi Kang, Shie Mannor
  • Key: pixel-based mbrl, token-based world models, retentive environment model
  • ExpEnv: atari 100k

• Do Transformer World Models Give Better Policy Gradients?

  • Michel Ma, Tianwei Ni, Clement Gehring, Pierluca D'Oro, Pierre-Luc Bacon
  • Key: actions world model
  • ExpEnv: double-pendulum, Myriad

• Dr. Strategy: Model-Based Generalist Agents with Strategic Dreaming

  • Hany Hamed, Subin Kim, Dongyeong Kim, Jaesik Yoon, Sungjin Ahn
  • Key: during strategeic dreaming, train three policies -- highway policy, explorer policy and achiever policy, and then achieve downstream tasks
  • ExpEnv: 2D Navigation, 3D-Maze Navigation, RoboKitchen

• Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption

  • Chenlu Ye, Jiafan He, Quanquan Gu, Tong