Reinforcement-Learning-Papers

Reinforcement-Learning-Papers

强化学习前沿论文收录与汇总

该项目全面收录强化学习领域最新论文,聚焦单智能体强化学习研究。内容涵盖模型无关、基于模型、离线、元学习等多个分支,并收录ICLR、ICML、NeurIPS等顶级会议论文。项目对每篇论文进行简要概括,为研究人员提供强化学习前沿进展参考。

强化学习论文集模型无关模型相关离线学习Github开源项目

Reinforcement Learning Papers

PRs Welcome

Related papers for Reinforcement Learning (we mainly focus on single-agent).

Since there are tens of thousands of new papers on reinforcement learning at each conference every year, we are only able to list those we read and consider as insightful.

We have added some ICLR22, ICML22, NeurIPS22, ICLR23, ICML23, NeurIPS23, ICLR24, ICML24 papers on RL

<!-- NeurIPS23 page 71 ICML24 page21-->

Contents

<a id='Model-Free-Online'></a>

Model Free (Online) RL

<!-- ## <span id='Model-Free-Online'>Model Free (Online) RL</span> ### <span id='classic'>Classic Methods</span> -->

<a id='model-free-classic'></a>

Classic Methods

TitleMethodConferenceon/off policyAction SpacePolicyDescription
Human-level control through deep reinforcement learning, [other link]DQNNature15offDiscretebased on value functionuse deep neural network to train q learning and reach the human level in the Atari games; mainly two trick: replay buffer for improving sample efficiency, decouple target network and behavior network
Deep reinforcement learning with double q-learningDouble DQNAAAI16offDiscretebased on value functionfind that the Q function in DQN may overestimate; decouple calculating q function and choosing action with two neural networks
Dueling network architectures for deep reinforcement learningDueling DQNICML16offDiscretebased on value functionuse the same neural network to approximate q function and value function for calculating advantage function
Prioritized Experience ReplayPriority SamplingICLR16offDiscretebased on value functiongive different weights to the samples in the replay buffer (e.g. TD error)
Rainbow: Combining Improvements in Deep Reinforcement LearningRainbowAAAI18offDiscretebased on value functioncombine different improvements to DQN: Double DQN, Dueling DQN, Priority Sampling, Multi-step learning, Distributional RL, Noisy Nets
Policy Gradient Methods for Reinforcement Learning with Function ApproximationPGNeurIPS99on/offContinuous or Discretefunction approximationpropose Policy Gradient Theorem: how to calculate the gradient of the expected cumulative return to policy
----AC/A2C----on/offContinuous or Discreteparameterized neural networkAC: replace the return in PG with q function approximator to reduce variance; A2C: replace the q function in AC with advantage function to reduce variance
Asynchronous Methods for Deep Reinforcement LearningA3CICML16on/offContinuous or Discreteparameterized neural networkpropose three tricks to improve performance: (i) use different agents to interact with the environment; (ii) value function and policy share network parameters; (iii) modify the loss function (mse of value function + pg loss + policy entropy)
Trust Region Policy OptimizationTRPOICML15onContinuous or Discreteparameterized neural networkintroduce trust region to policy optimization for guaranteed monotonic improvement
Proximal Policy Optimization AlgorithmsPPOarxiv17onContinuous or Discreteparameterized neural networkreplace the hard constraint of TRPO with a penalty by clipping the coefficient
Deterministic Policy Gradient AlgorithmsDPGICML14offContinuousfunction approximationconsider deterministic policy for continuous action space and prove Deterministic Policy Gradient Theorem; use a stochastic behaviour policy for encouraging exploration
Continuous Control with Deep Reinforcement LearningDDPGICLR16offContinuousparameterized neural networkadapt the ideas of DQN to DPG: (i) deep neural network function approximators, (ii) replay buffer, (iii) fix the target q function at each epoch
Addressing Function Approximation Error in Actor-Critic MethodsTD3ICML18offContinuousparameterized neural networkadapt the ideas of Double DQN to DDPG: taking the minimum value between a pair of critics to limit overestimation
Reinforcement Learning with Deep Energy-Based PoliciesSQLICML17offmain for Continuousparameterized neural networkconsider max-entropy rl and propose soft q iteration as well as soft q learning
Soft Actor-Critic Algorithms and Applications, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, [appendix]SACICML18offmain for Continuousparameterized neural networkbase the theoretical analysis of SQL and extend soft q iteration (soft q evaluation + soft q improvement); reparameterize the policy and use two parameterized value functions; propose SAC

<a id='exploration'></a>

Exploration

TitleMethodConferenceDescription
Curiosity-driven Exploration by Self-supervised PredictionICMICML17propose that curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills when rewards are sparse; formulate curiosity as the error in an agent’s ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model
Curiosity-Driven Exploration via Latent Bayesian SurpriseLBSAAAI22apply Bayesian surprise in a latent space representing the agent’s current understanding of the dynamics of the system
Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement LearningAIRSICML23select shaping function from a predefined set based on the estimated task return in real-time, providing reliable exploration incentives and alleviating the biased objective problem; develop a toolkit that provides highquality implementations of various intrinsic reward modules based on PyTorch
Curiosity in Hindsight: Intrinsic Exploration in Stochastic EnvironmentsCuriosity in HindsightICML23consider exploration in stochastic environments; learn representations of the future that capture precisely the unpredictable aspects of each outcome—which we use as additional input for predictions, such that intrinsic rewards only reflect the predictable aspects of world dynamics
Maximize to Explore: One Objective Function Fusing Estimation, Planning, and ExplorationNeurIPS23 spotlight
MIMEx: Intrinsic Rewards from Masked Input ModelingMIMExNeurIPS23propose that the mask distribution can be flexibly tuned to control the difficulty of the underlying conditional prediction task
<!--<a id='off-policy-evaluation'></a> ### Off-Policy Evaluation | Title | Method | Conference | Description | | ---- | ---- | ---- | ---- | | [Weighted importance sampling for off-policy learning with linear function approximation](https://proceedings.neurips.cc/paper/2014/file/be53ee61104935234b174e62a07e53cf-Paper.pdf) | WIS-LSTD | NeurIPS14 | | | [Importance Sampling Policy Evaluation with an Estimated Behavior Policy](https://arxiv.org/pdf/1806.01347.pdf) | RIS | ICML19 | | | [On the Reuse Bias in Off-Policy Reinforcement Learning](https://arxiv.org/pdf/2209.07074.pdf) | BIRIS | IJCAI23 | discuss the bias of off-policy evaluation due to reusing the replay buffer; derive a high-probability bound of the Reuse Bias; introduce the concept of stability for off-policy algorithms and provide an upper bound for stable off-policy algorithms | <a id='soft-rl'></a> ### Soft RL | Title | Method | Conference | Description | | ---- | ---- | ---- | ---- | | [A Max-Min Entropy Framework for Reinforcement Learning](https://arxiv.org/pdf/2106.10517.pdf) | MME | NeurIPS21 | find that SAC may fail in explore states with low entropy (arrive states with high entropy and increase their entropies); propose a max-min entropy framework to address this issue | | [Maximum Entropy RL (Provably) Solves Some Robust RL Problems ](https://arxiv.org/pdf/2103.06257.pdf) | ---- | ICLR22 | theoretically prove that standard maximum entropy RL is robust to some disturbances in the dynamics and the reward function | <a id='data-augmentation'></a> ### Data Augmentation | Title | Method | Conference | Description | | ---- | ---- | ---- | ---- | | [Reinforcement Learning with Augmented Data](https://arxiv.org/pdf/2004.14990.pdf) | RAD | NeurIPS20 | propose first extensive study of general data augmentations for RL on both pixel-based and state-based inputs | | [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://arxiv.org/pdf/2004.13649.pdf) | DrQ | ICLR21 Spotlight | propose to regularize the value function when applying data augmentation with model-free methods and reach state-of-the-art performance in image-pixels tasks | --> <!-- | [Equivalence notions and model minimization in Markov decision processes](https://www.ics.uci.edu/~dechter/courses/ics-295/winter-2018/papers/givan-dean-greig.pdf) | | Artificial Intelligence, 2003 | | | [Metrics for Finite Markov Decision Processes](https://arxiv.org/ftp/arxiv/papers/1207/1207.4114.pdf) || UAI04 || | [Bisimulation metrics for continuous Markov decision processes](https://www.normferns.com/assets/documents/sicomp2011.pdf) || SIAM Journal on Computing, 2011 || | [Scalable methods for computing state similarity in deterministic Markov Decision Processes](https://arxiv.org/pdf/1911.09291.pdf) || AAAI20 || | [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://arxiv.org/pdf/2006.10742.pdf) | DBC | ICLR21 || -->

<a id='Representation-RL'></a>

Representation Learning

Note: representation learning with MBRL is in the part World Models

TitleMethodConferenceDescription
CURL: Contrastive Unsupervised Representations for Reinforcement LearningCURLICML20extracts high-level features from raw pixels using contrastive learning and performs offpolicy control on top of the extracted features
Learning Invariant Representations for Reinforcement Learning without ReconstructionDBCICLR21propose using Bisimulation to learn robust latent representations which encode only the task-relevant information from observations
Reinforcement Learning with Prototypical RepresentationsProto-RLICML21pre-train task-agnostic representations and prototypes on environments without downstream task information
Understanding the World Through Action----CoRL21discusse how self-supervised reinforcement learning combined with offline RL can enable scalable representation learning
Flow-based Recurrent Belief State Learning for POMDPsFORBESICML22incorporate normalizing flows into the variational inference to learn general continuous belief states for POMDPs
Contrastive Learning as Goal-Conditioned Reinforcement LearningContrastive RLNeurIPS22show (contrastive) representation learning methods can be cast as RL algorithms in their own right
Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?----NeurIPS22conduct an extensive comparison of various self-supervised losses under the existing joint learning framework for pixel-based reinforcement learning in many environments from different benchmarks, including one real-world environment
Reinforcement Learning with Automated Auxiliary Loss SearchA2LSNeurIPS22propose to automatically search top-performing auxiliary loss functions for learning better representations in RL; define a general auxiliary loss space of size 7.5 × 1020 based on the collected trajectory data and explore the space with an efficient evolutionary search strategy
Mask-based Latent Reconstruction for Reinforcement LearningMLRNeurIPS22propose an effective self-supervised method to predict complete state representations in the latent space from the observations with spatially and temporally masked pixels
Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingVIPICLR23 Spotlightcast representation learning from human videos as an offline goal-conditioned reinforcement learning problem; derive a self-supervised dual goal-conditioned value-function objective that does not depend on actions, enabling pre-training on unlabeled human videos
Latent Variable Representation for Reinforcement Learning----ICLR23provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle in the face of uncertainty for exploration
Spectral Decomposition Representation for Reinforcement LearningICLR23
Become a Proficient Player with Limited Data through Watching Pure VideosFICCICLR23consider the setting where the pre-training data are action-free videos; introduce a two-phase training pipeline; pre-training phase: implicitly extract the hidden action embedding from videos and pre-train the visual representation and the environment dynamics network based on vector quantization; down-stream tasks: finetune with small amount of task data based on the learned models
Bootstrapped Representations in Reinforcement Learning----ICML23provide a theoretical characterization of the state representation learnt by temporal difference learning; find that this representation differs from the features learned by Monte Carlo and residual gradient algorithms for most transition structures of the environment in the policy evaluation setting
[Representation-Driven Reinforcement

编辑推荐精选

商汤小浣熊

商汤小浣熊

最强AI数据分析助手

小浣熊家族Raccoon,您的AI智能助手,致力于通过先进的人工智能技术,为用户提供高效、便捷的智能服务。无论是日常咨询还是专业问题解答,小浣熊都能以快速、准确的响应满足您的需求,让您的生活更加智能便捷。

imini AI

imini AI

像人一样思考的AI智能体

imini 是一款超级AI智能体,能根据人类指令,自主思考、自主完成、并且交付结果的AI智能体。

Keevx

Keevx

AI数字人视频创作平台

Keevx 一款开箱即用的AI数字人视频创作平台,广泛适用于电商广告、企业培训与社媒宣传,让全球企业与个人创作者无需拍摄剪辑,就能快速生成多语言、高质量的专业视频。

即梦AI

即梦AI

一站式AI创作平台

提供 AI 驱动的图片、视频生成及数字人等功能,助力创意创作

扣子-AI办公

扣子-AI办公

AI办公助手,复杂任务高效处理

AI办公助手,复杂任务高效处理。办公效率低?扣子空间AI助手支持播客生成、PPT制作、网页开发及报告写作,覆盖科研、商业、舆情等领域的专家Agent 7x24小时响应,生活工作无缝切换,提升50%效率!

TRAE编程

TRAE编程

AI辅助编程,代码自动修复

Trae是一种自适应的集成开发环境(IDE),通过自动化和多元协作改变开发流程。利用Trae,团队能够更快速、精确地编写和部署代码,从而提高编程效率和项目交付速度。Trae具备上下文感知和代码自动完成功能,是提升开发效率的理想工具。

AI工具TraeAI IDE协作生产力转型热门
蛙蛙写作

蛙蛙写作

AI小说写作助手,一站式润色、改写、扩写

蛙蛙写作—国内先进的AI写作平台,涵盖小说、学术、社交媒体等多场景。提供续写、改写、润色等功能,助力创作者高效优化写作流程。界面简洁,功能全面,适合各类写作者提升内容品质和工作效率。

AI辅助写作AI工具蛙蛙写作AI写作工具学术助手办公助手营销助手AI助手
问小白

问小白

全能AI智能助手,随时解答生活与工作的多样问题

问小白,由元石科技研发的AI智能助手,快速准确地解答各种生活和工作问题,包括但不限于搜索、规划和社交互动,帮助用户在日常生活中提高效率,轻松管理个人事务。

热门AI助手AI对话AI工具聊天机器人
Transly

Transly

实时语音翻译/同声传译工具

Transly是一个多场景的AI大语言模型驱动的同声传译、专业翻译助手,它拥有超精准的音频识别翻译能力,几乎零延迟的使用体验和支持多国语言可以让你带它走遍全球,无论你是留学生、商务人士、韩剧美剧爱好者,还是出国游玩、多国会议、跨国追星等等,都可以满足你所有需要同传的场景需求,线上线下通用,扫除语言障碍,让全世界的语言交流不再有国界。

讯飞智文

讯飞智文

一键生成PPT和Word,让学习生活更轻松

讯飞智文是一个利用 AI 技术的项目,能够帮助用户生成 PPT 以及各类文档。无论是商业领域的市场分析报告、年度目标制定,还是学生群体的职业生涯规划、实习避坑指南,亦或是活动策划、旅游攻略等内容,它都能提供支持,帮助用户精准表达,轻松呈现各种信息。

AI办公办公工具AI工具讯飞智文AI在线生成PPTAI撰写助手多语种文档生成AI自动配图热门
下拉加载更多