Yicheng Fu

Yicheng Fu

PhD @ ZJU

Reinforcement Learning Notes

less than 1 minute read

Published:

Resources

理论基础

  • 强化学习的数学原理 - 西湖大学赵世钰
  • Reinforcement Learning: An Introduction - Richard Sutton
  • Reinforcement Learning Course - David Silver
  • UIUC CS 598/542 - Nan Jiang
  • CMPUT 605: Theoretical Foundations of Reinforcement Learning W2023 - Csaba Szepesvári

实践基础

  • 动手学强化学习 - 张伟楠
  • 深度强化学习 - 王树森、张志华
  • Spinning Up - OpenAI
  • UCB CS 285 Deep RL - Pieter Abbeel, Sergey Levine

Libs and Envs

OpenAI Gym: 只提供基础环境，不提供算法

Algorithms and Tricks

General Tricks

Action normalization to [-1, 1]

Soft Actor-Critic (SAC)

SAC用Q-function更新critic而不用V-function更新critic：原始版本是用V的，但是现代版本用Q V的学习会引入额外误差，并且会将误差传递到Q更新当中，并且V是通过Q隐式表示的，因此没有必要再学一个V 好处：模型更简单，更新链路更短，减少误差，更容易稳定收敛 [1] Soft Actor-Critic Algorithms and Applications [2] OpenAI SpinningUP SAC: https://spinningup.openai.com/en/latest/algorithms/sac.html [3] CleanRL SAC: https://docs.cleanrl.dev/rl-algorithms/sac/#explanation-of-the-logged-metrics [4] Pytorch SAC: https://github.com/denisyarats/pytorch_sac/blob/master/agent/sac.py [5] 动手学强化学习 SAC: https://hrl.boyuai.com/chapter/2/sac%E7%AE%97%E6%B3%95/