Reinforcement Learning

• Is DQN True Gradient Descent? No. They are approximations to it. CS234 - Lecture 6 (t=3253)

  Because it is not a derivative of a loss function.

• GTD (Gradient Temporal Difference) are true gradient descent
• The first section of chapter on Function Approximation (Sutton & Barto) has few points on this.

From Assumptions of Decision Making Models in AGI:

It is unreasonable to assume that at any state, all possible actions are listed.

• Actions in small scale may be discrete or a finite collection of distributions, but at the level where planning happens set of all possible actions is infinite.

  Such actions can in principle to thought to be recursively composed of a set of basic operations/actions. But the decision making happens not at those basic actions but at level of composed actions.

• It is a task in itself to know what actions can be taken and what actions should we evaluate.

• Thus decision making involves composing short timestep actions to get longer term action over which planning can be done.

  i.e. decision making is often not about selection but selective composition. [Page 2]

So, one thing to explore would be Hierarchical Reinforcement Learning.

What does this even mean?

Is it even RL if there are no rewards. So, this must be limited to improving performance in downstream tasks that do indeed have rewards?

What is the utility?

• helps when the reward is sparse
• skills developed without reward can act as primitive for hierarchical RL

Work done so far:

• ICVF
• Diversity is all you need.

Action chunking works good in imitation learning. So, why not apply it to RL. [twitter]

\begin{align*} Q(s_t, a_{t:t+h}) = \sum_{t'=t}^{t+h-1} \gamma^{t'-t} r_{t'} + \gamma^h Q(s_{t+h}, a_{t+h:t+2h}) \end{align*}

This is n-step returns but the use of n-step Q-value fixes the problem of being biased.

We also need to make the policy match the offline data i.e. impose behavior constraint.

Paper: Horizon Reduction Makes RL Scalable [twitter][arXiv]

If we scale the data and compute for offline RL, the performance doesn't improve except for when you do Horizon reduction.

Using n-step returns improved "asymptotic" performance, and full hierarchical RL scaled even better.

Why does simple TD learning not scale? Because prediction targets are biased and biases accumulate over the horizon. And this bias is not easily mitigated by scaling data and compute.

Handling complex behaviour distribution requires more expressive policy class like diffusion/flow policies. [twitter]

Method: LEAST (Learn to Stop)

Paper: "The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep RL" [twitter] [arXiv]

RL agents need to finish episodes even if they start bad. This wastes time, fills replay buffer low quality data and thus slows learning.

To decide whether to stop or not it builds adaptive thresholds using:

• Median Q-values (for episode quality)
• Median gradient magnitudes (for learning potential)

Dual-criteria stopping outperforms simpler rules based on Q-values alone

This method can be used in conjunction with other methods (TD3, SAC, REDQ, …) and improves them.

Side benefit:

• Improves plasticity of network (i.e. network learns for longer) thus helping on long horizon tasks

Paper: General agents need world model [twitter][arXiv]

Is world model necessary for human level AI or is there a model-free shortcut? It turns out that agents themselves have world model inside them.

We prove that any agent capable of generalizing to a broad range of simple goal-directed tasks must have learned a predictive model capable of simulating its environment. And this model can always be recovered from the agent.

It is possible to extract an approximation to the environemnt transition function from any goal conditioned policy.

Paper: AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models [arXiv]

From verses.ai:

• lab grown brain cells "Dishbrain" learned to play Pong [arXiv]
• using Free Energy Principle and Active Inference
• AXIOM takes inspiration from that appraoch

This alternative path to applying neuroscience-based methodologies and biologically plausible techniques to tackle Atari began in 2022, when Professor Friston and our colleagues from Cortical Labs demonstrated how lab-grown brain cells, called “Dishbrain”, learned to play Pong demonstrating that neurons apply the Free Energy Principle and operate using Active Inference. A 2023 paper published in Nature confirmed the quantitative predictions of the Free Energy Principle using in vitro networks of rat cortical neurons that perform causal inference. In early 2024 we applied these same underlying Active Inference mechanics demonstrated with Dishbrain to playing Pong purely in software.

Resources:

• Papers by VERSES:
  • AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models [arXiv]
  • Variational Bayes Gaussian Splatting [arXiv]
  • Gradient-free variational learning with conditional mixture networks [arXiv]
• Biological Neurons Compete with Deep Reinforcement Learning in Sample Efficiency in a Simulated Gameworld [arXiv]

Paper: Hadamax Encoding: Elevating Performance in Model-Free Atari. [twitter][arXiv]

By just changing the encoder improved the performance in model free atari problem. E.g. C51 showed 60% improvement.

The most important design choices for encoder were:

1. Convolutional Hadamard Representations,
2. Max-pooling instead of convolutional down-sampling,
3. Gaussian Error Linear Unit activations.

This shows sometimes a lot is to be gained by without changing the algorithm but just by changing the architecture.