Reinforcement Learning

• Is DQN True Gradient Descent? No. They are approximations to it. CS234 - Lecture 6 (t=3253)

  Because it is not a derivative of a loss function.

• GTD (Gradient Temporal Difference) are true gradient descent
• The first section of chapter on Function Approximation (Sutton & Barto) has few points on this.

From Assumptions of Decision Making Models in AGI:

It is unreasonable to assume that at any state, all possible actions are listed.

• Actions in small scale may be discrete or a finite collection of distributions, but at the level where planning happens set of all possible actions is infinite.

  Such actions can in principle to thought to be recursively composed of a set of basic operations/actions. But the decision making happens not at those basic actions but at level of composed actions.

• It is a task in itself to know what actions can be taken and what actions should we evaluate.

• Thus decision making involves composing short timestep actions to get longer term action over which planning can be done.

  i.e. decision making is often not about selection but selective composition. [Page 2]

So, one thing to explore would be Hierarchical Reinforcement Learning.

What does this even mean?

Is it even RL if there are no rewards. So, this must be limited to improving performance in downstream tasks that do indeed have rewards?

What is the utility?

• helps when the reward is sparse
• skills developed without reward can act as primitive for hierarchical RL

Work done so far:

• ICVF
• Diversity is all you need.

Action chunking works good in imitation learning. So, why not apply it to RL. [twitter]

\begin{align*} Q(s_t, a_{t:t+h}) = \sum_{t'=t}^{t+h-1} \gamma^{t'-t} r_{t'} + \gamma^h Q(s_{t+h}, a_{t+h:t+2h}) \end{align*}

This is n-step returns but the use of n-step Q-value fixes the problem of being biased.

We also need to make the policy match the offline data i.e. impose behavior constraint.

Paper: Horizon Reduction Makes RL Scalable [twitter][arXiv]

If we scale the data and compute for offline RL, the performance doesn't improve except for when you do Horizon reduction.

Using n-step returns improved "asymptotic" performance, and full hierarchical RL scaled even better.

Why does simple TD learning not scale? Because prediction targets are biased and biases accumulate over the horizon. And this bias is not easily mitigated by scaling data and compute.

Handling complex behaviour distribution requires more expressive policy class like diffusion/flow policies. [twitter]

Method: LEAST (Learn to Stop)

Paper: "The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep RL" [twitter] [arXiv]

RL agents need to finish episodes even if they start bad. This wastes time, fills replay buffer low quality data and thus slows learning.

To decide whether to stop or not it builds adaptive thresholds using:

• Median Q-values (for episode quality)
• Median gradient magnitudes (for learning potential)

Dual-criteria stopping outperforms simpler rules based on Q-values alone

This method can be used in conjunction with other methods (TD3, SAC, REDQ, …) and improves them.

Side benefit:

• Improves plasticity of network (i.e. network learns for longer) thus helping on long horizon tasks

Paper: General agents need world model [twitter][arXiv]

Is world model necessary for human level AI or is there a model-free shortcut? It turns out that agents themselves have world model inside them.

We prove that any agent capable of generalizing to a broad range of simple goal-directed tasks must have learned a predictive model capable of simulating its environment. And this model can always be recovered from the agent.

It is possible to extract an approximation to the environemnt transition function from any goal conditioned policy.

Paper: AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models [arXiv]

From verses.ai:

• lab grown brain cells "Dishbrain" learned to play Pong [arXiv] using Free Energy Principle and Active Inference
• AXIOM takes inspiration from that appraoch

This alternative path to applying neuroscience-based methodologies and biologically plausible techniques to tackle Atari began in 2022, when Professor Friston and our colleagues from Cortical Labs demonstrated how lab-grown brain cells, called “Dishbrain”, learned to play Pong demonstrating that neurons apply the Free Energy Principle and operate using Active Inference. A 2023 paper published in Nature confirmed the quantitative predictions of the Free Energy Principle using in vitro networks of rat cortical neurons that perform causal inference. In early 2024 we applied these same underlying Active Inference mechanics demonstrated with Dishbrain to playing Pong purely in software.

Resources:

• Papers by VERSES:
  • AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models [arXiv]
  • Variational Bayes Gaussian Splatting [arXiv]
  • Gradient-free variational learning with conditional mixture networks [arXiv]
• Biological Neurons Compete with Deep Reinforcement Learning in Sample Efficiency in a Simulated Gameworld [arXiv]

Paper: Hadamax Encoding: Elevating Performance in Model-Free Atari. [twitter][arXiv]

By just changing the encoder improved the performance in model free atari problem. E.g. C51 showed 60% improvement.

The most important design choices for encoder were:

1. Convolutional Hadamard Representations,
2. Max-pooling instead of convolutional down-sampling,
3. Gaussian Error Linear Unit activations.

This shows sometimes a lot is to be gained by without changing the algorithm but just by changing the architecture.

[pdf]

Partition the initial state space \(S_0\) into clusters (called contexts) using clustering algorithm (k-means clustering). Then train a different policy (a local policy) for each cluster. This makes the learning problem easier when the initial state space is diverse and wide. When the initial state space is wide, the policy gradient updates have high variance and this affects learning.

Now, to create a global policy (called central policy in the paper) out of the learned policy for each cluster, minimize the KL divergence between the local policy and central policy for the visited trajectories. This step is called supervised distillation, and is done after some steps of local policy updates. After distillation the local policies are initialized to be global policy, and again their training repeats in their context for some steps until next distillation.

All this so far describes the "Unconstrained Divide and Conquer" algorithm. But it is better if the local policies share information with each other during learning. This is done through a divergence constraint (eqn 3) on the local policy that encourages each local policy to be close to other local policies in its own context and to mimic other local policies in their contexts.

This is the final algorithm, called "DnC" in the paper.

Notes:

• [Appendix C] Number of paritions (i.e. contexts) is a trade-off between information sharing and easier learning of local policies.

  When there are many contexts, there is less information sharing between the local policies but learning for each small set of initial states becomes easier.

Algorithm:

Generalization of Physiological Agility and Dexterity via Synergistic Action Representation [Blog][pdf][arXiv][Tutorial Code]

Process:

• Play Phase:
  • Train a NN on a simpler task using RL for some timesteps (e.g. 1.5M)
  • Let the model play for some timesteps (e.g. 20k) and collect actions that give good rewards (say, rewards above 80% percentile)
• Compute SAR:
  • Do n-component PCA on the collected actions
  • Do ICA on the PCA results
  • Do MinMaxScaler

• Train a NN on desired task with a modified action space of original actionspace + N component action space of SAR

  

Benefits:

• Sample efficiency is improved

  

• The trained method generalizes better - in zero shot as well as finetuning
• SAR learned in one task can be used on other different tasks
• The motions obtained were more natural