HRL with Timed Subgoals

[pdf][environments][arXiv]

Concurrent learning problem:

• necessary to learn concurrently on all levels
• but changing behaviour of lower level training is a difficult. This is equivalent to an non-stationary environment
• So, higher level won't learn until lower level has converged.

Solution (when subtask is to reach a goal state):

• In hindsight replace the subgoal chosen by higher level (Hindsight action relabling)
• But this makes the environment semi-markov decision process because time taken by lower level to reach the subgoal decreases as training progresses.

Demerit:

• An example: Say the agent has to play tennis, where timing is important, then when lower level policy learns to reach goal state faster, higher level policy has to specify longer sequence of subgoals to maintain correct timing.
• This happened because the environment has dynamic elements beyond the control of the agent (unlike, say, the ant maze problem).

So, the higher level policy is conditioned on the goal state as well as the dynamic component of environment state. That dynamic state is not in control of the policy. But when the action of agents don't alter the environment, or alter it on very few occassions, we can represent that dynamic state with time elapsed. [pg. 5]

So a better solution is to let the higher level choose what goal state to reach as well as when, i.e. timed subgoals.