Training Recurrent Nets is Optimization Over Programs

• If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs.

  RNNs combine the input vector with their state vector with a fixed (but learned) function to produce a new state vector. This can in programming terms be interpreted as running a fixed program with certain inputs and some internal variables. Viewed this way, RNNs essentially describe programs.

  In fact, it is known that RNNs are Turing-Complete in the sense that they can to simulate arbitrary programs (with proper weights).

  But similar to universal approximation theorems for neural nets you shouldn’t read too much into this.

  [from The Unreasonable Effectiveness of Recurrent Neural Networks - Andrej Karpathy]