SVD and NNs

In connection to Neural Networks and SVD there has been some research. Mostly (so far as I have found) in the direction of using SVD to improve certain aspects of NNs:

1. ¹ Represents weight matrix as SVD . This has uses in training RNNs to stabilize gradients,
2. ² Computes initialization for Neural Network Parameters using SVD of data. But this is only for single layer NNs.
3. ³ Uses SVD representation of Weights to train a CNN and get a low rank NNs. Useful for model compression. But uses regularization to ensure orthogonality (using technique from ¹ would have been better)

There is some research regarding computing spectral parameters using Neural Networks too:

1. ⁴ Trains NNs to predict Singular Values. But only singluar values, not vectors. And the experiments are for small matrices.
2. ⁵ Uses Neural Network to find singular values and singular functions of linear operators. In the paper they find orbitals of Hydrogen atoms using this method.

In overall only ⁵ works towards using NNs to find singular vectors.

Other papers:

• Deep learning combined with singular value decomposition to reconstruct databases in fluid dynamics. https://www.sciencedirect.com/science/article/abs/pii/S0957417423024260