Initial State of RNN

Question

Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.

Answer 1

Can I initialize the initial state of my RNN to be non-zero? Yes, the initial state / weights of a neural network can be initialized to non-zero values. In fact, the trainable weights can be initialized any way you want.

Exactly how you do this depends on the deep learning package that you are using (if you are using a package). For example, if you are using Keras/TensorFlow, there is built-in functionality to initialize layer weights and set model weights. Here is a nice article demonstrating how this can be done: "How to set custom weights in keras using NumPy array". The article uses random normal initialization, but you can use your desired initial state / weights.