The problem of weights initialization is explained here.
“This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.”
Basically, if done improperly, it would result in serious problems with learning features. This post is intended to provide some simple evidence of the importance of the asymmetry in weights initialization.
Configuration of the neural network:
Predictions if initialized with assymetrical weights:
Predictions if all weights are initialized with 0.1s:
After 10,000 iterations the network failed to solve simple XOR problem — embarrasing, kind of.