5

I'm learning Neural Networks, and everything works as planned but, like humans do, adjusting themselves to learn more efficiently, I'm trying to understand conceptually how one might implement an auto adjusting learning rate for a Neural Network.

I have tried to make it based on error, something like how bigger is error learning rate is getting bigger as well. [Could use some clarification here--not entirely sure what you're saying. If can clarify, I'm happy to clean up the English. -DukeZhou]

*If you want give me an example give it on a C based language or math because I don't have experience with Python or Pascal.

DukeZhou
  • 6,209
  • 5
  • 27
  • 54
Laceanu George
  • 137
  • 4

3 Answers3

3

Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:

  • AdaGrad
  • RMSProp
  • Adam

There are many more variants. You'll need to do some research. Please take a look at:

https://en.wikipedia.org/wiki/Stochastic_gradient_descent

enter image description here

SmallChess
  • 1,421
  • 1
  • 9
  • 14
3

Apart from the already mentioned methods there is one which I came across recently is the Cyclic Learning Rates.

Mainstream methods usually reduce the learning rates monotonically. In most of the cases they work well. Yet they do not do well with handling saddle points and local minima. CLR method claims to solve the problem by effectively handling the case of saddle points.

Ref: https://arxiv.org/abs/1506.01186

Yeshwanth Venkatesha
  • 31
  • 3
0

While adjusting the learning rate during the training is certainly interesting, so is finding of a good initial learning rate, too.

Cyclical Learning Rates for Training Neural Networks, fast.ai (pytorch) implementation.

Here's a good practitioner's overview of learning rate schedules (python, but rather readable and with nice plots).

Xpector
  • 161
  • 3