Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
Asked
Active
Viewed 2,697 times
7
1 Answers
6
First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specifically at the same number.
For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, and you don't want the LR to disappear too brutally.
However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if your ER is initially low), and also to the "learning speed" of your model: if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper here, where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.
