Why is a large replay buffer inefficient?

Question

Open AI spin up says

... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning.

1. Why would recent data overfit? Is it because they are more correlated to each other so I am training on a bunch of similar data?

2. Why is it slow to use much memory (besides the simple reason that I am storing more data). Below are specific doubts I have.

   1. I thought convergence is faster when fitting on uncorrelated data. For example, estimates on mean. Recent data would be more correlated. So why would training on longer data slow down training?

   2. Previous samples doesn't seem to have the notion of becoming outdated. They are still valid samples for evaluating the current loss. So why would they slow down training?

Answer 1

I read the same thing recently, and my interpretation was this:

If you only use the very-most recent data, you will overfit to that and things will break

We'd like to train the network to predict accurately for all states and actions. If we only show it recent data, we're only showing it a subset of the possible states and actions, so it might overfit those and forget how to predict in other parts of the environment.

Similar to training a model to classify images of cats, dogs, and elephants, but only showing it cats and dogs for a while. It can forget what elephants look like.

if you use too much experience, you may slow down your learning.

Specifically, I believe, when encountering novel data. With more memory, the novel data will be chosen for updates less often. The network will learn more slowly, because it spends much of its time reviewing old data that it has already learned.

The optimal balance feeds the network as much new data as possible while feeding it enough old data so it doesn't forget.