In my QR-DQN application, the resulting quantiles for a state s and action a take the form of the blue line in the figure. The method works well in expected values and trains effectively. However, I know that in my problem the return distribution for (s, a) is multinomial with PMF only at {-2, +2}. Thus, I would expect the quantiles to take the form of the green line.
Is my expectation of the resulting quantiles correct? If yes, why does QR-DDQN behave this way?
