For questions related to the concept of ergodicity, which occurs, for example, in the context of reinforcement learning and Markov decision processes.
Questions tagged [ergodicity]
4 questions
9
votes
1 answer
What is ergodicity in a Markov Decision Process (MDP)?
I have read about the concept of ergodicity on the safe RL paper by Moldovan (section 3.2) and the RL book by Sutton (chapter 10.3, 2nd paragraph).
The first one says that "a belief over MDPs is ergodic if and only if any state is reachable from any…
josealeixo.pc
- 225
- 2
- 6
2
votes
2 answers
Can the limiting distribution depend on the initial distribution?
I am a bit confused about the definition of limiting distribution in Markov chains.
My understanding is that it represents the behavior of the chain in-the-limit. That is, I start from the initial distribution $\mu_0(s)$ and repeat $\mu_1(s) =…
Simon
- 263
- 1
- 8
2
votes
1 answer
Ergodic MDP: does it have to be aperiodic?
Puterman defines an ergodic MDP as
if the transition matrix corresponding to every deterministic stationary policy consists of a single recurrent class.
If the transition matrix is recurrent, it means that there is one aperiodic communicating…
Simon
- 263
- 1
- 8
1
vote
2 answers
Why are ergodic MDPs also communicating?
An MDP is ergodic if the Markov chain induced by any policy is
ergodic, which means any state is reachable from any other state by
following a suitable policy. [Source]
The part after "which means ..." is the definition of communicating MDP. Why…
Simon
- 263
- 1
- 8