For questions about the use of Markov models in the field of AI/ML.
A Markov chain is "a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event."
Questions tagged [markov-chain]
23 questions
9
votes
1 answer
What is ergodicity in a Markov Decision Process (MDP)?
I have read about the concept of ergodicity on the safe RL paper by Moldovan (section 3.2) and the RL book by Sutton (chapter 10.3, 2nd paragraph).
The first one says that "a belief over MDPs is ergodic if and only if any state is reachable from any…
josealeixo.pc
- 225
- 2
- 6
7
votes
2 answers
What is a Markov chain and how can it be used in creating artificial intelligence?
I believe a Markov chain is a sequence of events where each subsequent event depends probabilistically on the current event. What are examples of the application of a Markov chain and can it be used to create artificial intelligence? Would a…
WilliamKF
- 2,533
- 1
- 26
- 31
7
votes
2 answers
What is the difference between a Bayesian Network and a Markov Chain?
I am trying to understand the difference between a Bayesian Network and a Markov Chain.
When I search for this one the web, the unanimous solution seems to be that a Bayesian Network is directional (i.e. it's a DAG) and a Markov Chain is not…
Newskooler
- 173
- 6
5
votes
1 answer
Detect patterns in sequences of actions
I have to analyse sequences of actions that look more or less like this JSON blob. The question I'm trying to answer is whether there are recurring (sub)patterns that different users adopt when asked to perform a certain specific task -- in this…
Morpheu5
- 101
- 4
4
votes
1 answer
Why Are the Standard and Markov Chain Derivations of the Policy Gradient Theorem Equivalent?
While studying the proof of the Policy Gradient Theorem, I have come across two different approaches.
The first seems to be a more standard approach involving "unrolling" across every time step. A good resource discussing this method is section 3.1…
Jamie Stephenson
- 43
- 4
3
votes
2 answers
Can an Markov decision process be dependent on the past?
As far as I know MDP are independent from the past. But the definition says that the same policy should always take the same action depending on the state.
What if I define my state as the current "main" state + previous decisions?
For Example in…
Miemels
- 389
- 2
- 12
3
votes
1 answer
How can I use a Hidden Markov Model to recognize images?
How could I use a 16x16 image as an input in a HMM? And at the same time how would I train it? Can I use backpropagation?
david david
- 212
- 1
- 7
3
votes
2 answers
Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5
Excercise 3.5 The equastions in Section 3.1 are for the continuing
case and need to be modified (very slightly) to apply to episodic
tasks. Show that you know the modifications needed by giving the
modified version of…
Jakub Bielan
- 165
- 4
2
votes
2 answers
Can the limiting distribution depend on the initial distribution?
I am a bit confused about the definition of limiting distribution in Markov chains.
My understanding is that it represents the behavior of the chain in-the-limit. That is, I start from the initial distribution $\mu_0(s)$ and repeat $\mu_1(s) =…
Simon
- 263
- 1
- 8
2
votes
4 answers
Forward Diffusion Process Derivation In Diffusion Models
In papers and other material regarding diffusion models the forward diffusion process is defined by adding a small amount of Gaussian noise to an image $x_0$ for $T$ time steps. In each time step the noise has a variance of $\beta_t$. This process…
nkam
- 23
- 2
2
votes
0 answers
Can $Q$-learning or SARSA be thought of a Markov Chain?
I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot.
Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm to find the optimal policy or the optimal…
dezdichado
- 182
- 8
2
votes
0 answers
Are there any ways to model markov chains from time series data?
I want to make a thing that produces a stochastic process from time series data.
The time series data is recorded every hour over the year, which means 24-hour of patterns exist for 365 days.
What I want to do is something like below:
Fit a…
JH Lee
- 125
- 4
1
vote
2 answers
Why can the function that turns the history into one Markov state be any function?
Summary
In David Silver's RL lecture slides, he defines the State $S_t$ formally as a function of the history:
David then goes on to define the Markov state as any state $S_t$ such that the probability of the next timestep is conditionally…
Andrew
- 11
- 2
1
vote
2 answers
Why are ergodic MDPs also communicating?
An MDP is ergodic if the Markov chain induced by any policy is
ergodic, which means any state is reachable from any other state by
following a suitable policy. [Source]
The part after "which means ..." is the definition of communicating MDP. Why…
Simon
- 263
- 1
- 8
1
vote
0 answers
How does this distribution change in "Understanding Diffusion Models: A Unified Perspective"?
In the paper Understanding Diffusion Models: A Unified Perspective, how did we go from equation $(44)$ to $(45)$? I couldn't find the details in the paper. How does the distribtuion for, the expectation change as marked? $q(x_t | x_{t - 1})$ and…
Harry
- 11
- 2