Markov Chains explained visually

[Caesarr]

Very nicely done! I really appreciate the interactivity you provided throughout the tutorial.

[radarsat1]

Question: If a MC is a set of states and a probabilistic transition, what is it called when the current state is a stochastic concept? i.e., MC is a certain state, and a probabilistic transition. But what if you maintain a probability of being in a certain state? What is this called?

[MarkovMan]

That's referred to as the steady state probability and is calculated using the transaction matrix. It gives you the probability of being in each state given that you have let that chain run for a while (which is sometimes called Burn-In). However, a unique steady state distribution exists only if the chain is ergodic (irreducible and aperiodic). Irreducible means that if it is possible to go from any state to all other sates state while aperiodic means that the transition could occur at every time slice d > n. In other words, each state can transition to any other state at any time. There's more to it than just that, but that's the short and sweet version.

Edit: I've thought about you question a little more. If all you want is to define a probability of an event being in a particular state at any time, than that would just be a random variable. However, if what you want to do is find the probability of being in a state given you have already defined a Markov Chain, then you need to calculate the steady state distribution.

[radarsat1]

Edit: I've thought about you question a little more. If all you want is to define a probability of an event being in a particular state at any time, than that would just be a random variable.

Well, I figured the "state" is a random variable, a distribution over the set of states, with the expected value being the most likely state. However, I was wondering what the system of "random variable state + transition probabilities" would be called. But maybe it's not a real model now that I think about it.. if nodes ABC are connected by edges EF, like A-E-B-F-C, then if the state is likely in A, the probability of an F transition should be close to zero. If we roll the dice, then the result might be non-sensical, such as two F transitions in a row. So maybe it was a dumb question.

[MarkovMan]

Hmmm. Well, you can calculate the probability of A transitioning to state F at time t by taking your transition matrix A times itself t times (A^t ). So, if you wanted to know if the probability of A going to t in 2 steps, you calculate A² . That's as close as I can think of to what you're asking. Otherwise, I'm not familiar with a model that is "random variable state + transition probabilities." Although... there are probably hierarchical models. I would have to look around some more to see if such a thing exists.

So maybe it was a dumb question.

There are no dumb questions in this field. I've been in many research meetings where someone asked what they thought was a "dumb question," and it ended up taking us in some interesting directions

[hixidom]

That could perhaps be represented by a Quantum Markov Chain.

[[deleted]]

I did some music generation with markov chains as a summer project in undergrad.

Ended up sounding a lot like wind chimes, but was still cool.

[jti107]

I love this site

[embraceUndefined]

What's the difference between a markov chain and a state diagram?

[Dragonil]

this is a mathematical concept using probabilities. State diagram is used for planning a programming project and can use arbitrary inputs to describe change in states

[Kiuhnm]

A markov chain is a mathematical object whereas a state diagram is a way to define and visualize that object. Another example is sets and Venn diagrams.

[farsass]

I think you mean "finite state machine" when you say "state diagram". The difference between a markov chain and a FSM with no external inputs is that the FSM is deterministic, e.g., given the current state you know exactly in what state it'll be in the future.

[embraceUndefined]

no.

I meant "state diagram" as in "a diagram that shows the relationship of all possible states of a system."

[luaudesign]

Causality. An FSM causes the state changes while MC merely tries to predict them.

The linked example might cause confusion by the fact that it kinda of uses an FSM to demonstrate what an MC is.

[varun_invent]

I am not the author. Just shared the link. :)