Continuous-Time Markov Chain

A continuous-time Markov chain (CTMC) is a stochastic process $\{X_t{\}}_{t\ge 0}$ on a countable state space satisfying the Markov property in continuous time: $P(X_{t+s}=j\mid X_t=i,\{X_u{\}}_{u<t})=P(X_{t+s}=j\mid X_t=i)$. Its dynamics are fully described by a generator matrix (or rate matrix) $Q$, where off-diagonal entries $q_{ij}\ge 0$ give the instantaneous transition rate from $i$ to $j$, and diagonal entries $q_{ii}=-{\sum }_{j\ne i}{q}_{ij}$ ensure each row sums to zero. Equivalently, the chain holds in state $i$ for a duration that is exponentially distributed with rate $-q_{ii}={\sum }_{j\ne i}{q}_{ij}$ (the memoryless property of the exponential distribution matches the chain's Markov property), then jumps to state $j\ne i$ with probability $q_{ij}/(-q_{ii})$. The state's probability distribution over time, $p_j(t)=P(X_t=j)$, evolves according to the Kolmogorov forward equation $\frac{d}{dt}\mathbf{p}(t)=\mathbf{p}(t)Q$, the continuous-time analogue of the discrete transition matrix update.

• The exponential distribution's memorylessness is not incidental, it is the only continuous distribution with that property, and the entire CTMC framework depends on holding times being memoryless for the Markov property to hold in continuous time.
• The generator matrix $Q$ plays the role for CTMCs that the transition matrix $P$ plays for discrete-time chains, but it holds instantaneous rates rather than one-step probabilities, and the relationship between them is $P(t)=e^{Qt}$ (matrix exponential), the transition probabilities over an elapsed time $t$.
• Birth-death processes, the M/M/1 queue, and continuous-time epidemic models (like the SIR model in its stochastic form) are all specific, structured examples of CTMCs, built by restricting which entries of $Q$ can be nonzero.