Abstract:
A continuous-time bivariate Markov chain comprises a pair of continuous-time random
processes which are jointly Markov. One of the two processes is an underlying process while
the other is assumed observable. An important special bivariate Markov chain is given by
the continuous-time Markov modulated Poisson process (MMPP). The underlying process of
an MMPP is a Markov chain, and the observable process is conditionally Poisson. Discretetime
bivariate Markov chains may also be de ned, but they shall not be studied in this
thesis. Bivariate Markov chains are useful in modeling ion-channel currents in living cells,
Internet tra c, and in other problems in queuing theory. In this thesis we focus on recursive
estimation of the parameter of a bivariate Markov chain which comprises its in nitesimal
generator. We study a stochastic approximation approach using a newly developed recursion
for the gradient of the log-likelihood function of the observed signal. The recursive algorithm
is compared with a batch expectation-maximization (EM) algorithm developed earlier. The
bias and mean squared error obtained in estimating each component of the parameter using
each algorithm are evaluated and compared. The recursive algorithm requires far more
data to provide an estimate comparable to that obtained by the EM algorithm, but the
EM algorithm iterates over the entire data multiple times. The main advantage of the recursive estimator is its ability to adapt to slow changes in the underlying statistics of the
model. The EM algorithm is a batch approach which must be re-applied whenever new
data becomes available or there is a change in the underlying statistics of the model.
