## MATH2750 Introduction to Markov Processes

### 10 creditsClass Size: 200

Taught: Semester 2 (Jan to Jun) View Timetable

Year running 2005/06

### Pre-requisites

MATH1740 or equivalent.

This module is approved as an Elective

### Objectives

To provide a simple introduction to stochastic processes. On completion of this module, students should be able to: (a) have an understanding of, and ability to solve, elementary problems of first passage time distributions; (b) understand about barriers in a random walk; (c) solve equilibrium distribution problems; (d) know the difference between an equilibrium distribution and a stationary distribution; (e) have a knowledge of Markov chains and elementary theory thereof; (f) learn about continuous time Markov process models; (g) have knowledge about the Poisson process; (h) extend the Poisson process model to other simple examples, and solve associated problems; (i) understand the role of forward and backward equations; (j) understand the use of simulation in modelling.

### Syllabus

A stochastic process refers to any quantity which changes randomly in time. The number of people in a queue, the capacity of a reservoir, the size of a population, are all examples from the real world. The linking model for all these examples is the simple random walk. The gambler's ruin problem, met in the first year, is an example of a simple random walk with two absorbing barriers. Replacing these absorbing barriers with reflecting barriers provides a model for reservoir capacity. With appropriate modifications the random walk can be extended to model stochastic processes which change over continuous time, not just at regularly spaced time points. As a birth-death process this can be used to model population growth, the spread of diseases like AIDS, traffic flow, the queuing of students at a coffee bar, and so on. Topics covered include: 1. Background material. Notation, L'Htpital's rule, difference equations, generating functions, pgfs, mgfs. 2. Random walks. Simple random walks - transition probability and first passage time, equilibrium distribution. 3. Markov chains. General theory - Chapman-Kolmogorov equations; two-state Markov chain; classification of states in a Markov chain - decomposition theorem, irreducible Markov chains, equilibrium distribution, stationary distribution. 4. Markov processes in continuous time with discrete state space. General theory - transition rates; Poisson process and other examples. 5. Simulation of stochastic processes. Elementary inference.

### Teaching methods

Due to COVID-19, teaching and assessment activities are being kept under review - see module enrolment pages for information

Lectures (22 hours); examples classes (7 hours) and practicals (4 hours).

### Methods of assessment

Due to COVID-19, teaching and assessment activities are being kept under review - see module enrolment pages for information

2 hour written examination at end of semester (80%), coursework (20%).