Winter 2014

- Location
ENGTR 0070 (Monday and Wednesday, 8:30 to 10:00)

- Office hours
Wednesdays, 10:00 to noon, MC 533.

- Course Outline
Modelling of stochastic control systems, controlled Markov processes, dynamic programming, imperfect and delayed observations, linear quadratic and Gaussian (LQG) systems, team theory, information structures, static and dynamic teams, dynamic programming for teams,multi-armed bandits.

- Course pre-requisites
- A graduate course on Probability (ECSE 509 or equivalent) is a
*required*pre-requisite. - A graduate course on Stochastic processees (ECSE 510 or equivalent) is a
*recommended*co-requisite.

- A graduate course on Probability (ECSE 509 or equivalent) is a
- Grading policy
**20% biweekly assignments**. Depending on the availability of graders, only a few questions, at random, will be graded. There will be a 10% penalty per day for late submissions.**60% mid-terms**. Two take home mid-terms (4 days each). Feb 12th-17th and March 12th-17th.**20% term project**. A month long project to be done in groups of two. Present one or two papers on any topic of your interest related to the material covered in class. A 10 to 15 page report and a 10 minute in-class presentations, Apr 2nd, 7th, and 10th.

- Textbooks
Kumar and Varaiya,

*Stochastic Systems: Estimation, Identification, and Adaptive Control,*Prentice Hall, 1986.

(The book is out of print, but PDF is available online).Bertsekas,

*Dynamic programming and optimal control*, vol 1 and 2, Athena Publications, 2005.This book has perhaps the most comprehensive coverage of different topics in dynamic programming.

Puterman,

*Markov decision processes: discrete time dynamic programming*, Wiley 1994.This is an excellent source algorithms for solving infinite horizon dynamic programs.

For most of the course, I will loosely follow the notation of Kumar and Varaiya. Many of the examples done in class are taken from Bertsekas. Some material on numerical implementation is from Puterman.

- Reference books
Ross,

*Introduction to Stochastic Dynamic Programming,*Academic Press, 1983.This is an excellent and gentle introduction to dynamic programming.

Dernardo,

*Dynamic Programming: Models and Applications,*Prentice Hall, 1982.This is also an excellent and gentle introduction to dynamic programming and its applications to operations research.

- Similar courses