ENGTR 0070 (Monday and Wednesday, 8:30 to 10:00)
Wednesdays, 10:00 to noon, MC 533.
Modelling of stochastic control systems, controlled Markov processes, dynamic programming, imperfect and delayed observations, linear quadratic and Gaussian (LQG) systems, team theory, information structures, static and dynamic teams, dynamic programming for teams,multi-armed bandits.
20% biweekly assignments. Depending on the availability of graders, only a few questions, at random, will be graded. There will be a 10% penalty per day for late submissions.
20% term project. A month long project to be done in groups of two. Present one or two papers on any topic of your interest related to the material covered in class. A 10 to 15 page report and a 10 minute in-class presentations, Apr 2nd, 7th, and 10th.
Kumar and Varaiya, Stochastic Systems: Estimation, Identification, and Adaptive Control, Prentice Hall, 1986.
(The book is out of print, but PDF is available online).
Bertsekas, Dynamic programming and optimal control, vol 1 and 2, Athena Publications, 2005.
This book has perhaps the most comprehensive coverage of different topics in dynamic programming.
Puterman, Markov decision processes: discrete time dynamic programming, Wiley 1994.
This is an excellent source algorithms for solving infinite horizon dynamic programs.
For most of the course, I will loosely follow the notation of Kumar and Varaiya. Many of the examples done in class are taken from Bertsekas. Some material on numerical implementation is from Puterman.
Ross, Introduction to Stochastic Dynamic Programming, Academic Press, 1983.
This is an excellent and gentle introduction to dynamic programming.
Dernardo, Dynamic Programming: Models and Applications, Prentice Hall, 1982.
This is also an excellent and gentle introduction to dynamic programming and its applications to operations research.