**Realistic (& Advanced) Image Synthesis** *Fall 2021* - Instructor: [Krishna Murthy Jatavallabhula](https://krrish94.github.io); Coordinator: [Derek Nowrouzezahrai](https://www.cim.mcgill.ca/~derek) This course presents modern mathematical models of lighting and the algorithms needed to solve them and generate beautiful realistic images. Both traditional numerical methods and modern machine learning-based approaches will be covered. ![](http://www.cim.mcgill.ca/~derek/images/ift6095/shading_course.jpg width="42%" border="2") Administrative Details ======================== The weekly schedule includes two 80-minute lectures on *Tuesdays* and *Thursdays*, from 11:35 am to 12:55 pm in *ENGTR 0100*, and a tutorial/lab session. Tutorial/lab schedules depend on the section you registered in (TBD). ECSE Course Code | Credit Value ---------------------------|--------------------
ECSE 446
3 Credits
ECSE 546
4 Credits
Course Overview ================ ![The effects of different reflectance models](images/ift6095/3statues.png width="40%" border="2") This course dives deep into the math and algorithms behind visual effects in modern *feature films* and *games*. After a quick overview of basic radiometry, we present the mathematical models of appearance and light's propagatation in real environments. We then develop algorithms for generating realistic images. You will code state-of-the-art numerical methods used in the visual effects, video games and machine learning industries. We will guide you through the implementation of rendering systems capable of generating realistic images based on the physics of light. Please refer to the [topics](#CourseSchedule) and [programming assignments](#Assignments) we'll cover in the class, below. Radiometry & Physics-based Shading | Light Transport & Shading Algorithms | Differentiable Rendering and Inverse Problems ----------------------------------------|---------------------------------------|----------------------------------------------- What is light? What are the physical units and intuitive interpretations behind the quantities most commonly used to measure and represent light?

We'll very quickly establish a common language to use when discussing the physics of light. The emphasis here (as with the entire course) is to build an intuition, rather than unnecessarily complex mathematical constructs.| We'll derive a [rendering equation](http://en.wikipedia.org/wiki/Rendering_equation) that governs the equilibrium of light in a scene. We will generate realistic images of 3D worlds by solving this equation.

Specifically, you'll code offline and interactive numerical solvers. The methods we explore also apply to other problems in engineering, physics and statistics: as such, you'll develop a broadly applicable "_numerical toolbox_". | This section of the course will focus on recent advances along the intersections of differentiable programming, graphics, and machine learning.

We will discuss how these emerging tools enable us to tackle various inverse problems in rendering, and more generally graphics pipelines.

_The course syllabus and schedule may be revised during the semester. The latest version is always online here, at_ http://www.cim.mcgill.ca/~derek/ecse446.html Requirements ================================================================================ We outline some high-level requirements and expectations, below. Prerequisites (only enforced for ECSE students) --------------------------------------------------- * [*ECSE 202*: Introduction to Software Development](https://www.mcgill.ca/study/2018-2019/courses/ecse-202) * [*ECSE 205*: Probability and Statistics for Engineers](https://www.mcgill.ca/study/2018-2019/courses/ecse-205) and * [*COMP 250*: Introduction to Computer Science](https://www.mcgill.ca/study/2018-2019/courses/comp-250) Generally speaking, we expect students to be comfortable with basic probability concepts (discrete and continuous), differential calculus and linear algebra. You'll be coding quite a bit in this course, so comfort with the build-implement-debug workflow, data structures and algorithms, and basic software engineering will come in handy. Textbook (available online: [HTML](http://www.pbr-book.org) and [PDF](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering).) --------------------------------------------------- ![Optional Textbook](images/ift6095/pbrt3e.jpg width="25%") *Physically Based Rendering: From Theory to Implementation* (_3rd ed._)
by Pharr, Humphreys and Jakob - This text is *the* comprehensive reference on realistic image synthesis. The book is packaged with its own open-source renderer (PBRT), and this implementation is exposed in a literate programming style throughout the book.

At over 1500 pages, we only expect students to refer to this book as a supplemental resource for topics covered in class. You can access an e-publication version online for free in a [web-friendly format](http://www.pbr-book.org) and through [the McGill library network](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering). !!! Tip You don't need to buy the textbook to be successful in the course, but it can help. Leveraging *lecture* and *tutorial* time (i.e., consistent attendance, taking notes and asking questions) correlates much more with success than reading the text. Equipment --------------------------------------------------- You do not need to buy special computing equipment for this class. All of our assignments will also be handed out as Jupyter-like notebooks and can be run on cloud services such as [Google Colab](https://colab.research.google.com/). Time --------------------------------------------------- We *strongly encourage* attending every lecture. Plan to invest *9 hours per week* for the 3-credit course or *12 hours per week* for the 4-credit version (this includes lecture & tutorial time.) For assignments, it's reasonable to spend roughly *65% of your time coding* -- the remainder will be split between planning, debugging and testing. Start early and schedule appropriately. Attending tutorials with your prepared questions is a good way to skirt avoidable difficulties. Evaluation ===================================================================== ECSE 446 students are evaluated based on four (4) programming [assignments](#assignments), to be completed _individually_. ECSE 546 students have an additional programming assignment (i.e., a total of five (5) assignments). Your score relies on the successful completion of the programming tasks, each described below. There is *no midterm or final exam*. A detailed breakdown of each of these grading components follows. Coding Assignments ------------------------------------------------------------ ECSE 446: 4 assignments, each contributing 25% of the final grade
ECSE 546: 5 assignments, each contributing 20% of the final grade Assignments will primarily use the [Python](https://www.python.org/) programming language. We recommend using [Jupyter notebooks](https://jupyter.org/) or [Google Colab](https://colab.research.google.com/). Assignments are to be completed individually.

(ECSE 446 / ECSE 546)
Due Date
-------------------------------|-------------------------|----------------- *Assignment 1* --
Rendering - primary visibility and systems |
25% / 20%
| Start Week #5
[October 4] *Assignment 2* --
Monte Carlo Direct Illumination |
25% / 20%
| Start Week #7
[October 18] *Assignment 3* --
Advanced Monte Carlo and Path Tracing |
25% / 20%
| Start Week #9
[November 1] *Assignment 4* --
Differentiable Shading |
25% / 20%
| Start Week #12
[November 22] *Assignment 5* --
Advanced Differentiable Rendering (for ECSE 546 students only) |
N/A / 20%
| End of Semester
[December 21]
!!! ERROR: Late Policy: don't submit late work Failure to submit a (valid) assignment on time will result in a *score of 0* on the assignment. We will not grant extensions. Exceptional circumstances will be treated as specified in [McGill's Policies on Student Rights and Responsibilities](https://www.mcgill.ca/students/srr/). The assignment handouts -- which will be posted on myCourses during the semester -- detail the required tasks and submission procedures. Assignments must be handed in *no later than 11:59pm EST* on their due date. Extensions, Solutions and Absences ------------------------------------------------------------ Due to the cumulative nature of the coding assignments, it is essential that everyone begin assignments early, review their corrected assignments and -- when necessary -- track down and correct bugs in old assignments. We will **not provide source code solutions** for assignments -- it's your responsibility to make sure your code is correct, consulting with the TAs **during the tutorial/lab sessions**, as necessary. As [outlined earlier](#LatePolicy), late work will generally result in a score of 0. Please read the [course schedule](#CourseSchedule) and cross-reference with the [assignment deadlines](#AssignmentDeadlines) to avoid missing deadlines -- especially if you have **planned absences** -- and start work early each week to minimize the impact of **unplanned absences**. !!! ERROR: Online Solutions Do not post solutions online. Refer to the [collaboration & plagiarism policies](#CollaborationPolicy) below. Course Topics & Schedule ===================================================================== The course combines slide-based lectures, hands-on tutorials, and take-home programming assignments. Attendance and in-class participation tend to correlate with success on assignments and the course. The course will roughly follow the schedule below, modulo adjustments during the semester: !!! Tip **Office hours** are held **in (or just outside) the classroom** immediately after lectures.
: Welcome & Systems I - Course administration, outline and schedule - An overview of the graphics pipeline : Systems II -- Programmable Rasterization - Rasterization, Z-buffer and the programmable pipeline - Rays, intersections, parametric surfaces - Spatial-hierarchy accleration : Phenomenological and Physically-based Shading - Phenomenological reflectance and emission models - Radiometry 101 - Bi-directional reflectance distribution functions (BRDFs) - Physically-based reflectance and emission models - The surface reflection equation : Analytic Solutions - Unshadowed direct illumination - Diffuse Reflection from Point Lights - Diffuse Reflection from Polygonal Area Lights - Diffuse Reflection from Environmental Lights - Glossy Reflection : Monte Carlo Methods I - Monte Carlo Integration - Basic estimator design - Variance reduction - Importance sampling : Monte Carlo Methods II - Monte Carlo Estimators for Direct Illumination - Hemispherical and surface-area parameterizations - Case study: Ambient Occlusion & Importance Sampling : Direct Illumination I - Light Importance Sampling - Spherical, mesh and environment emitters - BRDF Importance Sampling - Multiple Importance Sampling - Product importance sampling - Hierarchical sample warping : Global Illumination I - Light paths - Heckbert notation - The Rendering Equation - Naive estimator - Geometric fan-out : Global Illumination II - Deriving the (Implicit) Path Tracing Monte Carlo Estimator - Explicit Path Tracing - Russian Roulette Unbiased Path Termination - Challenges of "complex paths" : Advanced Global Illumination I - Duality of Light Transport - The Measurement Equation - Local **and** Global Reciprocity in Light Transport - Applying Duality - Light tracing - The Path Integral Formulation of Light Transport - Solving Integrals in Path Space : Advanced Global Illumination II - The Path Integral Formulation of Light Transport - Monte Carlo Estimators - Multiple Importance Sampling - Bidirectional Path Tracing - Complications -- SDS Paths : Density Estimation I - Motivation -- Spatial Regularization - "Backward" Ray Tracing - Description & Limitations - Photon Mapping - Basic Algorithm - Importance Sampling - Relationship to Bidirectional Techniques - Pros/Cons : Density Estimation II - Improving Basic Photon Mapping - Segmenting Maps and Techniques - Progressive Photon Mapping - Convergence Guarantees - Parallelization (): Thanksgiving -- No Lecture : Intro to differentiable programming and ML - Stochastic gradient descent - Automatic differentiation - Simple regression demo - Multi-layer perceptrons : Differentiable rasterization - Discontinuities: primary visibility problem - Soft rasterization - Differentiable shading : Differentiable ray tracing - Naive differentiable ray tracing - Physically-based differentiable rendering - Speeding up non-differentiable rendering via differentiable rendering : Advanced differentiable rendering I - Differential irradiance - Edge sampling for differentiable path tracing : Denoising using NNs - Content TBD : Differentiable volume rendering - Ray marching - sphere tracing - Implicit differentiable volume rendering : Advanced differentiable rendering II - Implicit function theorem - Radiative backpropagation - Path-space differentiable rendering : Neural coordinate-based representations - Neural radiance fields and friends : Neural scene representations II - Interactive rendering using neural scene representations - Baked NeRF, Kilo NeRF : Guest lecture (TBD) - Content TBD : Guest lecture (TBD) - Content TBD : Open problems - inverse graphics pipelines - Content TBD : Summary and Wrap-up - Content TBD
Collaboration & Plagiarism ===================================================================== *Plagiarism* is an academic offense of misrepresenting authorship. This can result in penalties up to expulsion. It is also possible to plagiarise _your own work_, e.g., by submitting work from another course without proper attribution. **When in doubt, attribute!** We expect you to submit your own work. Assignments are individual tasks. That said, we want to promote an environment where you are comfortable discussing ideas together. **A good rule to follow**: fully understand every solution you submit and only submit code that was written by you. McGill values academic integrity and students should take the time to fully understand the meaning and consequences of cheating, plagiarism and other academic offenses (as defined in the Code of Student Conduct and Disciplinary Procedures -- see [these](www.mcgill.ca/integrity) [two](www.mcgill.ca/students/srr/honest) links). In accordance with article 15 of the Charter of Students' Rights, students may submit any written or programming components in either French or English. If you have a disability, please advise the [Office for Students with Disabilities](www.mcgill.ca/osd) (514-398-6009) as early in the semester as possible. In the event of circumstances beyond our control, the evaluation scheme as set out above may require modification. Additional policies governing academic issues which affect students can be found in the Handbook on Student Rights and Responsibilities.