**Realistic (& Advanced) Image Synthesis** *Fall 2024* -- Instructor: [Derek Nowrouzezahrai](https://www.cim.mcgill.ca/~derek) This course presents modern mathematical models of lighting and shading, and the numerical methods and algorithms needed to solve them and generate beautiful realistic images. Technical content will be delivered using a combination of in-class lectures, tutorial/lab sessions, and (optional) supplemental reading. Learning objectives will be solidified and evaluated through `Python` coding assignments (relying heavily on careful vectorization with `numpy`.) Administrative Details ======================== The weekly schedule includes two 80-minute lectures on *Tuesdays* and *Thursdays*, from 1:05 pm to 2:25 pm in *McConnell Engineering Building 11* (ENG MC 11), and a tutorial/lab session. Tutorial/lab sessions (intertwined) depend on the section you registered in -- please refer to Minerva to ensure you attend the right session. ECSE Course Code | Credit Value ---------------------------|--------------------
ECSE 446
|
3 Credits
ECSE 546
|
4 Credits
Course Overview ================ ![The effects of different reflectance models](images/ift6095/3statues.png width="40%" border="2") This course dives deep into the math and algorithms behind visual effects in modern *feature films* and *games*. After an overview of basic radiometry, we present the mathematical models of appearance and light's propagatation in physical environments. We then develop algorithms to help simulate these processes, generating realistic images. You will code state-of-the-art numerical methods used in visual effects, video games and machine learning. During the tutorial/lab sessions, we will direct you through common implementation stumbling points; however, students are those primarily responsible for overcoming the modeling and coding challenges behind the learning objectives. Please refer to the [topics](#CourseSchedule) and [programming assignments](#Assignments) we'll cover in the class, below. __For many students, this course will provide a first exposure to `Python` and -- arguably more importantly -- vectorization using `numpy`. While the teaching staff will provide support during the tutorial/lab sessions, students are expected to commit to (potentially significant) independent at-home coding and debugging efforts during the semester.__ Physics-based Shading | Light Transport Algorithms | ----------------------------------------|---------------------------------------| What is light? What are the physical units and intuitive interpretations behind those quantities most commonly used to measure and represent light?

We'll very quickly establish a common language to use when discussing the physics of light. The emphasis here (as with the entire course) is to build an intuition, rather than unnecessarily complex mathematical constructs.| We'll derive a [rendering equation](http://en.wikipedia.org/wiki/Rendering_equation) that governs the steady-state equilibrium of light energy in a scene. We will generate realistic images of 3D worlds by numerically solving this equation.

Specifically, you'll code numerical solvers based on Monte Carlo integration, the leading family of strategies employed in the graphics industry. The methods we explore also apply to other problems in engineering, physics and statistics: as such, you'll develop a broadly applicable "_numerical toolbox_". |

_The course syllabus and schedule may be revised during the semester. The latest version is always online here, at_ http://www.cim.mcgill.ca/~derek/ecse446.html Requirements ================================================================================ We outline some high-level requirements and expectations, below. Prerequisites (only enforced for ECSE students) --------------------------------------------------- * [*ECSE 202*: Introduction to Software Development](https://www.mcgill.ca/study/2018-2019/courses/ecse-202) * [*ECSE 205*: Probability and Statistics for Engineers](https://www.mcgill.ca/study/2018-2019/courses/ecse-205) and * [*COMP 250*: Introduction to Computer Science](https://www.mcgill.ca/study/2018-2019/courses/comp-250) Generally speaking, we expect students to be comfortable with basic probability concepts (discrete and continuous), differential calculus and linear algebra. You'll be coding quite a bit in this course, so comfort with the build-implement-debug workflow, data structures and algorithms, and basic software engineering will come in handy. The assignments will be implemented in `Python`, relying _heavily_ on `numpy` for efficient vectorized code. A warm up assignment will help onboard you to these concepts, early in the course. Textbook (available online: [HTML](http://www.pbr-book.org) and [PDF](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering).) --------------------------------------------------- ![Textbook](https://www.pbrt.org/images/bookcover-4ed.jpg width="25%") *Physically Based Rendering: From Theory to Implementation* (_4th ed._)
by Pharr, Humphreys and Jakob - This text is *the* comprehensive reference on realistic image synthesis. The book is packaged with its own open-source renderer (PBRT), and this implementation is exposed in a literate programming style throughout the book.

At over 1500 pages, we only expect students to refer to this book as a supplemental resource for topics covered in class. You can access an e-publication version online for free in a [web-friendly format](http://www.pbr-book.org) and through [the McGill library network](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering). !!! Tip Leveraging *lecture* and *tutorial/lab* time (i.e., consistent attendance, taking notes and asking questions) correlates much more with success than relying solely on the text. As alluded to earlier, sticking to a diligent coding routine that does not leave assignment tasks to the last minute is imperative. Equipment --------------------------------------------------- You do not need to buy special computing equipment for this class (i.e., any computer that's less than six years old should suffice). All of our assignments will rely on a baseline `Python` and `numpy` environment and can (optionally) be run on cloud services such as [Google Colab](https://colab.research.google.com/), with minor modification. Time --------------------------------------------------- Attending and participating in lectures is strongly correlated with the successful retention of the learning objectives: we encourage attending every lecture. It is not unreasonable to plan to invest *nine (9) hours per week* for the 3-credit course or *12 hours per week* for the 4-credit version (this includes lecture, tutorial/lab and at-home coding time) on average. For assignments, it's reasonable to spend roughly *70% of your time coding* -- the remainder will be split between planning, debugging and testing. Start early and schedule appropriately. Attending tutorials/labs with your prepared questions is a good way to skirt any avoidable difficulties. Evaluation ===================================================================== Students will be evaluated based on four (4) core programming [assignments](#assignments) and one (1) warm-up assignment, to be completed _individually_. ECSE 546 students will have additional deliverables assigned to them in each of the assignments. Your score relies on the successful completion of the programming tasks, each described below. There is *no midterm or final exam*. A detailed breakdown of each of these graded components follows. Coding Assignments ------------------------------------------------------------ Assignments will primarily use the [`Python`](https://www.python.org/) programming language. Assignments are to be completed individually.

Assignment
|
Weight
|
Due Date
-------------------------------|-------------------------|----------------- *Assignment 0* --
Warm-up -- `Python` and vectorization with `numpy`|
0% + 10% bonus
| Start Week #15
[December 5] *Assignment 1* --
Ray-tracing -- primary visibility and shading |
20%
| Start Week #6
[October 3] *Assignment 2* --
Progressive Monte Carlo Estimation |
25%
| Start Week #9
[October 24] *Assignment 3* --
Advanced Direct Illumination |
25%
| Start Week #12
[November 14] *Assignment 4* --
Path Tracing |
30%
| Start Week #15
[December 5] The assignment handouts -- which will be posted on myCourses during the semester -- detail the required tasks and submission procedures. Assignments must be handed in *no later than 11:59:59pm EST* on their due date. !!! ERROR: Late Policy: Students have __six__ (6) late days that they can use throughout the semester. Assignments handed in after the deadline will deduct from the late day allotment, rounded up to the nearest day. We will not grant extensions beyond the grace period. Once the grace days are exhausted, any subsequent late assignments will receive a grade of 0. Exceptional circumstances will be treated as specified in [McGill's Policies on Student Rights and Responsibilities](https://www.mcgill.ca/students/srr/). Extensions, Solutions and Absences ------------------------------------------------------------ Due to the cumulative nature of the coding assignments, it is essential that everyone begin assignments early, review their corrected assignments and -- when necessary -- track down and correct bugs in old assignments. We will **not provide source code solutions** for assignments -- it's your responsibility to make sure your code is correct, consulting with the TAs **during the tutorial/lab sessions**, as necessary. As [outlined earlier](#LatePolicy) -- once the six grace days are exhausted -- late work will result in a score of 0. Please read the [course schedule](#CourseSchedule) and cross-reference with the [assignment deadlines](#AssignmentDeadlines) to avoid missing deadlines -- especially if you have **planned absences** -- and start work early each week to minimize the impact of **unplanned absences**. !!! ERROR: Online Solutions Do not post solutions online. Refer to the [collaboration & plagiarism policies](#CollaborationPolicy) below. Course Topics & Schedule ===================================================================== The course combines slide-based lectures, hands-on tutorials, and take-home programming assignments. Attendance and in-class participation correlate with success on assignments and in the course. The course will roughly follow the schedule below, modulo adjustments during the semester:
: Welcome - Course administration, outline and schedule : Systems I -- Ray-tracing - Rays, intersections, parametric surfaces - Spatial-hierarchy acceleration : Phenomenological Shading - Phenomenological reflectance and emission models : Appearance Modeling - Radiometry 101 - Bi-directional reflectance distribution functions (BRDFs) : The Reflection Equation - Physically-based reflectance and emission models - The surface reflection equation : Monte Carlo Methods I - Monte Carlo Integration - Basic estimator design : Monte Carlo Methods I - Variance reduction - Importance sampling : Monte Carlo Methods II - Monte Carlo Estimators for Direct Illumination : Monte Carlo Methods II - Hemispherical and surface-area parameterizations - Case study: Ambient Occlusion & Importance Sampling : Direct Illumination I
Assignment 1 Due Date - Light Importance Sampling - Spherical, mesh and environment emitters : Direct Illumination I - BRDF Importance Sampling - Multiple Importance Sampling (): Fall Reading Week -- No Lecture : Direct Illumination I - Product importance sampling - Hierarchical sample warping : Global Illumination I - Light paths - Heckbert notation - The Rendering Equation - Naive estimator - Geometric fan-out : Global Illumination II - Deriving the (Implicit) Path Tracing Monte Carlo Estimator - Explicit Path Tracing - Russian Roulette Unbiased Path Termination - Challenges of "complex paths" : Advanced Global Illumination I
Assignment 2 Due Date - Duality of Light Transport - The Measurement Equation - Local **and** Global Reciprocity in Light Transport - Applying Duality - Light tracing - The Path Integral Formulation of Light Transport - Solving Integrals in Path Space : Advanced Global Illumination II
- The Path Integral Formulation of Light Transport - Monte Carlo Estimators - Multiple Importance Sampling - Bidirectional Path Tracing - Complications -- SDS Paths : Density Estimation I
- Motivation -- Spatial Regularization - "Backward" Ray Tracing - Description & Limitations - Photon Mapping - Basic Algorithm - Importance Sampling - Relationship to Bidirectional Techniques - Pros/Cons : Density Estimation II - Improving Basic Photon Mapping - Segmenting Maps and Techniques : Density Estimation II - Progressive Photon Mapping - Convergence Guarantees - Parallelization : Many Light Techniques - Virtual Point Lights - Scalability : Volumetric Light Transport
Assignment 3 Due Date - Theoretical Framework : Volumetric Light Transport - Radiative Transport Equation : Volumetric Monte Carlo Methods - Volumetric Path Tracing : Volumetric Light Transport -- Many Light Methods - Virtual Beam Lights : Volumetric Light Transport -- Many Light Methods - Virtual Ray Lights (): No Lecture : Systems II -- Programmable Rasterization
Assignment 4 Due Date - (continued) Rasterization, Z-buffer and the programmable pipeline - Rays, intersections, parametric surfaces - Spatial-hierarchy acceleration : Assignment 0 Due Date
Collaboration & Plagiarism ===================================================================== *Plagiarism* is an academic offense of misrepresenting authorship. This can result in penalties up to expulsion. It is also possible to plagiarise _your own work_, e.g., by submitting work from another course without proper attribution. **When in doubt, attribute!** We expect you to submit your own work. Assignments are individual tasks. That said, we want to promote an environment where you are comfortable discussing ideas together. **A good rule to follow**: * fully understand every solution you submit, * only submit code that was written by you, and * don't share your code (whether by file or "by eye"). McGill values academic integrity and students should take the time to fully understand the meaning and consequences of cheating, plagiarism and other academic offenses (as defined in the Code of Student Conduct and Disciplinary Procedures -- see [these](www.mcgill.ca/integrity) [two](www.mcgill.ca/students/srr/honest) links). !!! warning: Computational plagiarism detection tools are employed as part of the evaluation procedure of this course. In accordance with article 15 of the Charter of Students' Rights, students may submit any written or programming components in either French or English. If you have a disability, please advise the [Office for Students with Disabilities](www.mcgill.ca/osd) (514-398-6009) as early in the semester as possible. In the event of circumstances beyond our control, the evaluation scheme as set out above may require modification. Additional policies governing academic issues which affect students can be found in the Handbook on Student Rights and Responsibilities.