**Realistic (& Advanced) Image Synthesis** *Fall 2019* - [McGill University](http://www.mcgill.ca) -- Prof. [Derek Nowrouzezahrai](http://www.cim.mcgill.ca/~derek) This course will detail modern mathematical models of lighting and shading, as well as exploring methods to solve these models and robust numerical algorithms to generate realistic images. ![](http://www.cim.mcgill.ca/~derek/images/ift6095/shading_course.jpg width="42%" border="2") Administrative Details ======================== The weekly schedule includes two 80-minute lectures on *Mondays* and *Wednesdays*, from 2:35pm to 3:55pm in *WONG 1020*, and a _single_ tutorial session. Tutorial schedules depend on the section you registered for -- tutorials are held on *Mondays*, *Tuesdays*, *Wednesdays* and *Fridays*. Both the 3- and 4-credit variants of this course are cross-listed in ECSE and COMP: ECSE Course Code | COMP Course Code | Credit Value ---------------------------|--------------------|--------------
ECSE 446
COMP 598-008
3 Credits
ECSE 546
COMP 599-003
4 Credits
Course Overview ================ ![The effects of different reflectance models](images/ift6095/3statues.png width="40%" border="2") This course dives deep into the math and algorithms behind visual effects in modern *feature films* and *games*. After a quick overview of basic radiometry, we present the mathematical models of appearance and light's propagatation in real environments. We then develop algorithms for generating realistic images. You will code state-of-the-art numerical methods used in the visual effects, video games and machine learning industries. We will guide you through the implementation of rendering systems capable of generating realistic images based on the physics of light. Please refer to the [topics](#CourseSchedule) and [programming assignments](#Assignments) we'll cover in the class, below. Radiometry & Physics-based Shading | Light Transport & Shading Algorithms ----------------------------------------|--------------------------------------- What is light? What are the physical units and intuitive interpretations behind the quantities most commonly used to measure and represent light?

We'll very quickly establish a common language to use when discussing the physics of light. The emphasis here (as with the entire course) is to build an intuition, rather than unnecessarily complex mathematical constructs.| We'll derive a [rendering equation](http://en.wikipedia.org/wiki/Rendering_equation) that governs the equilibrium of light in a scene. We will generate realistic images of 3D worlds by solving this equation.

Specifically, you'll code offline and interactive numerical solvers. The methods we explore also apply to other problems in engineering, physics and statistics: as such, you'll develop a broadly applicable "_numerical toolbox_". Here are links to two online demos of techniques you'll be learning about and implementing during the course:
Interactive path tracing with editable lights, camera and objects.
Shading a cathedral with dynamic lighting using irradiance volumes. [see blog post for technical details]

_The course syllabus and schedule may be revised during the semester. The latest version is always online here, at_ http://www.cim.mcgill.ca/~derek/ecse446.html Requirements ================================================================================ We outline some high-level requirements and expectations, below. Prerequisites (only enforced for ECSE students) --------------------------------------------------- * [*ECSE 202*: Introduction to Software Development](https://www.mcgill.ca/study/2018-2019/courses/ecse-202) * [*ECSE 205*: Probability and Statistics for Engineers](https://www.mcgill.ca/study/2018-2019/courses/ecse-205) and * [*COMP 250*: Introduction to Computer Science](https://www.mcgill.ca/study/2018-2019/courses/comp-250) Generally speaking, we expect students to be comfortable with basic probability concepts (discrete and continuous), differential calculus and linear algebra. You'll be coding quite a bit in this course, so comfort with the build-implement-debug workflow, data structures and algorithms, and basic software engineering will come in handy. Textbook (available online: [HTML](http://www.pbr-book.org) and [PDF](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering).) --------------------------------------------------- ![Optional Textbook](images/ift6095/pbrt3e.jpg width="25%") *Physically Based Rendering: From Theory to Implementation* (_3rd ed._)
by Pharr, Humphreys and Jakob - This text is *the* comprehensive reference on realistic image synthesis. The book is packaged with its own open-source renderer (PBRT), and this implementation is exposed in a literate programming style throughout the book.

At over 1500 pages, we only expect students to refer to this book as a supplemental resource for topics covered in class. You can access an e-publication version online for free in a [web-friendly format](http://www.pbr-book.org) and through [the McGill library network](https://www.sciencedirect.com/book/9780128006450/physically-based-rendering). !!! Tip You don't need to buy the textbook to be successful in the course, but it can help. Leveraging *lecture* and *tutorial* time (i.e., consistent attendance, taking notes and asking questions) correlates much more with success than reading the text. Equipment --------------------------------------------------- You don't need to buy any equipment. IT has configured lab machines in Trottier with an ECSE 446/546 development stack. The [first assignment](#Assignments) provides multi-platform instructions for setting up a local development environment, if you choose to work on a personal machine. Time --------------------------------------------------- We *strongly encourage* attending every lecture. Plan to invest *9 hours per week* for the 3-credit course or *12 hours per week* for the 4-credit version (this includes lecture & tutorial time.) For assignments, it's reasonable to spend roughly *65% of your time coding* -- the remainder will be split between planning, debugging and testing. Start early and schedule appropriately. Attending tutorials with your prepared questions is a good way to skirt avoidable difficulties. Evaluation ===================================================================== ECSE 446/COMP 598 students are evaluated based on five (5) coding [assignments](#assignments) and a [final exam](#Final). ECSE 546/COMP 599 students have an additional _scribing_ task, [detailed below](#Scribing).

Grading Component
(ECSE 446 & COMP 598 / ECSE 546 & COMP 599)
-------------------------------|------------------- Individual Coding Assignments |
75% / 50%
Final Exam |
25% / 25%
Scribing |
-- / 25%
Your score relies on the successful completion of the programming tasks, each described below. There is *no midterm exam*. A final exam will test the fundamental understanding you build from the lecture content and while working through the assignments. Generally speaking, if you attend lecture, ask questions and individually complete the assignments, then the exam should not pose a large difficulty. A detailed breakdown of each of these grading components follows. Coding Assignments ------------------------------------------------------------ ECSE 446 / COMP 598: 75% of the final grade
ECSE 546 / COMP 599: 50% of the final grade We will provide base code for a _hybrid rendering system_ that combines offline ray tracing and interactive rasterization. Assignments are to be completed individually. The difficulty of each assignment is roughly proportional to its weight, with the exception of *bonus tasks*: these are undervalued tasks meant for students who want to go beyond basic requirements. The assignment weight breakdown is:
(ECSE 446 & COMP 598 / ECSE 546 & COMP 599)
Due Date
-------------------------------|-------------------------|----------------- *Assignment 1* --
Setup & Basic Renders |
10% / 5%
| End of Week #3
[September 22] *Assignment 2* --
Light & Shadows |
10% / 5%
| End of Week #5
[October 6] *Assignment 3* --
MC Direct Lighting |
17% / 12%
| End of Week #8
[October 27] *Assignment 4* --
Advanced Monte Carlo |
18% / 13%
| End of Week #10
[November 10] *Assignment 5* --
Path Tracing |
20% / 15%
| End of Week #12
[November 24]
!!! ERROR: Late Policy: don't submit late work Failure to submit a (valid) assignment on time will result in a *score of 0* on the assignment. We will not grant extensions. Exceptional circumstances will be treated as specified in [McGill's Policies on Student Rights and Responsibilities](https://www.mcgill.ca/students/srr/). The assignment handouts detail the required tasks and submission procedures. Assignments must be handed in *no later than 11:59pm EST* on their due date. ![Example A1 Output](http://www.cim.mcgill.ca/~derek/ecse446/imgs/bunny_normal.png width="22%") **Assignment 1 -- Setup & Basic Render**
- Familiarize yourself with the base code & build process - Implement basic _rendering loops_ (offline & interactive) - Surface normal vector visualization output - Jittered anti-aliasing - *Estimated Work*: 3 -- 6 hours - Due end of week #3 [September 22] - [Read Me First](http://www.cim.mcgill.ca/~derek/ecse446/readme.md.html) - [Handout](http://www.cim.mcgill.ca/~derek/ecse446/a1.md.html) ![Example A2 Output](http://www.cim.mcgill.ca/~derek/ecse446/imgs/teapots_simple.png width="52.5%") **Assignment 2 -- Lights & Shadows**
- Diffuse direct lighting -- point lights - Interactive diffuse point light shader - *Bonus*: Real-time shadow mapping - *Estimated Work*: 6 -- 8 hours - Due end of week #5 [October 6] - [Handout](http://www.cim.mcgill.ca/~derek/ecse446/a2.md.html) ![A3 -- Sampling Directions on a Sphere Light](http://www.cim.mcgill.ca/~derek/ecse446/imgs/solid_angle_diagram.png width="37.5%") **Assignment 3 -- Monte Carlo Direct Illumination**
- Phong BRDF evaluation & sampling - Sampling spherical lights - Surface area and solid angle sampling - Ambient and reflection occlusion - *Estimated Work*: 10 -- 15 hours - Due end of week #8 [October 27] - [Handout](http://www.cim.mcgill.ca/~derek/ecse446/a3.md.html) **Assignment 4 -- Advanced Monte Carlo Methods**
- ![A4 -- Area Lights](images/butterfly.png width="22%") Multiple importance sampling - Control variates for polygonal lights - Interactive unshadowed shading - Hybrid Rendering - *Bonus*: Advanced analytic shading - *Estimated Work*: 10 -- 15 hours - Due end of week #10 [November 10] - [Handout](http://www.cim.mcgill.ca/~derek/ecse446/a4.md.html) **Assignment 5 -- Path Tracing**
- ![Example A5 Output](http://www.cim.mcgill.ca/~derek/ecse446/imgs/cbox_path_explicit_rr.png width="25%") Implicit & explicit path tracing - Precomputed interactive lighting - *Bonus*: Light tracing - *Estimated Work*: 8 -- 15 hours - Due end of week #12 [November 24] - [Handout](http://www.cim.mcgill.ca/~derek/ecse446/a5.md.html)
Scribe Notes (only for ECSE 546/COMP 599 students) ------------------------------------------------------------ ECSE 446 / COMP 598: 0% of the final grade
ECSE 546 / COMP 599: 25% of the final grade > "I didn't have time to write a short letter, so I wrote a long one instead." > -- Mark Twain Strong technical writing is an important skill and -- just as with coding and mathematical rigour -- practice is the best way to build it. Graduate students are expected to be able to clearly and concisely communicate complicated concepts to a broad audience. ECSE 546/COMP 599 students will form **groups of two**, and each group will author an illustrated and professionally-typeset _technical note_ documenting _one prescribed topic_ related to the course content. The list of [available topics](#ScribeTopics) is provided below and will be assigned on a **first come, first served** basis. Once a topic is assigned to a group, it can not be chosen by another group. We will provide you with a $\LaTeX$ [template](ecse446/scribe_template.zip) and instructions on exactly how to modify it: you are required to write your note while following the formatting and style of the [template](ecse446/scribe_template.zip). Click [this link](ecse446/scribe_template.zip) to download the template. We will also provide an example note (on ``Implicit Path Tracing'') as a grading guideline for the **content**, **structure** and **quality** expected of your work. You'll receive the full source for this note -- including for its diagrams, which you should use to bootstrap your own -- as well as a **marked up** version of the note that highlights important points for you to consider. This note will serve as the bar for a perfect score. Your note must respect the following parameters: - it should be between two (2) and four (4) pages in length, - it must clearly introduce the topic-specific problem, - to avoid repeating content between the various notes, you must assume "_relevant background_", established in discussions with the Professor, - diagrams must be used to contextualize the problem and explain technical details, - relevant pseudocode listings must be included, _and_ - in some cases, you must **implement the method** and include a comparitive result + **implementation is optional** for notes marked with $\star$ -- here, including an implementation can earn up to 10% bonus **per $\star$** (e.g., $\star\star\star$= up to 30% bonus.)
Here is the list of scribe note topics and associated lecture sections: Topic | Lecture References | Implementation Bonus | Assigned to ... -------------------------------------------------------------------------------------|-------------------------------------------------------------|-----------------------------------------------------------|---------------- Programmable Rasterization | **Systems I** | | **Hu** & **Koretski** Ray-tracing Fundamentals | **Systems II** | | **El Nakib** Phenomenological Shading | **Phenomenological Shading I & II** | | Still Available Ambient & Reflection Occlusion | **Monte Carlo Methods** &
**Direct Illumination II** | | Still Available Direct Lighting | **Direct Illumination I** & **II** | | Still Available Advanced Direct Lighting | **Advanced Direct Illumination I** | $\star$ | Still Available Advanced Direct Lighting -- Motion Blur | **Advanced Direct Illumination I** | $\star\star$ | **Ho** Appearance Models | **Appearance Models** &
**Advanced Materials I** | $\star$ | Still Available Implicit Path Tracing
([example](ecse446/ipt.pdf) & [mark up](ecse446/ipt_markup.pdf)) | **Global Illumination II** | | **Nowrouzezahrai** Explicit Path Tracing | **Global Illumination II** | | Still Available (Explicit) Light Tracing | **Global Illumination III** | $\star$ | Still Available Bidirectional Path Tracing | **Global Illumination III** | $\star\star\star$ | **Li** & **Zhao** Basic Photon Mapping | **Density Estimation I** | $\star\star$ | **Musing** & **Rahman** Final Gathering | **Density Estimation II** | $\star\star\star$ | Still Available Progressive Photon Mapping | **Density Estimation II** | $\star\star$ | **Lu** Virtual Point Lights | **Many Light Methods I** | $\star\star$ | **Hai** & **Wei** Volume Path Tracing | **Participating Media II** | $\star\star$ | Still Available Volume Photon Mapping | **Participating Media III** | $\star\star\star$ | **Lakshminarayanan** & **Shobeiri** Beam Radiance Estimate | **Participating Media III** | $\star\star\star\star$ | Still Available Photon Beams | **Participating Media III** | $\star\star\star\star$ | Still Available Diffusion BSSRDF | **Advanced Materials II** | $\star$ | Still Available The deadline for submitting your note is the **last day of lectures** [December 3]. ***Consult the Professor*** to brainstorm your exposition, diagram, layout and content.
Final Exam ------------------------------------------------------------ ECSE 446 / COMP 598: 25% of the final grade
ECSE 546 / COMP 599: 25% of the final grade The final exam will assess: 1. your technical understanding of the assignments -- by working towards implementing correct solutions to the assignment tasks (including revising incorrect solutions after your graded feedback), you should fair well on this portion of the exam, _and_ 2. your theoretical understanding of the course material -- here, lecture attendance and participation is the best preparation. Extensions, Solutions and Absences ------------------------------------------------------------ Due to the cumulative nature of the coding assignments, it is essential that everyone begin assignments early, review their corrected assignments and -- when necessary -- track down and correct bugs in old assignments. We will **not provide source code solutions** for assignments -- it's your responsibility to make sure your code is correct, consulting with the TAs **during the tutorial sessions**, if necessary. As [outlined earlier](#LatePolicy), late work will generally result in a score of 0. Please read the [course schedule](#CourseSchedule) and cross-reference with the [assignment deadlines](#AssignmentDeadlines) to avoid missing deadlines -- especially if you have **planned absences** -- and start work early each week to minimize the impact of **unplanned absences**. !!! ERROR: Online Solutions Do not post solutions online. Refer to the [collaboration & plagiarism policies](#CollaborationPolicy) below.
Course Topics & Schedule ===================================================================== The course combines slide-based lectures, hands-on tutorials, take-home programming assignments, scribing (in ECSE 546/COMP 599) and a final exam. Attendance and in-class participation tend to correlate with success on assignments and the final exam. The course will roughly follow the schedule below, modulo adjustments during the semester: !!! Tip **Office hours** are held **in (or just outside) the classroom** immediately after lectures.
: Welcome - Course administration - Course outline and schedule : Systems I -- Programmable Rasterization - Rasterization - Transformation pipeline - Z-buffer for primary visibility - Hierarchical acceleration - Programmable rasterization pipeline - Vertex and Pixel/Fragment Shaders - GPGPU - Modern programmable rasterization pipeline - Pros/Cons & Limitations : Systems II -- Ray-tracing - Rays and parameteric surfaces - Basic intersection testing - Spatial-hierarchy acceleration - Pros/Cons & Limitations : Phenomenological Shading I - Shading -- definition - Material vs. Lighting - Basic shading models - Diffuse/Lambertian reflection - Point light emitters - Ray-tracing shadows : Phenomenological Shading II - Mirror/perfect reflection & refraction - Simple glossy reflection - Directional light emitters : Appearance Models - Radiometry 101 - Bi-directional reflectance distribution functions (BRDFs) - Mathematical definition - Properties of physically-based local scattering - Well-founded Diffuse & Glossy reflection : Monte Carlo Methods - Definitions - Canonical random variables - Expected value - Variance - Monte Carlo Integration : Monte Carlo Methods - Monte Carlo Estimator Design - Sampling distributions - Importance sampling - Case study: Ambient Occlusion & Importance Sampling : Direct Illumination I - Definition - Monte Carlo Estimators - Hemispherical parameterization - Surface-area parameterization - Light Importance Sampling - Spherical lights - Mesh lights : Direct Illumination II - Revisiting Ambient Occlusion - BRDF Importance Sampling - Combining Sampling Strategies - Multiple Importance Sampling : Advanced Direct Illumination I -- Advanced Importance Sampling - Environment light sampling - Product importance sampling - Hierarchical sample warping - Primary Distribution Effects - Depth of field - Motion blur (): Thanksgiving -- No Lecture : Advanced Direct Illumination II -- Control Variates - Control variates with unshadowed direct illumination - Diffuse Reflection from Point Lights - Diffuse Reflection from Polygonal Area Lights - Diffuse Reflection from Environmental Lights - Glossy Reflection : Global Illumination I -- Path Space - Light paths - Heckbert notation - The Rendering Equation - Operator notation - Finite Element Solutions -- Radiosity : Global Illumination II -- Monte Carlo Estimators - Naive estimator - Geometric fan-out - Deriving the (Implicit) Path Tracing Monte Carlo Estimator - Explicit Path Tracing - Russian Roulette Unbiased Path Termination - Challenges of "complex paths" : Global Illumination III -- Duality of Light Transport - The Measurement Equation - Local **and** Global Reciprocity in Light Transport - Applying Duality - Light tracing - Light tracing with next-event estimation - The Path Integral Formulation of Light Transport - Solving Integrals in Path Space - Monte Carlo Estimators - Applying Multiple Importance Sampling - Bidirectional Path Tracing - Complications -- SDS Paths : Density Estimation I - Motivation -- Spatial Regularization - "Backward" Ray Tracing - Description & Limitations - Photon Mapping - Basic Algorithm - Importance Sampling - Relationship to Bidirectional Techniques - Pros/Cons : Density Estimation II - Improving Basic Photon Mapping - Segmenting Maps and Techniques - Progressive Photon Mapping - Convergence Guarantees - Parallelization : Many Light Methods I - Characterizing Estimators - Virtual Point Lighting Methods - Advanced VPL Generation - Bias from Clamping and Energy Loss - Spatial Regularization for Virtual Sphere Lights : Many Light Methods II - Relationship to Photon Mapping - Final Gathering versus VPLs - Scalability of Many Light Methods - _Lightcut_ Strategies : Participating Media I -- Theory - Definitions - Examples of Volumetric Media - Properties of Volumetric Media - The Radiative Transport Equation - Deriving the Volume Rendering Equation : Participating Media II -- Monte Carlo Methods - Basic Monte Carlo Estimators for the Volume Rendering Equation - Volumetric (Forward) Path Tracing - Implicit Volumetric Path Tracing - Explicit Volumetric Path Tracing - Importance Sampling Strategies - Heterogeneous Volumes - Woodcock Tracking : Participating Media III -- Density Estimation - Volumetric Photon Mapping - Volumetric Radiance Density Estimators - Photon Point Primitives - Fixed-kernel vs. Kernel-adaptive Density Estimates - The Beam Radiance Estimator : Participating Media IV -- Advanced Density Estimationopope & Many Lights Methods - Photon Beam Primitives - Higher-order Photon Primitives - Volumetric Virtual Point Lights - Volumetric Virtual Ray Lights - Spatial Regularization - Volumetric Virtual Beam Lights : Advanced Materials I -- Realistic Surface Reflectance - Towards More Realistic Local Reflectance - Conductors versus Dielectrics - The Three Levels of Material Details - Scattering from Rough Surfaces - Normalized Phong Reflectance (review) - Blinn-Phong - Isotropic and Anisotropic Ward - Microfacet Theory from Statistical Physics - General Microfacet Model - Instances of the Microfacet Model - Beckmann Distributions - Cook-Torrance Model - Oren-Nayar Microfacet Diffuse Model - Data-driven BRDFs : Advanced Materials II -- Subsurface Scattering - Blurring the Line between Surface and Volume Scattering - Bi-directional Subsurface Scattering Reflectance Distribution Function - Definition and Properties - The High-scattering Regime - Diffusion Approximations - The Searchlight Problem - An End-to-end Multi-scattering Subsurface Scattering Model - Diffusion Dipoles : Advanced Materials III -- BSSRDF Variants - Practical Considerations - Separating Single- and Multiple-scattering - Hierarchical Sampling - From Dipoles to Multipoles - Quantized Diffusion - Photon Beam Diffusion
Collaboration & Plagiarism ===================================================================== *Plagiarism* is an academic offense of misrepresenting authorship. This can result in penalties up to expulsion. It is also possible to plagiarise _your own work_, e.g., by submitting work from another course without proper attribution. **When in doubt, attribute!** We expect you to submit your own work. Assignments are individual tasks. That said, we want to promote an environment where you are comfortable discussing ideas together. **A good rule to follow**: fully understand every solution you submit and only submit code that was written by you. McGill values academic integrity and students should take the time to fully understand the meaning and consequences of cheating, plagiarism and other academic offenses (as defined in the Code of Student Conduct and Disciplinary Procedures -- see [these](www.mcgill.ca/integrity) [two](www.mcgill.ca/students/srr/honest) links). In accordance with article 15 of the Charter of Students' Rights, students may submit any written or programming components in either French or English. If you have a disability, please advise the [Office for Students with Disabilities](www.mcgill.ca/osd) (514-398-6009) as early in the semester as possible. In the event of circumstances beyond our control, the evaluation scheme as set out above may require modification. Additional policies governing academic issues which affect students can be found in the Handbook on Student Rights and Responsibilities.