Context
We are currently working on lighting transfer for architectural design, where the goal is to learn light characteristics from a single image and replicate them to a 3D scene, giving it a similar look and mood. However, extracting meaningful information from images is a complicated problem, since the resulting pixels of the image are a combination of materials, lighting, and reflections.
Description
The goal of this project is to implement the paper "Real-time Global Illumination Decomposition of Videos" of [Meka et al. 2021], which decomposes images into base colors (or materials), direct illumination, and indirect illumination. This is done by expressing constraints on materials, illumination and the resulting image as an energy function, which is then minimized with optimization algorithms.
Tasks
Implement the most important parts of [Meka et al. 2021]:
- Static clustered colors as estimation of the materials
- The most important terms of the energy function
- The optimization algorithm
The project can optionally be extended to include:
- Dynamic correction of misclustered regions
- All the terms of the energy function
- Data parallel GPU optimization
Requirements
- Knowledge of English language (source code, report and meetings should be in English)
- Solid knowledge of a programming language is advantageous (Python or C++ is recommended)
The project should be implemented as a standalone application, and work on Linux and/or Windows.
References
[Meka et al. 2021] Meka, A., Shafiei, M., Zollhöfer, M., Richardt, C., & Theobalt, C. (2021). Real-time Global Illumination Decomposition of Videos. ACM Transactions on Graphics (TOG), 40(3), 1-16.