Context
Lighting design is a key component to control the look and feel of the environment, and plays a central role when designing buildings or installations. It can be used to provide a comfortable environment for residential and office buildings, to set the atmosphere of theater stage or movie sets, or even to showcase art pieces in museums.
Description
While it's possible for a human to express how a scene feels to them, it's currently not possible for a computer to provide the same type of analysis. The goal of this project is to use recent deep learning methods to allow an AI to rate the lighting and mood of an image. This can be done in multiple ways, for example by computing the style similarity between an image and a target, or by classifying images in distinct predefined categories based on the type of lighting that they represent.
Tasks (depending on PR/BA/DA and number of students)
- Use a network like VGG [Simonyan and Zisserman 2014] and a style-loss [Gatys et al. 2015] to compare the style of a test image and a reference image. This will give information on how close the style of the two images are, but might require additional tweaks to recognize features specific to the illumination instead of just low level features such as the color palette.
Fine-tune a classifier network like ResNet to classify images in predefined illumination styles. The classifier would then be able to label illumination style, but those different classes need to be carefully crafted to obtain meaningful results. The creation of the training dataset will also be a challenge.[already taken]Test the ability of CLIP [Radford et al. 2021] or similar text to image networks to optimize images according to abstract concepts regarding the mood of the scene. This could include concepts such as "gloomy", "cheerful", etc.[unconvincing results]
Requirements
- Knowledge of English language (source code, report, and communication should be in English)
- Knowledge of Python and a deep learning framework is advantageous
The project will most likely be implemented in Python using standard deep learning frameworks, but this can be discussed depending on the experience of the student.
References
[Simonyan and Zisserman 2014] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition.
[Gatys et al. 2015] Gatys, L. A., Ecker, A. S., & Bethge, M. (2015). A neural algorithm of artistic style.
[Radford et al. 2021] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021). Learning transferable visual models from natural language supervision.