Virtual Vienna: Modeling the City with Machine Learning and Procedural Geometry

Description

Many 3D city models are available in on-line systems, such as Google Earth or 3DCityDB. However, the geometry of these models is usually low-quality, since their main purpose is to provide an overview or enable navigation/orientation. Both obtaining and storing/transferring detailed geometry would be too demanding for storage and internet traffic. Since the geometry of these models is usually quite compact, it doesn't hold up to scrutiny when we zoom in close. Specifically, it would look particularly poor in a 3D or virtual reality application. In this project, we want to develop a way to model the entire city of Vienna that can be used in such immersive 3D scenarios, without requiring high-quality 3D scans or exabytes of hand-made detailed geometry. To do so, we are collaborating with the city of Vienna, which has provided us with a basic, textured 3D CityGML model of Vienna. 

Based on the textured model, we first want you to use machine learning to detect meaningful properties of individual buildings: facade color, style, number of floors, number of windows, etc. These properties can then be stored in a compact feature vector, which is easily transferred to end-user machines. From these properties, you can then invoke a procedural geometry generator on the target machine, which augments the facades of the basic city model with corresponding geometry from the feature vector. The generator adds high-detail windows, doors, ornaments, pillars, etc. Since the base model is provided as CityGML, the resulting high-detail geometry is extremely clean and suitable for immersive 3D applications. This way, the application can produce a high-quality, detailed model of any building in the city for interactive cinematics, navigation apps, physical simulations or promotional videos, without transferring vast amounts of data. 

Tasks

There are several research tasks that need to be addressed in this project. Your work will focus on one or a combination of these topics:

• Research, try and suggest existing methods for detecting important features of buildings (color, number of floors, window type...) from the textured model
• Build a system that can identify the most relevant properties of buildings based on the textured CityGML model of Vienna (e.g., via image analysis) that can be compactly stored in a feature vector
• Find out how additional information, such as point cloud scans or hand-made ground truth geometry compares to the textured model or can be used to further improve quality. These data sets may be provided by the city of Vienna on demand. 
• Write shape grammars for our (existing!) procedural geometry generator that use these properties as input to produce high-quality, high-detail geometry for the properties from the feature vector
• Write a decent rendering client to render the generated high-quality city geometry up close. If you want to, you can do it as a VR application and test it on one of our VR HMDs, such as the Vive Pro.
• If you have additional ideas in this direction, the city of Vienna may have opportunities to actually produce new data (aerial recordings, scans) for interesting use cases.

Requirements

• Self-initiative and motivation to research and tackle a topic where you can push the status quo of computer graphics!
• Knowledge of English language (source code comments and final report should be in English)
• Knowledge of C++ required
• Prior experience with image processing is advantageous
• Prior experience with machine learning (for corresponding topics) is advantageous
• Knowledge of Python is advantageous, but not necessary
• More knowledge is always advantageous

Environment

We are looking for at least 3 motivated students. If two or more of you would like to work together, feel free to apply as a team. What particular part of the topic you will work on is up for discussion, just shoot us a message and we can talk about expected workload, your schedule and other factors before we decide on the actual tasks. Any work on the project should be implemented in commonly accessible programming languages such as C++ and Python. The procedural geometry generator is written in C++, grammars are written based on an easy-to-understand rule set. For machine learning tasks, you are free to use any environment that you feel comfortable with. If you develop a virtual reality application, we recommend that you use an existing engine such as Unity3D or Unreal 4/5.