High-Quality Reflections in Mobile Augmented Reality

Type: 
BA/PR/DA
Persons: 
1-2
Workgroup: 

Description

High-quality reflections can contribute immensely to the visual fidelity and the perceived realism of virtual objects in Augmented Reality (AR) scenes. The first step would be to take the surrounding into account, i.e. using a mobile device's camera to capture the surrounding. Optimally, as much as possible should be captured around the mobile device, so it would be desirable to simultaneously capture with both cameras of a mobile device (the one usually facing away from the user and the one facing towards the user) simultaneously. Not all mobile devices support this, but you we might be able to provide you with one which is capable of it.

Additionally to the images of the real world, virtual objects should be reflected in high quality. A technique, which has become very popular in games released recently, are Screen-Space Reflections (SSR). SSR shall be used in a first step to reflect virtual objects. While for BA and PR this will be sufficient scope-wise, further reflection methods will have to be evaluated for DA with the ultimate goal to omnidirectionally reflect both, the real-world and virtual objects.

Screen-Space Reflections have become very popular in recent games. They Port a given 3D rendering engine to the Android platform, or develop your own rendering engine, and implement Screen-Space Reflections in the context of a small Augmented Reality Game or a small Augmented Reality scene. Use the mobile device’s camera to capture the surrounding and include the surroundings into the algorithm. Evaluate capturing the surroundings with both of the mobile device’s cameras in order to capture almost the full sphere of light.

Tasks

  • Develop a mobile application for the Android platform and include your technique. The substance of your application can be a small AR game or any other AR scene with moving virtual objects.
  • Develop your application either with a custom engine (there is the option to port an existing C++/OpenGL engine to Android) or use an existing game engine which is both mobile- and customization-friendly (Unity3D is a good choice for that).
  • Access both cameras of a mobile device using e.g. Google's Camera API [1], Samsung's API [2] or similar.
  • Implement the Screen-Space Reflections technique and add it to your game/scene [3], [4]
  • (Optional) Implement further reflection techniques to improve your results. (Required for DA)
  • Write about your findings, your approach, describe your technique and present your results in your thesis or project report.

Requirements

  • Knowledge of C++ and OpenGL (if you are using a custom engine) or knowledge of C# and Unity3D
  • Knowledge of screen-space rendering techniques
  • Some fundamental knowledge about the Android platform
  • Knowledge of the English language (source code comments and thesis should be in English)
  • Prerequisite: In order to apply for this topic, you are obligated to prepare a small Android application (can be native or based on Unity3D) which accesses at least one of the mobile device's cameras simultaneously ([1], [2] or a similar API) and display the camera image. If you are going for Unity3D, you'll probably want to use Unity's AndroidJavaObject [5] or build an Android plugin [6] for that purpose. (Note: Accessing both cameras at the same time will only work on certain mobile devices.)

Environment

The project should be implemented as a standalone AR application for the Android platform. You may use (or port) a custom engine or use Unity3D for implementing your technique.

References

[1] Google Developers. Camera API. https://developer.android.com/guide/topics/media/camera#access-camera

[2] Samsung Developers. Camera. https://developer.samsung.com/galaxy/camera

[3] Erik Sintorn. Screen-space Reflections. Chalmers University of Technology and Göteborg University. http://www.cse.chalmers.se/edu/year/2017/course/TDA361/Advanced%20Computer%20Graphics/Screen-space%20reflections.pdf

[4] McGuire and Mara. Efficient GPU Screen-Space Ray Tracing. Journal of Computer Graphics Techniques. 2014. http://jcgt.org/published/0003/04/04/paper.pdf

[5] Unity - Scripting API. AndroidJavaObject. https://docs.unity3d.com/ScriptReference/AndroidJavaObject.html 

[6] Unity - Manual. Building and using plugin-ins for Android. https://docs.unity3d.com/Manual/PluginsForAndroid.html

Contact

For more information please contact Johannes Unterguggenberger (unterguggenberger@cg.tuwien.ac.at).