Cross-Modal

Duration: Dec 2005 - Dec 2008
Project leader: Michael Wimmer
Funding:
EU IST - 6th Framework Program

General Information

Virtual environments (VEs) play an increasingly important role in our society. Currently two main sensorial channels are exploited in VEs: visual and auditory, although sound is greatly underused. The ever-increasing scene complexity of VEs means that it is currently not possible to display highly realistic scenes in real time despite the availability of modern high-performance graphics and audio processors. However, the realism and quality of a virtual image/sound needs to be as good as what the user can perceive: we only need to display what is necessary. Despite recent research in psychology, little work exists on cross-modal effects, i.e., the effects that each channel (visual and auditory) has on the other, to improve the efficiency and quality of VEs. CROSSMOD will study these effects and develop a better understanding of how perceptual issues affect auditory/visual display; this understanding will lead to the development of novel algorithms for selectively rendering VEs. The cross-modal effects studied will include the effect of spatial/latency congruence on quality perception, attention-control, sound-induced changes in visual perception and foveal/peripheral audiovisual effects. The research will be guided by its applicability to the improvement of VE display and VE authoring. The solutions developed by CROSSMOD will enable the display of perceptually highly realistic environments in real time even for very complex scenes, as well as the use of active cross-modal effects such as attention control in the display and authoring of VEs. An integrated software cross-modal manager will be developed, using results of experiments identifying which cross-modal effects are useful for VE display. We will evaluate our approach on three target applications: computer games, design/architecture and clinical psychiatry, using platforms adapted to each application: PCs with audio/graphics cards, a projector-enhanced “personal VE” setup and large-screen immersive VE systems.

Publications

8 Publications found:
Image Bib Reference Publication Type
2011
Matthias Bernhard, Karl Grosse, Michael Wimmer
Bi-modal Task Faciliation in a Virtual Traffic Scenario through Spatialized Sound Rendering
ACM Transactions on Applied Perception, 8(4):1-22, November 2011.
Journal Paper (without talk)
Matthias Labschütz
Realistic real-time sound rendering in virtual environments
[paper]
Bachelor Thesis
2010
Matthias Labschütz
Enabling bimodal search in games by spatial sound rendering
Student Project
2009
David Grelaud, Nicolas Bonneel, Michael Wimmer, Manuel Asselot, George Drettakis
Efficient and Practical Audio-Visual Rendering for Games using Crossmodal Perception
In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2009, pages 177-182. February 2009.
[Preprint]
Conference Paper
2008
Veronica Sundstedt, Efstathios Stavrakis, Michael Wimmer, Erik Reinhard
A Psychophysical Study of Fixation Behavior in a Computer Game
In ACM Symposium on Applied Perception in Graphics and Visualization 2008, pages 43-50. August 2008.
[draft]
Conference Paper
Daniel Scherzer, Michael Wimmer
Frame Sequential Interpolation for Discrete Level-of-Detail Rendering
Computer Graphics Forum (Proceedings EGSR 2008), 27(4):1175-1181, June 2008. [draft]
Journal Paper with Conference Talk
Oliver Mattausch, Jiří Bittner, Michael Wimmer
CHC++: Coherent Hierarchical Culling Revisited
Computer Graphics Forum (Proceedings Eurographics 2008), 27(2):221-230, April 2008. [draft]
Journal Paper with Conference Talk
2007
Daniel Scherzer, Stefan Jeschke, Michael Wimmer
Pixel-Correct Shadow Maps with Temporal Reprojection and Shadow Test Confidence
In Rendering Techniques 2007 (Proceedings Eurographics Symposium on Rendering), pages 45-50. June 2007.
[Preprint]
Conference Paper
Download list as Bibtex, CSV, HTML, JSON