PT Journal
AU Schiller, I
   Bartczak, B
   Kellner, F
   Koch, R
TI Increasing Realism and Supporting Content Planning for Dynamic Scenes in a Mixed Reality System incorporating a Time-of-Flight Camera
SO JVRB - Journal of Virtual Reality and Broadcasting
PY 2010
VL 7(2010)
IS 4
DI 10.20385/1860-2037/7.2010.4
DE 3D-Modeling; 3DTV; Depth Keying; GPU; Mixed Reality; PMD; Shadow Mapping; TOF; Time-of-Flight
AB For broadcasting purposes MIXED REALITY,the combination of real and virtual scene content, hasbecome ubiquitous nowadays. Mixed Reality recordingstill requires expensive studio setups and is often limitedto simple color keying. We present a system for MixedReality applications which uses depth keying andprovides threedimensional mixing of real and artificialcontent. It features enhanced realism through automaticshadow computation which we consider a core issue toobtain realism and a convincing visual perception,besides the correct alignment of the two modalities andcorrect occlusion handling. Furthermore we present apossibility to support placement of virtual content in thescene.Core feature of our system is the incorporation of a     TIME-OF-FLIGHT (TOF)-camera device. This device  delivers real-time depth images of the environment at a  reasonable resolution and quality. This camera is used to  build a static environment model and it also allows correct  handling of mutual occlusions between real and virtual  content, shadow computation and enhanced content  planning.  The presented system is inexpensive, compact, mobile,  flexible and provides convenient calibration procedures.  Chroma-keying is replaced by depth-keying which is  efficiently performed on the GRAPHICS PROCESSING  UNIT (GPU) by the usage of an environment model and  the current ToF-camera image. Automatic extraction and  tracking of dynamic scene content is herewith performed  and this information is used for planning and alignment of  virtual content.  An additional sustainable feature is that depth maps of the  mixed content are available in real-time, which makes the  approach suitable for future 3DTV productions. The  presented paper gives an overview of the whole system  approach including camera calibration, environment  model generation, real-time keying and mixing of virtual                                                                                                                                                                                        and real content, shadowing for virtual content anddynamic object tracking for content planning.
ER