My master thesis at the Fraunhofer Institute for Computer Graphics was all
about mixed reality. The problem with mixed reality is that it often either looks very ugly,
i.e. simple lighting and no shadows, or takes huge amounts of time for processing. Imagine this: A virtual
object is placed into a real scene. The virtual object must have a shadow, visible on real objects. If someone
has a red light source shining in direction of the virtual object, it must reflect red light. Also, a virtual
light must illuminate the real scene, while a real shadow must also shade a virtual object. This interaction
between virtual and real lighting has often been attacked with slow, clustered global illumination techniques.
My task was to reimplement fast methods for real-time GPU usage.
Ambient Occlusion
In my first step, I wrote an ambient occlusion renderer, which maps the statistical light distribution
over the surface of the model. The technique that is employed is a modification of
Hardware-accelerated ambient occlusion computation by Sattler et al. The samples used
to calculate the accesibility factor are gathered in a Monte Carlo style manner, using boost::uniform_on_sphere.
Irradiance Mapping
In the next step, the incoming light, which is captured by 180 degree wide angle lens or a light probe,
is transfered to the model via irradiance mapping. Real-time irradiance map generation is done
through the use of spherical harmonics, as described in
Spherical Harmonic Lighting:
The Gritty Details by Robin Green, and in
Real-Time Computation of Dynamic Irradiance Environment Maps by Gary King, both of which I can
highly recommend. Depending on the materials specularity and light transfer intensity (a HDR parameter),
the colors are selected from different irradiance maps and then multiplied with the ambient occlusion parameters
on the surface. In the following picture, the Grace Cathedral probe
from Paul Debevec was used to calculate different irradiance maps. I also toyed around with
PRT, but it didn't handle animated
objects. The combination of ambient occlusion and irradiance mapping covers for most models.
Material Reconstruction
Because inserting virtual objects into real scenes leads to some changes in shading and illumination of real surfaces,
surface BRDF parameters have to be known upfront of each and every material the virtual object will interact with.
A mirror behaves differently from a diffuse surface in a sense that they both reflect incoming shadows or light differently.
I implemented the method in Flexible Image-Based
Photometric Reconstruction using Virtual Light Sources by Gibson et al., which analytically tries to reconstruct the
surface BRDF from one or more images with a corresponding 3d model. This was a little rough-ride (it worked in some cases).
Also, complex surfaces have so many parameters to solve for, that I used genetic algorithms through
GAlib for this job instead.
Differential Rendering
After light, geometry and materials have been reconstructed, it's time for some rendering action. The fusion of real and
virtual objects is done via differential rendering in a shader. The paper for it is from Paul Debevec called
Rendering Synthetic Objects into Real Scenes. Here
are some results.
The models have been taken from the
Large Geometric Models Archive
of the Georgia Institute of Technology. Rendering was done with OpenSG
and Avalon. Render speed varied between 8 FPS for the dragon model and 60-80 FPS for the bunny.
Addendum: Yay, I won the
"Best Thesis Award"! Here's a picture of the "Computer Grafik Abend" in Darmstadt
at the Fraunhofer IGD were everything happend.
