The eyes provide important social cues and even let to determine the subject’s emotional state. Therefore, in the development of photorealistic 3D avatar technologies, significant attention must be paid to methods and systems that allow for fine-grained photorealistic control over human eye imagery.
A recent paper on arXiv.org proposes a novel hybrid representation that combines the best of mesh-based and volumetric reconstruction to achieve animatable synthesis of the eye region under desired environmental lighting. A lightweight capture system is designed from a small number of static cameras and lights along with a single hand-held camera with a co-located light source.
The method enables photorealistic view synthesis and relighting. Moreover, fine-grained control over the subject’s gaze is achieved. These capabilities are demonstrated on several real subjects with varying facial and ocular characteristics.
A unique challenge in creating high-quality animatable and relightable 3D avatars of people is modeling human eyes. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing. These challenges have traditionally necessitated the use of expensive and cumbersome capture setups to obtain high-quality results, and even then, modeling of the eye region holistically has remained elusive. We present a novel geometry and appearance representation that enables high-fidelity capture and photorealistic animation, view synthesis and relighting of the eye region using only a sparse set of lights and cameras. Our hybrid representation combines an explicit parametric surface model for the eyeball with implicit deformable volumetric representations for the periocular region and the interior of the eye. This novel hybrid model has been designed to address the various parts of that challenging facial area – the explicit eyeball surface allows modeling refraction and high-frequency specular reflection at the cornea, whereas the implicit representation is well suited to model lower-frequency skin reflection via spherical harmonics and can represent non-surface structures such as hair or diffuse volumetric bodies, both of which are a challenge for explicit surface models. We show that for high-resolution close-ups of the eye, our model can synthesize high-fidelity animated gaze from novel views under unseen illumination conditions.
Research article: Li, G., Meka, A., Müller, F., Bühler, M. C., and Hilliges, O., “EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes”, 2022. Link: https://arxiv.org/abs/2206.08428
Follow News Everything for News Today, Breaking News, Latest News, World News, Breaking News Headlines, National News, Today’s News
#EyeNeRF #Hybrid #Representation #Photorealistic #Synthesis #Animation #Relighting #Human #Eyes