SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage – NewsEverything | Technology

It is difficult to take a portrait photograph under direct sunlight. Professional photographers use elaborate lighting setups, but casual users must rely on alternative solutions.

A portrait of a man. Image credit: رضاصاد via Wikimedia, CC-BY-SA-4.0

A recent paper published on arXiv.org proposes a method to enable high-quality, physically-grounded portrait relighting without additional hardware.

The proposed capture setup requires rotating the camera, allowing the face to be observed under varying sunlight angles. The method then reconstructs geometry, material, camera pose, and lighting information. This information is used to realistically render the subject in unseen lighting conditions.

Complex details that do not explicitly model geometry, like self-occluding cast shadows, are also rendered realistically. Moreover, the method can be used to render the person in an entirely new environment, or add freckles, makeup, or stickers that realistically interact with scene lighting.

Outdoor portrait photographs are often marred by the harsh shadows cast under direct sunlight. To resolve this, one can use post-capture lighting manipulation techniques, but these methods either require complex hardware (e.g., a light stage) to capture each individual, or rely on image-based priors and thus fail to reconstruct many of the subtle facial details that vary from person to person. In this paper, we present SunStage, a system for accurate, individually-tailored, and lightweight reconstruction of facial geometry and reflectance that can be used for general portrait relighting with cast shadows. Our method only requires the user to capture a selfie video outdoors, rotating in place, and uses the varying angles between the sun and the face as constraints in the joint reconstruction of facial geometry, reflectance properties, and lighting parameters. Aside from relighting, we show that our reconstruction can be used for applications like reflectance editing and view synthesis. Results and interactive demos are available at this https URL.

Research article: Wang, Y., Holynski, A., Zhang, X., and Zhang, X. C., “SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage”, 2022. Link: https://arxiv.org/abs/2204.03648

You can offer your link to a page which is relevant to the topic of this post.