Dancing Under the Stars: Video Depicting the Starlight

To take pictures in dark settings, photographers use cameras that increase gain, effectively making each pixel more sensitive to light. However, it amplifies the noise present in each frame. A recent paper on arXiv.org proposes a new method for submillilux video denoising.

The researchers proposed a camera optimized for low-light imaging and set to the highest gain setting. The camera noise model is learned using physics-inspired noise generators and still noise images easily obtainable from the camera. The noise model then generates synthetic clean/noisy video pairs to train a video denoiser.

The effectiveness of the denoising network is demonstrated on 5-10 fps video taken on a clear moonlit night. Several challenging scenes are presented with sweeping motion, such as dancing only with the lights of the Milky Way as a meteor showers from above.

Imaging in low light is extremely challenging due to the low number of photons. Using sensitive CMOS cameras, it is currently possible to take video at night under moonlight (0.05–0.3 lux illumination). In this paper, we demonstrate photorealistic video under starlight (no moon exists, <0.001 lux) for the first time. To enable this, we develop a GAN-tuned physics-based noise model to more accurately represent camera noise at the lowest light levels. Using this noise model, we train a video denoiser using a combination of simulated noise video clips and real noise still images. We capture 5-10 fps video datasets with significant speed at about 0.6-0.7 millilux with no active illumination. Compared to alternative methods, we achieve better video quality at the lowest light levels, displaying photorealistic video in Starlight for the first time.

Research Article: Monakhova, K., Richter, S. R., Waller, L., and Koltun, V., “Dancing Under the Stars: Video Denoising in Starlight”, 2022. Link of Paper: https://arxiv.org/abs/2204.04210
Project Page: https://kristinamonakhova.com/starlight_denoising/

You can submit your link to a page that is relevant to the topic of this post.