Researchers with Google Research and the Google Brain deep learning AI team have published a new study detailing Neural Radiance Fields for Unconstrained Photo Collections (NeRF). The system works by taking ‘in the wild’ unconstrained images of a particular location — tourist images of a popular attraction, for example — and using an algorithm to turn them into a dynamic, complex, high-quality 3D model.
The researchers detail their project in a new paper, explaining that their work involves adding ‘extensions’ to neural radiance fields (NeRF) that enable the AI to accurately reconstruct complex structures from unstructured images, meaning ones taken from random angles with different lighting and backgrounds.
This contrasts to NeRF without the extensions, which is only able to accurately model structures from images that were taken in controlled settings. The obvious benefit to this is that 3D models can be created using the huge number of Internet photos that already exist of these structures, transforming those collections into useful datasets.
The Google researchers call their more sophisticated AI ‘NeRF-W,’ one used to create ‘photorealistic, spatially consistent scene representations’ of famous landmarks from images that contain various ‘confounding factors.’ This represents a huge improvement to the AI, making it far more useful compared to a version that requires carefully controlled image collections to work.
Talking about the underlying technology, the study explains how NeRF works, stating:
‘The Neural Radiance Fields (NeRF) approach implicitly models the radiance field and density of a scene within the weights of a neural network. Direct volume rendering is then used to synthesize new views, demonstrating a heretofore unprecedented level of fidelity on a range of challenging scenes.’
There’s one big problem, though, which is that NeRF systems only work well if the scene is captured in controlled settings, as mentioned. Without a set of structured images, the AI’s ability to generate models ‘degrades significantly,’ limiting its usefulness compared to other modeling approaches.
Source.: Digital Photography Review