Google researchers have developed a 3D modeling technique for monuments (or, in general, any object in a public place) using a neural network to process thousands of photographs taken from different points.
Google’s development is based on NeRF (Neural Radiance Fields), a technique that allows extracting 3D depth data from 2D images by determining where the light rays end. It is a sophisticated technique that can create textured 3D models.
Google has gone further by developing NeRF in the Wild (NeRF-W). First, it uses “collections of photos in the wild” as inputs, expanding the ability of a computer to view reference items from multiple angles. These photographs can be snapshots taken by tourists from different places.
The images are then evaluated for structures, separating photographic and environmental variations such as image exposure, scene lighting, post-processing, and weather conditions, as well as differences between photographed objects, such as people who may be in one image but not in another.
Then it recreates scenes as mixtures of static elements – geometry of the structure and textures – with other transients.
As a result, Google’s 3D models can be viewed from multiple angles without looking artificial. Video comparison of standard NeRF results with NeRF-W suggests that the new neural system can convincingly recreate 3D landmarks that could be leveraged for virtual reality and augmented reality applications.
Google isn’t the only company investigating ways to use photos as input for 3D modeling; for example, Intel researchers are advancing their own work of generating synthesized versions of real-world locations , using multiple photographs plus a recurring codec network to interpolate uncaptured angles.
While Intel’s system appears to outperform numerous alternatives – including standard NeRF – in terms of pixel-level sharpness and model smoothness, it doesn’t seem to offer the variable lighting capabilities of the NeRF-W or have the same focus on the model. use of photos of random origin to recreate real-world locations.
