As we learned in one of our most recent articles, we can find several popular datasets in computer vision and autonomous driving tasks in the visual perception market. But if we focus on synthetic datasets, aiming for the highest data quality standards and, on the basis that “all synthetic data are not created equal”, why are Anyverse’s synthetic datasets for computer vision different? Why are they unique, and what are their key differentiators? Why are they more accurate and comprehensive than other available datasets?
Why Anyverse's synthetic datasets for computer vision are different?
Hyperspectral synthetic data is different because it renders a scene and all its content based on the physics of the light. Anyverse implements a pure spectral path-tracing engine that computes the spectral radiance of every light beam, simulating lights, cameras, and materials with physical accuracy. This allows for a precise simulation of the amount of light reaching the camera sensor to generate a final image containing full spectral information per pixel to minimize the domain gap of datasets assuring machine learning models generalize to real-world inputs properly.
Image 1: Wavelengths that reach the camera sensor for a given pixel of an image
The AI deep learning models behind the perception systems need to understand the real world by example with vast amounts of training data for specific environments and use cases including corner cases. Trying to use only real-world images may be difficult and expensive to get. That is why the use of synthetically generated data is helping improve the performance and accuracy of perception systems. Real-world details are infinite. When generating synthetic images simulating cameras, we need to be able to reproduce and capture as many details as possible from a computer-generated 3D world as we would capture using real cameras in the real world.
Don’t forget that, at the end of the day, the perception systems will use real cameras (and other sensors). Those details that we need to generate more faithful images to feed our perception brain is what we call hyperspectral data.
Hyperspectral in this context means that when rendering the synthetic 3D world we have created we sample several wavelengths within a given spectrum (for cameras, visible + near-infrared part of the electromagnetic spectrum). So rays with different wavelengths travel and bounce back and forth hitting the different objects in the 3D scene and reaching the camera at some point with a given energy dependent on the wavelength (Planck Dixit).
All this info, the photons (rays) with different wavelengths, that reach the camera sensor for every pixel is the hyperspectral data for an image. The more wavelengths we sample in the given spectrum, the more details we will capture, the more accurate the camera simulation can be, and the better results you will get.
It is worth noticing that as important as it is that the render supports hyperspectral, is that the computer-generated 3D world and all the objects in it are characterized in a way that the render knows the behavior of every ray when it hits the objects in its way to the camera.
This behavior depends on the materials and it is wavelength-dependent (for those of you not in the know, these are the BRDFs or bidirectional reflectance distribution functions).
Image 2: Color RGB - AV use cases
Image 3: Color near-infrared - Driver monitoring use cases
Image 4: LiDAR- AV uses cases
Image 5: Color RGB - Warehouse use cases
Image 6: Color RGB - Aerial use cases
Image 7: Color RGB - Security & defense use cases
Image samples for various use cases generated with Anyverse
Hyperspectral synthetic data is the only way to physically simulate optics and sensors. It’s as simple as that. Faithfully simulating the sensor is a great way to reduce the domain gap in your synthetic datasets generating images closer to what the real cameras would take. So training with such images you have a better chance that your perception deep learning model will generalize to real-world images improving the performance of your perception system in a very cost-effective way.
Even if you don’t have sensor simulation capability, the images generated from hyperspectral data are richer and more accurate since the information used to calculate the color of every pixel in the final RGB image includes all wavelengths sampled by the render giving much more detail to the final result.
As opposed to other non-hyperspectral ray tracing render engines that can generate nice images but with less detail and quality.
After understanding what hyperspectral synthetic data is and why it makes a difference, let’s look at the technological pillars that support the platform and that allow the generation of this highly accurate data.
The technology behind the data
- Hyperspectral render engine
It all starts with Anyverse’s proprietary hyperspectral render engine. Anyverse implements a pure spectral path-tracing engine that computes the spectral radiance of every light beam, simulating lights, cameras, and materials with physical accuracy. This allows for a precise simulation of the amount of light reaching the camera sensor to generate a final image containing full spectral information per pixel.
Energy is simulated as an electromagnetic wave across the rendering pipeline to its final digital values. Light sources, including the sky and sun, are modeled through their characteristic spectrum profile that depends on the type of light source and temperature (LED, incandescent, etc.), and materials are also physically modeled using BSDF functions.
- Physically-based sensor simulation
Anyverse combines a physics description of lights and materials in a 3D scene with a detailed hyperspectral simulation of sensor intrinsics. The sensor pipeline calculates energy’s propagation per wavelength, considering many subtle physics effects such as the conversion of photons into voltage and RAW digital values before producing a final image.
In addition, Anyverse performs optical simulation using advanced path-tracing technology, capturing effects such as extreme optics distortion, lens shading, lens blurring (depth of field), complex assembly of multiple lenses, and more.
The platform supports a variety of sensors and lenses in visible and non-visible (near infrared).
- Scene generation
Anyverse offers almost infinite possibilities to generate custom or random scenes that allow data scientists to recreate a diverse synthetic reality in static and dynamic conditions.
To make this task easier for the user, Anyverse provides a library of predefined high-quality 3D scenes that serve as the base layer for building a complex simulation, as well as an extensive asset library to compose those scenes.
Additionally, it includes a variations engine and a complete Python API that allows for automatic endless compositions of scenarios to reduce the Content Domain Gap.
This means unprecedented power for the user who can take complete control over the scene configuration and program the scene generation itself with all desired variability conditions.
- Fine-grain control on scenarios
To give life to the scene being generated synthetically, developers can apply (with Python scripts) and combine different behaviors and actions in the scene, adding dynamical properties to the ego vehicle, pedestrians, and other assets.
Scenarios can adapt to any scene, allowing a wide range of scene/scenario variability.
- Scalable flexible architecture
Anyverse provides a modular platform that enables efficient scene definition and dataset production.
It is deployed in a cloud environment and can be easily operated by a Data Engineer without complex hardware requirements. Once the simulation conditions are set, the platform scales the simulation process automatically, obtaining the required data in a matter of hours, compared with weeks of real-world data processing.
Now that we know the powerful tech pillars on which Anyverse is built, let’s get to know its datasets in detail.
Anyverse synthetic datasets in detail
One of the main differentiators is that Anyverse datasets can include many (all or selection on-demand) of the following output channels depending on their purpose:
- Color images
- Hyperspectral images
- Raw sensor images
- LiDAR point clouds
- Annotation metadata
- Per-pixel metadata
Data scientists and ML engineers can generate custom datasets containing any of the output data below* (provided per-pixel metadata):
- Color image
- Instance segmentation
- Label segmentation
- Material segmentation
- Depth
- 3D position
- Surface normals
- Roughness
- Reflectance
- Motion vectors 2D and 3D
- Spectral radiance
*Download the The power of hyperspectral datasets eBook if you wish to read further descriptions of each output channel.
Image 8: Color RGB
Image 9: Normal
Image 10: Instance
Image 11: Label
Image 12: Depth
Image 13: Position 3D
Image samples and metadata generated with Anyverse for self-driving
At this point, we have a better understanding of why Anyverse synthetic datasets for computer vision are different. Probably the most advanced game engines or generative AI can create amazingly realistic images today, images that can fool the human eye, but are these images what the ML model needs to generalize the inputs it will receive from the real world? Do these images have accurate ground truth that does not contaminate the system? Are these images aligned with the sensors the systems will implement in the real world?
Stay tuned to know more
Stay tuned if you want to learn more about how Anyverse datasets overcome some of the challenges engineers typically face when developing a dataset.
Learn why Anyverse's synthetic datasets are more accurate and comprehensive than other available datasets
- Reduce domain gap in datasets
- Generate pixel-perfect ground truth data
- Control all environmental variables
Learn why Anyverse's synthetic datasets are more accurate and comprehensive than other available datasets
- Reduce domain gap in datasets
- Generate pixel-perfect ground truth data
- Control all environmental variables
