Freepik image
AI is booming these days.
If you haven’t had a conversation with chatGPT to give you tips on how to program that utility you need for your project and see what it comes up with. Or if you haven’t played around with DALL-E or Stable Diffusion to generate new art forms or have fun seeing pictures of you with your favorite celebrity enjoying a basketball game courtside wearing ugly Christmas sweaters, you are leaving on another planet.
AI is permeating many aspects of our life, some of those AI-based systems surrounding us are the ones that require a faithful perception of the environment to seamlessly help with various tasks, like the ADAS in your car or computer vision systems that help autonomous robots organize a warehouse.
The AI deep learning models behind the perception systems need to understand the real world by example with vast amounts of training data for specific environments and use cases including corner cases. Trying to use only real-world images may be difficult and expensive to get. That is why the use of synthetically generated data is helping improve the performance and accuracy of perception systems.
Real-world details are infinite. When generating synthetic images simulating cameras, we need to be able to reproduce and capture as many details as possible from a computer-generated 3D world as we would capture using real cameras in the real world. Don’t forget that, at the end of the day, the perception systems will use real cameras (and other sensors). Those details that we need to generate more faithful images to feed our perception brain is what we call hyperspectral data.
What is hyperspectral synthetic data?
Hyperspectral in this context means that when rendering the synthetic 3D world we have created we sample several wavelengths within a given spectrum (for cameras, visible + near-infrared part of the electromagnetic spectrum). So rays with different wavelengths travel and bounce back and forth hitting the different objects in the 3D scene reaching the camera at some point with a given energy dependent on the wavelength (Planck Dixit). All this info, the photons (rays) with different wavelengths, that reach the camera sensor for every pixel is the hyperspectral data for an image.
The more wavelengths we sample in the given spectrum, the more details we will capture, the more accurate the camera simulation can be and the better results you will get.
It is worth noticing that as important as it is that the render supports hyperspectral, is that the computer-generated 3D world and all the objects in it are characterized in a way that the render knows the behavior of every ray when it hits the objects in its way to the camera. This behavior depends on the materials and it is wavelength dependent (for those of you not in the know, these are the BRDFs or bi-directional reflectance distribution functions).
Samples of different materials simulated with Anyeverse Hyperspectral render
Why is having hyperspectral synthetic data useful?
First of all, it is the only way to physically simulate optics and sensors. It’s as simple as that. Faithfully simulating the sensor is a great way to reduce the domain gap in your synthetic datasets generating images closer to what the real cameras would take. So training with such images you have a better chance that your perception deep learning model will generalize to real-world images improving the performance of your perception system in a very cost-effective way.
Even if you don’t have sensor simulation capability, the images generated from hyperspectral data are richer and more accurate since the information used to calculate the color of every pixel in the final RGB image includes all wavelengths sampled by the render giving much more detail to the final result. As opposed to other non-hyperspectral ray tracing render engines that can generate nice images but with less detail and quality.
HYPERSPECTRAL RENDER WITH SENSOR SIMULATION
HYPERSPECTRAL RENDER WITHOUT SENSOR SIMULATION
And, if you are designing your own sensors, you can directly use the hyper spectral data to programmatically simulate the response of your sensor before implementing it on silicon. It is a powerful tool that can significantly reduce the cost and time to market of new sensors. If these sensors are going to be used as part of a perception system stack, using the hyperspectral data to fine tune its characteristics to have the best performance possible for the perception use case, will assure higher levels of confidence in the overall performance of the system.
If knowledge is power, now you have the power to decide if you want to go hyperspectral or not.
Build a robust in-cabin monitoring system to meet Euro NCAP requirements
Craft pixel-accurate synthetic data to train and validate your DMS & OMS with Anyverse Synthetic Data platform for in-cabin monitoring AI.
About Anyverse™
Anyverse™ helps you continuously improve your deep learning perception models to reduce your system’s time to market applying new software 2.0 processes. Our synthetic data production platform allows us to provide high-fidelity accurate and balanced datasets. Along with a data-driven iterative process, we can help you reach the required model performance.
With Anyverse™, you can accurately simulate any camera sensor and help you decide which one will perform better with your perception system. No more complex and expensive experiments with real devices, thanks to our state-of-the-art photometric pipeline.
Need to know more?
