Coverage Isn’t the Problem: Synchronization in Multi-Sensor In-Cabin Synthetic Data
Vehicle cabin interior with a driver and a child in a rear-facing child restraint system, beside the text Four Sensors, One Moment
SHARE

Table of Contents

Table of Contents

In-Cabin Monitoring Has Outgrown a Single Sensor

The regulatory timeline is precise, and the precision matters. Under the EU’s General Safety Regulation, Regulation (EU) 2019/2144, Driver Drowsiness and Attention Warning systems became mandatory for new vehicle types on 6 July 2022 and for all new vehicles on 7 July 2024. Advanced Driver Distraction Warning followed the same phasing: new types from 7 July 2024, all new vehicles from 7 July 2026, a deadline that has now passed. GSR mandates. Euro NCAP awards points separately.

Euro NCAP’s 2026 protocols introduce Safe Driving as a new Stage of Safety, scored out of 100 points, part of the restructuring set out in Euro NCAP’s Vision 2030 roadmap, split across Occupant Monitoring, Driver Engagement, and Vehicle Assistance. Inside Occupant Monitoring sits the detail that best illustrates what has changed. To earn the points available for direct child presence detection (CPD), a system has to confirm a child through movement, breathing, heartbeat, or another sign of life. Where a child is locked in, the warning must start as soon as possible after locking, with no more than 15 seconds allowed for the system to assess occupancy. Where a child has entered an unlocked vehicle, the initial warning may follow door closure by up to 10 minutes. The criteria are set out in Section 3.1 of the Euro NCAP Safe Driving Occupant Monitoring Protocol v1.2 (July 2026, implementation January 2026). In the open case, with an awake child, a camera can satisfy that criterion — visible limb or head movement is enough. The constraint arises with the test conditions in Euro NCAP Technical Bulletin SD-102 (CPD evaluation, v2.0): the sleeping newborn, one-year-old and three-year-old are tested under a blanket of at least 70 × 90 cm and 300 GSM, in cotton or polyester, and sleeping children are modelled with respiratory movement only. With the child covered and still, the visual signal a camera depends on is largely obscured, while radar’s micro-motion sensing keeps working through fabric. 

None of these protocols specify sensor fusion. They specify phenomena, and no single modality observes all of them well. Breathing and heartbeat are a radar micro-motion problem. Gaze direction, eyelid dynamics, and occlusion tolerance under sunglasses, hats, or masks are a near-infrared (NIR) camera problem. Seatbelt routing, occupant stature classification, and out-of-position detection are geometric problems, best served by depth. Sensor fusion driver monitoring systems (DMS) — stacks combining two or more of these modalities into a single perception pipeline — did not emerge from an architecture preference. They emerged because the requirement set stopped being solvable by one sensor.

That shift changes the interesting question. It is no longer which sensors a multi-sensor in-cabin monitoring system should carry; programmes are converging on combinations of NIR, RGB, radar, and depth already. It is how a training programme gets data in which all of those sensors are describing the same moment.

What Sensor Fusion DMS Training Actually Requires

A fusion model does not learn from four datasets. It learns from co-registered observations of one physical state, and the difference between those two things is where sensor fusion DMS programmes lose ground before a single epoch runs.

Call them the three co-registration requirements. All three have to hold across every modality:

  • Temporal — the modalities describe the same instant, not adjacent instants.
  • Spatial — extrinsics between sensors are known and stay stable across the dataset.
  • Ontological — one shared ground truth definition across modalities, not per-modality labels reconciled after the fact.
Three requirements listed in stacked rows: temporal and spatial, each marked as partially addressed by hardware, and ontological, marked as not addressed by hardware

The first two are well understood, and hardware gets you partway there: synchronized triggers address timing, careful rig design addresses geometry. The third is different in kind. No amount of hardware synchronization produces a shared definition of what is actually happening in the scene. You have to design that into the data itself, or it doesn’t exist.

That claim needs testing against how multi-sensor programmes actually run.

Where Multi-Sensor Ground Truth Comes Apart

The familiar argument for synthetic data is coverage — that isn’t the argument here.

What is specific to multi-sensor programmes is structural, not statistical. Four reasons alignment degrades before training even begins:

  • Separate capture campaigns per modality — often different rigs, different vehicles, different subjects, different schedules.
  • Separate annotation pipelines — frequently different vendors, each with its own per-modality label schema.
  • Calibration drift across the life of a campaign — thermal, mechanical, and from re-mounting sensors between sessions.
  • No way to re-shoot one modality in isolation. Move the NIR illuminator, and you have to recollect the radar and depth data too, or the dataset becomes internally inconsistent.

The obvious objection is a hardware-synchronized rig, which appears to dispose of two of the three co-registration requirements outright — triggers line up the timestamps, a rigid mount holds the extrinsics. But that guarantee is scoped to the rig that was built: change the illuminator position, swap the radar, move the camera for a new trim level, and the guarantee expires rather than migrates. The dataset still carries aligned timestamps; it just no longer describes the sensor stack being trained. Programmes that budget for one campaign and then iterate through five learn this late.

Ground truth definition is not even on that list, because hardware never touched it. Here is one occupant, a child in a rear-facing child restraint system (CRS), partially covered by a blanket:

  • The radar ground truth is a micro-motion signature, integrated over several seconds.
  • The NIR ground truth is a per-frame occupancy and keypoint annotation.
  • The depth ground truth is a per-frame volumetric occupancy.
Timeline with three lanes: a continuous radar band, and two rows of per-frame tick marks for NIR camera and ToF depth, offset from each other

Three time bases, three label ontologies, one child. Somewhere downstream, someone decides how those three descriptions reconcile into a single training label, and that decision is a modeling assumption, made by a person, that never gets written down anywhere the model or its validators can inspect.

The label noise this produces is not spread evenly across the dataset. It concentrates on moving, occluded, and non-standard occupants, which is precisely the population the programme was built to handle in the first place.

Simultaneity Is a Property of the Generation Method

“Use synthetic data” is not, by itself, the fix, because several common ways of generating multi-sensor synthetic data reintroduce the same reconciliation problem:

  • Generating modalities sequentially rather than in one pass, then stitching them together afterward.
  • Approximating one sensor’s output from another — deriving a NIR frame from an RGB render, or depth from stereo matching on rendered frames.
  • Reconstruction-based pipelines encode appearance as it was captured under the recorded conditions and cannot extrapolate to a sensor configuration that was never built.

Each of these produces four outputs that look synchronized. None of them guarantees that the four outputs describe the same physical event, because the modalities were never generated from one shared description of the scene. They were derived, adjusted, or reconstructed into apparent agreement.

The requirement, then, is narrower than “synthetic data” as a category: one shared physical scene state, with every modality derived from it in the same generation pass. The generation method must build in simultaneity, not check or correct it downstream.

That is necessary and not sufficient. What is shared has to be physical rather than pixels, and that distinction is where multi-sensor synthetic data pipelines fail without anyone noticing.

Photorealism Is Not Sensor Accuracy

You cannot derive a NIR frame from an RGB render, and the reason matters more than the claim itself. Four sensors, four separate physics problems.

NIR reflectance and visible-band appearance are uncorrelated

NIR response depends on material reflectance at 850 or 940 nm, and reflectance in those bands is something visible-band appearance says nothing reliable about:

  • Two fabrics that look identical under visible light — a cotton and a polyester child-restraint-system cover, or a dark jacket against dark upholstery — can diverge substantially once measured in-band.
  • Skin, hair, cosmetics, and eyeglass coatings each carry their own NIR reflectance signature, independent of how they look.
  • A scene authored to look correct in RGB carries no information about any of this.
The illuminator is part of the sensor

A NIR DMS is an active-illumination system, not a passive camera pointed at ambient light. What the camera captures is determined jointly by the scene and by the illuminator: its radiant intensity distribution, mounting position, wavelength, and synchronization with the camera’s exposure window. Wavelength is the clearest example of why that matters. Silicon quantum efficiency at 940 nm runs at roughly half its 850 nm value, so 850 nm delivers more signal for the same illuminator power and supports shorter exposures — but 850 nm LEDs produce a faint visible red glow in a dark cabin, and 940 nm sits inside a solar irradiance dip caused by atmospheric water-vapor absorption, so ambient sunlight interferes substantially less. In-cabin systems have largely converged on 940 nm despite the quantum-efficiency penalty. A pipeline that treats NIR as a band rather than as a specific illuminator at a specific wavelength cannot represent that choice at all. The resulting failure modes are ones a sensor engineer will recognize immediately — hotspots across the nose bridge and forehead where irradiance peaks, falloff toward the cabin edges where it doesn’t reach, specular return off eyeglass lenses saturating the eye region exactly where the system needs signal most. Move the illuminator a few centimeters, and the failure surface changes entirely, not because the scene changed but because the sensor did. NIR sensor simulation that does not model the illuminator is not modeling the sensor.

The optics-to-ISP chain determines what signal survives

Between the photons and the label sits a chain: lens transmission (wavelength-dependent), vignetting, point spread, and IR-cut filter behavior on the way in; quantum efficiency, read noise, shot noise, and dark current at the sensor; gain and tone mapping in the ISP on the way out. In low light, SNR degrades sharply, and the noise characteristics of the sensor start to dominate the image the model is trained on. A generation pipeline that does not model that noise chain is training a model on a cabin no physical camera will ever deliver.

Depth is an active measurement, not a geometry read-out

A time-of-flight (ToF) depth sensor — the in-cabin norm, measuring phase shift rather than pulse return time — has its own illuminator and its own failure surface, and neither is inherited from the cabin’s geometry.

It emits in the same NIR band the DMS camera works in, so the two illuminators interfere; strong ambient NIR through a side window raises the noise floor on the depth measurement. Multipath inside a reflective cabin biases range estimates on exactly the surfaces that matter for seatbelt routing and occupant stature classification — glossy trim, glass, the moulded shell of a rear-facing CRS — and at silhouette boundaries, where the occupant’s outline meets the seat behind, flying pixels put returns at ranges nothing occupies.

A depth map synthesized from scene geometry alone carries none of this. It describes the cabin, not what the depth sensor reports about the cabin.

Radar is not a rendering mode

Everything above is still, at some level, image formation. Radar is not. A frequency-modulated continuous-wave (FMCW) waveform, an antenna pattern, propagation and multipath inside a cabin that is mostly metal and glass, the radar cross-section of an occupant and of CRS materials, micro-Doppler produced by chest displacement of a few millimetres — none of it has an equivalent in a camera pipeline, however sophisticated. Cabin multipath is the hard part of FMCW radar simulation, and it is a wave-propagation problem to be solved on its own physical terms, not a shading problem that happens to involve radio waves.

Table 1. What determines each modality’s output in a multi-sensor in-cabin monitoring stack, and what goes wrong when the sensor is left unmodeled.

Modality

What determines the output

What changes it

Failure mode if unmodeled

NIR camera

Material reflectance at 850/940 nm, illuminator geometry, optics and noise chain, ISP gain

Illuminator position, wavelength, exposure sync, low-light noise floor

Hotspots, edge falloff, specular saturation off eyewear, training on noise-free renders

RGB camera

Visible-spectrum reflectance, ambient and cabin lighting, optics and noise chain, ISP tone mapping

Ambient light, exposure, tone mapping, low-light noise floor

Tone-mapped renders stand in for what the sensor and ISP actually output

ToF depth

Active NIR emission and phase-based range measurement, surface reflectivity, cabin geometry

Ambient NIR load, reflective trim and glass, range to occupant

Multipath range bias, flying pixels at silhouettes, rear-row precision loss

FMCW radar

Antenna pattern, propagation and multipath, radar cross-section

Cabin geometry, occupant material, chest micro-motion

Multipath and micro-Doppler effects with no camera equivalent

Take a single frame that is correct by every measure an RGB pipeline checks: the face well exposed, the expression legible, no motion blur. At that same instant, a NIR DMS evaluated on that same scene fails, because the illuminator’s return off an eyeglass lens saturates the eye region and the system has no eye state left to report. Nothing in the RGB frame predicts this. It is a NIR-specific, illumination-specific failure, invisible from any other modality’s point of view.

Simultaneity is therefore a physics result, not a file-format one. One scene state, evaluated through a distinct physical transfer model per sensor, all at the same instant. A pipeline that renders attractive views of the same scene has not synchronized four sensors. It has synchronized four approximations of them.

Spectral Rendering From a Single Physical Scene State

Scene state, defined once. Before any sensor model runs, the scene itself is specified as a single physical description: cabin geometry, the occupant’s body model and pose, the child restraint system if one is present, materials assigned spectral reflectance properties rather than a visible-light color, and light sources defined by spectral power distribution and physical placement rather than by how a render should look. Every downstream modality reads from this one description. None of it is authored per-sensor.

Spectral rendering is the mechanism, not a detail. Radiance is computed per wavelength band across the scene, and each sensor model then samples that shared radiance field through its own response curve. This is why NIR is not a filter applied to an RGB render. NIR and RGB are not two views of the same image; they are two samples of the same underlying physical radiance, taken through two different response curves.

Four sensor models draw from that same spectral scene state in the same generation pass:

  • NIR camera — illuminator model, spectral quantum efficiency, optics, noise, ISP.
  • RGB camera — the same optical pipeline, sampled in the visible band. The relevant distinction is not RGB versus NIR as separate categories; it is aesthetic rendering versus spectral rendering with a modeled sensor behind it.
  • ToF depth — the emission, phase-measurement, and multipath behavior described above, applied to the same cabin and occupant geometry, not re-authored separately.
  • FMCW radar — propagation and material-response modeling applied to the same occupant pose and CRS geometry the camera models are reading from.

This is the physics-based synthetic data pipeline Anyverse INCABIN implements: a single scene state, sampled through four sensor-specific physical models, in one pass.

Annotations are projections, not interpretations. 3D occupant pose and keypoints, per-seat occupancy and class, gaze vector, eyelid aperture, seatbelt routing, occlusion state, and material assignments all exist as scene state already. They are not derived by a person or a model looking at each sensor’s output afterward. Every modality’s labels are a projection of that same shared scene description. They agree by construction, and they share one ontology, because there was only ever one ontology to begin with. That is the ontological requirement, the one hardware synchronization could not touch.

There is nothing to align, because nothing was captured separately in the first place.

The programme economics follow as arithmetic, not as a claim. Today, N modalities mean N capture campaigns, N annotation pipelines, N calibration regimes, and N recollections every time a single sensor’s spec changes. A shared scene state collapses that structure to one generation pass across all of them. Move the illuminator position and regenerate: the radar and depth ground truth do not shift, because the occupant never moved. What changed was one sensor’s model, not the scene.

What Changes in Training and Validation

A mechanism is only interesting for what it removes. Five things change, and they are specific.

Label noise on moving occupants disappears. When modalities are captured on different time bases, error concentrates exactly in the cases with motion or occlusion, the same cases that drive most of a programme’s real risk. When every modality describes the same instant instead, that source of error is not reduced. It is not present.

The fusion head stops learning the rig. Extrinsic calibration error between sensors, however small, propagates into the fusion layer. A model trained against it does not just tolerate the error; it learns to compensate for one specific vehicle’s sensor geometry. That compensation does not transfer to the next programme’s sensor layout, and it stays invisible until it fails there, because nothing flagged it as an error the first time.

The ontology stops being a lowest common denominator. When labels are reconciled across modalities after collection, teams under time pressure default to whichever schema every modality’s annotators can agree on, usually the coarsest one available. Detail an individual sensor was capable of contributing gets discarded at the reconciliation step, not because that detail was uncollectible but because reconciling it across three schemas was more work than the deadline allowed.

Sensor placement iteration stops meaning a rig rebuild. Illuminator position, camera field of view, radar mounting angle normally mean rebuilding a rig and recollecting; with a shared scene state, they mean regenerating a parameter instead.

Validation evidence becomes parameterized. Sweeps state physical units a reviewer can inspect — illuminance swept from 50 to 100,000 lux at defined spectral profiles, illuminator irradiance and mounting geometry, occupant stature by percentile, eyewear transmittance, occlusion state, blanket coverage in the CRS case — evidence an OEM builds its own argument on against DMS, occupant monitoring system (OMS), and CPD test cases, not a compliance claim made on its behalf.

What This Does Not Solve

That is the list of what changes. This is the list of what does not, and it is the more important of the two for anyone planning a programme against it.

Synthetic data does not remove the need for real-world data. It changes what real data is for. Once bulk coverage stops being the constraint, real-world collection moves to two jobs it is actually suited for: correlating each sensor model against real hardware, and final sign-off. Both stay real-world tasks; neither goes away.

The domain gap is not the same size across modalities. A well-modeled NIR camera and a well-modeled in-cabin radar do not carry the same residual gap to real hardware. Radar’s propagation physics — wave reflection inside a metal-and-glass cabin, cross-section variation across materials — is harder to model to the same fidelity as a camera’s optical chain, and there is no widely agreed method for measuring that sim-to-real gap in automotive radar. Camera modalities close faster than radar does, and a programme that schedules both on the same maturity timeline is scheduling against physics it cannot change.

Internal consistency is not correctness. A shared scene state guarantees that four modalities agree with each other. It says nothing about whether any one of them agrees with the sensor that will actually ship. Each sensor model still has to be validated against measured output from real hardware. That sensor model validation step is a methodology requirement, not an optional check, and it is where automotive sensor-modeling experience actually matters, rather than being a credential attached to a pitch.

One error source removed leaves the rest untouched. Removing modality misalignment removes one specific source of error from a fusion stack. It does not fix a fusion architecture that is the wrong shape for the problem, does not rebalance a class distribution that is still skewed, and does not replace the engineering judgment of deciding how modalities should be weighted against each other. That work is unaffected by where the training data came from.

Synchronization, Not Just Coverage

The standard case for synthetic data in-cabin is coverage: filling in the rare classes and edge cases that real-world collection under-samples by construction. That case is true, and it is not specific to any one vendor. Every company in this category makes some version of it.

The multi-sensor case made across this piece is different in kind. Simultaneity is not a matter of collection budget or campaign scale that a larger real-world programme could eventually buy its way toward. A synchronized rig buys temporal and spatial co-registration once, for one sensor configuration, and the purchase expires the moment that configuration changes. The ontological requirement was never for sale at all, because the modalities were never observing one defined physical state to begin with.

And simultaneity only counts if every modality is physically correct at that same instant. Four renders that line up on a timestamp but were reconstructed from one another are still four approximations. Synchronization without physical accuracy is coincidence, not a fusion dataset.

Enjoyed this article?

Add us as a Preferred Source on Google to see more content like this in Discover and Top Stories.

About Anyverse

With over 25 years of heritage in physics-based simulation and a 2008 Academy Technical Achievement Award, Anyverse is a leading provider of high-fidelity synthetic data and validation tools for advanced Perception AI.

While Anyverse remains a premier partner for the Automotive market, specializing in In-Cabin Monitoring systems and ADAS, its technology is increasingly adopted across Defence, Robotics, and Industry 4.0. Leveraging a proprietary physics-grounded engine, procedural synthetic data generation, and next-gen Generative AI, Anyverse produces datasets with unmatched realism and sensor-level accuracy (including RGB, NIR, Radar, LiDAR, and Thermal).

Its unified validation framework helps partners ensure compliance with specific regulations like Euro NCAP, as well as broader safety standards for autonomous systems. By bridging the gap between simulation and reality, Anyverse accelerates development, reduces costs, and ensures trustworthy perception for vehicles, robots, and intelligent machines.

Coverage Isn’t the Problem: Synchronization in Multi-Sensor In-Cabin Synthetic Data

Vehicle cabin interior with a driver and a child in a rear-facing child restraint system, beside the text Four Sensors, One Moment
SHARE

Table of Contents

Table of Contents

In-Cabin Monitoring Has Outgrown a Single Sensor

The regulatory timeline is precise, and the precision matters. Under the EU’s General Safety Regulation, Regulation (EU) 2019/2144, Driver Drowsiness and Attention Warning systems became mandatory for new vehicle types on 6 July 2022 and for all new vehicles on 7 July 2024. Advanced Driver Distraction Warning followed the same phasing: new types from 7 July 2024, all new vehicles from 7 July 2026, a deadline that has now passed. GSR mandates. Euro NCAP awards points separately.

Euro NCAP’s 2026 protocols introduce Safe Driving as a new Stage of Safety, scored out of 100 points, part of the restructuring set out in Euro NCAP’s Vision 2030 roadmap, split across Occupant Monitoring, Driver Engagement, and Vehicle Assistance. Inside Occupant Monitoring sits the detail that best illustrates what has changed. To earn the points available for direct child presence detection (CPD), a system has to confirm a child through movement, breathing, heartbeat, or another sign of life. Where a child is locked in, the warning must start as soon as possible after locking, with no more than 15 seconds allowed for the system to assess occupancy. Where a child has entered an unlocked vehicle, the initial warning may follow door closure by up to 10 minutes. The criteria are set out in Section 3.1 of the Euro NCAP Safe Driving Occupant Monitoring Protocol v1.2 (July 2026, implementation January 2026). In the open case, with an awake child, a camera can satisfy that criterion — visible limb or head movement is enough. The constraint arises with the test conditions in Euro NCAP Technical Bulletin SD-102 (CPD evaluation, v2.0): the sleeping newborn, one-year-old and three-year-old are tested under a blanket of at least 70 × 90 cm and 300 GSM, in cotton or polyester, and sleeping children are modelled with respiratory movement only. With the child covered and still, the visual signal a camera depends on is largely obscured, while radar’s micro-motion sensing keeps working through fabric. 

None of these protocols specify sensor fusion. They specify phenomena, and no single modality observes all of them well. Breathing and heartbeat are a radar micro-motion problem. Gaze direction, eyelid dynamics, and occlusion tolerance under sunglasses, hats, or masks are a near-infrared (NIR) camera problem. Seatbelt routing, occupant stature classification, and out-of-position detection are geometric problems, best served by depth. Sensor fusion driver monitoring systems (DMS) — stacks combining two or more of these modalities into a single perception pipeline — did not emerge from an architecture preference. They emerged because the requirement set stopped being solvable by one sensor.

That shift changes the interesting question. It is no longer which sensors a multi-sensor in-cabin monitoring system should carry; programmes are converging on combinations of NIR, RGB, radar, and depth already. It is how a training programme gets data in which all of those sensors are describing the same moment.

What Sensor Fusion DMS Training Actually Requires

A fusion model does not learn from four datasets. It learns from co-registered observations of one physical state, and the difference between those two things is where sensor fusion DMS programmes lose ground before a single epoch runs.

Call them the three co-registration requirements. All three have to hold across every modality:

  • Temporal — the modalities describe the same instant, not adjacent instants.
  • Spatial — extrinsics between sensors are known and stay stable across the dataset.
  • Ontological — one shared ground truth definition across modalities, not per-modality labels reconciled after the fact.
Three requirements listed in stacked rows: temporal and spatial, each marked as partially addressed by hardware, and ontological, marked as not addressed by hardware

The first two are well understood, and hardware gets you partway there: synchronized triggers address timing, careful rig design addresses geometry. The third is different in kind. No amount of hardware synchronization produces a shared definition of what is actually happening in the scene. You have to design that into the data itself, or it doesn’t exist.

That claim needs testing against how multi-sensor programmes actually run.

Where Multi-Sensor Ground Truth Comes Apart

The familiar argument for synthetic data is coverage — that isn’t the argument here.

What is specific to multi-sensor programmes is structural, not statistical. Four reasons alignment degrades before training even begins:

  • Separate capture campaigns per modality — often different rigs, different vehicles, different subjects, different schedules.
  • Separate annotation pipelines — frequently different vendors, each with its own per-modality label schema.
  • Calibration drift across the life of a campaign — thermal, mechanical, and from re-mounting sensors between sessions.
  • No way to re-shoot one modality in isolation. Move the NIR illuminator, and you have to recollect the radar and depth data too, or the dataset becomes internally inconsistent.

The obvious objection is a hardware-synchronized rig, which appears to dispose of two of the three co-registration requirements outright — triggers line up the timestamps, a rigid mount holds the extrinsics. But that guarantee is scoped to the rig that was built: change the illuminator position, swap the radar, move the camera for a new trim level, and the guarantee expires rather than migrates. The dataset still carries aligned timestamps; it just no longer describes the sensor stack being trained. Programmes that budget for one campaign and then iterate through five learn this late.

Ground truth definition is not even on that list, because hardware never touched it. Here is one occupant, a child in a rear-facing child restraint system (CRS), partially covered by a blanket:

  • The radar ground truth is a micro-motion signature, integrated over several seconds.
  • The NIR ground truth is a per-frame occupancy and keypoint annotation.
  • The depth ground truth is a per-frame volumetric occupancy.
Timeline with three lanes: a continuous radar band, and two rows of per-frame tick marks for NIR camera and ToF depth, offset from each other

Three time bases, three label ontologies, one child. Somewhere downstream, someone decides how those three descriptions reconcile into a single training label, and that decision is a modeling assumption, made by a person, that never gets written down anywhere the model or its validators can inspect.

The label noise this produces is not spread evenly across the dataset. It concentrates on moving, occluded, and non-standard occupants, which is precisely the population the programme was built to handle in the first place.

Simultaneity Is a Property of the Generation Method

“Use synthetic data” is not, by itself, the fix, because several common ways of generating multi-sensor synthetic data reintroduce the same reconciliation problem:

  • Generating modalities sequentially rather than in one pass, then stitching them together afterward.
  • Approximating one sensor’s output from another — deriving a NIR frame from an RGB render, or depth from stereo matching on rendered frames.
  • Reconstruction-based pipelines encode appearance as it was captured under the recorded conditions and cannot extrapolate to a sensor configuration that was never built.

Each of these produces four outputs that look synchronized. None of them guarantees that the four outputs describe the same physical event, because the modalities were never generated from one shared description of the scene. They were derived, adjusted, or reconstructed into apparent agreement.

The requirement, then, is narrower than “synthetic data” as a category: one shared physical scene state, with every modality derived from it in the same generation pass. The generation method must build in simultaneity, not check or correct it downstream.

That is necessary and not sufficient. What is shared has to be physical rather than pixels, and that distinction is where multi-sensor synthetic data pipelines fail without anyone noticing.

Photorealism Is Not Sensor Accuracy

You cannot derive a NIR frame from an RGB render, and the reason matters more than the claim itself. Four sensors, four separate physics problems.

NIR reflectance and visible-band appearance are uncorrelated

NIR response depends on material reflectance at 850 or 940 nm, and reflectance in those bands is something visible-band appearance says nothing reliable about:

  • Two fabrics that look identical under visible light — a cotton and a polyester child-restraint-system cover, or a dark jacket against dark upholstery — can diverge substantially once measured in-band.
  • Skin, hair, cosmetics, and eyeglass coatings each carry their own NIR reflectance signature, independent of how they look.
  • A scene authored to look correct in RGB carries no information about any of this.
The illuminator is part of the sensor

A NIR DMS is an active-illumination system, not a passive camera pointed at ambient light. What the camera captures is determined jointly by the scene and by the illuminator: its radiant intensity distribution, mounting position, wavelength, and synchronization with the camera’s exposure window. Wavelength is the clearest example of why that matters. Silicon quantum efficiency at 940 nm runs at roughly half its 850 nm value, so 850 nm delivers more signal for the same illuminator power and supports shorter exposures — but 850 nm LEDs produce a faint visible red glow in a dark cabin, and 940 nm sits inside a solar irradiance dip caused by atmospheric water-vapor absorption, so ambient sunlight interferes substantially less. In-cabin systems have largely converged on 940 nm despite the quantum-efficiency penalty. A pipeline that treats NIR as a band rather than as a specific illuminator at a specific wavelength cannot represent that choice at all. The resulting failure modes are ones a sensor engineer will recognize immediately — hotspots across the nose bridge and forehead where irradiance peaks, falloff toward the cabin edges where it doesn’t reach, specular return off eyeglass lenses saturating the eye region exactly where the system needs signal most. Move the illuminator a few centimeters, and the failure surface changes entirely, not because the scene changed but because the sensor did. NIR sensor simulation that does not model the illuminator is not modeling the sensor.

The optics-to-ISP chain determines what signal survives

Between the photons and the label sits a chain: lens transmission (wavelength-dependent), vignetting, point spread, and IR-cut filter behavior on the way in; quantum efficiency, read noise, shot noise, and dark current at the sensor; gain and tone mapping in the ISP on the way out. In low light, SNR degrades sharply, and the noise characteristics of the sensor start to dominate the image the model is trained on. A generation pipeline that does not model that noise chain is training a model on a cabin no physical camera will ever deliver.

Depth is an active measurement, not a geometry read-out

A time-of-flight (ToF) depth sensor — the in-cabin norm, measuring phase shift rather than pulse return time — has its own illuminator and its own failure surface, and neither is inherited from the cabin’s geometry.

It emits in the same NIR band the DMS camera works in, so the two illuminators interfere; strong ambient NIR through a side window raises the noise floor on the depth measurement. Multipath inside a reflective cabin biases range estimates on exactly the surfaces that matter for seatbelt routing and occupant stature classification — glossy trim, glass, the moulded shell of a rear-facing CRS — and at silhouette boundaries, where the occupant’s outline meets the seat behind, flying pixels put returns at ranges nothing occupies.

A depth map synthesized from scene geometry alone carries none of this. It describes the cabin, not what the depth sensor reports about the cabin.

Radar is not a rendering mode

Everything above is still, at some level, image formation. Radar is not. A frequency-modulated continuous-wave (FMCW) waveform, an antenna pattern, propagation and multipath inside a cabin that is mostly metal and glass, the radar cross-section of an occupant and of CRS materials, micro-Doppler produced by chest displacement of a few millimetres — none of it has an equivalent in a camera pipeline, however sophisticated. Cabin multipath is the hard part of FMCW radar simulation, and it is a wave-propagation problem to be solved on its own physical terms, not a shading problem that happens to involve radio waves.

Table 1. What determines each modality’s output in a multi-sensor in-cabin monitoring stack, and what goes wrong when the sensor is left unmodeled.

Modality

What determines the output

What changes it

Failure mode if unmodeled

NIR camera

Material reflectance at 850/940 nm, illuminator geometry, optics and noise chain, ISP gain

Illuminator position, wavelength, exposure sync, low-light noise floor

Hotspots, edge falloff, specular saturation off eyewear, training on noise-free renders

RGB camera

Visible-spectrum reflectance, ambient and cabin lighting, optics and noise chain, ISP tone mapping

Ambient light, exposure, tone mapping, low-light noise floor

Tone-mapped renders stand in for what the sensor and ISP actually output

ToF depth

Active NIR emission and phase-based range measurement, surface reflectivity, cabin geometry

Ambient NIR load, reflective trim and glass, range to occupant

Multipath range bias, flying pixels at silhouettes, rear-row precision loss

FMCW radar

Antenna pattern, propagation and multipath, radar cross-section

Cabin geometry, occupant material, chest micro-motion

Multipath and micro-Doppler effects with no camera equivalent

Take a single frame that is correct by every measure an RGB pipeline checks: the face well exposed, the expression legible, no motion blur. At that same instant, a NIR DMS evaluated on that same scene fails, because the illuminator’s return off an eyeglass lens saturates the eye region and the system has no eye state left to report. Nothing in the RGB frame predicts this. It is a NIR-specific, illumination-specific failure, invisible from any other modality’s point of view.

Simultaneity is therefore a physics result, not a file-format one. One scene state, evaluated through a distinct physical transfer model per sensor, all at the same instant. A pipeline that renders attractive views of the same scene has not synchronized four sensors. It has synchronized four approximations of them.

Spectral Rendering From a Single Physical Scene State

Scene state, defined once. Before any sensor model runs, the scene itself is specified as a single physical description: cabin geometry, the occupant’s body model and pose, the child restraint system if one is present, materials assigned spectral reflectance properties rather than a visible-light color, and light sources defined by spectral power distribution and physical placement rather than by how a render should look. Every downstream modality reads from this one description. None of it is authored per-sensor.

Spectral rendering is the mechanism, not a detail. Radiance is computed per wavelength band across the scene, and each sensor model then samples that shared radiance field through its own response curve. This is why NIR is not a filter applied to an RGB render. NIR and RGB are not two views of the same image; they are two samples of the same underlying physical radiance, taken through two different response curves.

Four sensor models draw from that same spectral scene state in the same generation pass:

  • NIR camera — illuminator model, spectral quantum efficiency, optics, noise, ISP.
  • RGB camera — the same optical pipeline, sampled in the visible band. The relevant distinction is not RGB versus NIR as separate categories; it is aesthetic rendering versus spectral rendering with a modeled sensor behind it.
  • ToF depth — the emission, phase-measurement, and multipath behavior described above, applied to the same cabin and occupant geometry, not re-authored separately.
  • FMCW radar — propagation and material-response modeling applied to the same occupant pose and CRS geometry the camera models are reading from.

This is the physics-based synthetic data pipeline Anyverse INCABIN implements: a single scene state, sampled through four sensor-specific physical models, in one pass.

Annotations are projections, not interpretations. 3D occupant pose and keypoints, per-seat occupancy and class, gaze vector, eyelid aperture, seatbelt routing, occlusion state, and material assignments all exist as scene state already. They are not derived by a person or a model looking at each sensor’s output afterward. Every modality’s labels are a projection of that same shared scene description. They agree by construction, and they share one ontology, because there was only ever one ontology to begin with. That is the ontological requirement, the one hardware synchronization could not touch.

There is nothing to align, because nothing was captured separately in the first place.

The programme economics follow as arithmetic, not as a claim. Today, N modalities mean N capture campaigns, N annotation pipelines, N calibration regimes, and N recollections every time a single sensor’s spec changes. A shared scene state collapses that structure to one generation pass across all of them. Move the illuminator position and regenerate: the radar and depth ground truth do not shift, because the occupant never moved. What changed was one sensor’s model, not the scene.

What Changes in Training and Validation

A mechanism is only interesting for what it removes. Five things change, and they are specific.

Label noise on moving occupants disappears. When modalities are captured on different time bases, error concentrates exactly in the cases with motion or occlusion, the same cases that drive most of a programme’s real risk. When every modality describes the same instant instead, that source of error is not reduced. It is not present.

The fusion head stops learning the rig. Extrinsic calibration error between sensors, however small, propagates into the fusion layer. A model trained against it does not just tolerate the error; it learns to compensate for one specific vehicle’s sensor geometry. That compensation does not transfer to the next programme’s sensor layout, and it stays invisible until it fails there, because nothing flagged it as an error the first time.

The ontology stops being a lowest common denominator. When labels are reconciled across modalities after collection, teams under time pressure default to whichever schema every modality’s annotators can agree on, usually the coarsest one available. Detail an individual sensor was capable of contributing gets discarded at the reconciliation step, not because that detail was uncollectible but because reconciling it across three schemas was more work than the deadline allowed.

Sensor placement iteration stops meaning a rig rebuild. Illuminator position, camera field of view, radar mounting angle normally mean rebuilding a rig and recollecting; with a shared scene state, they mean regenerating a parameter instead.

Validation evidence becomes parameterized. Sweeps state physical units a reviewer can inspect — illuminance swept from 50 to 100,000 lux at defined spectral profiles, illuminator irradiance and mounting geometry, occupant stature by percentile, eyewear transmittance, occlusion state, blanket coverage in the CRS case — evidence an OEM builds its own argument on against DMS, occupant monitoring system (OMS), and CPD test cases, not a compliance claim made on its behalf.

What This Does Not Solve

That is the list of what changes. This is the list of what does not, and it is the more important of the two for anyone planning a programme against it.

Synthetic data does not remove the need for real-world data. It changes what real data is for. Once bulk coverage stops being the constraint, real-world collection moves to two jobs it is actually suited for: correlating each sensor model against real hardware, and final sign-off. Both stay real-world tasks; neither goes away.

The domain gap is not the same size across modalities. A well-modeled NIR camera and a well-modeled in-cabin radar do not carry the same residual gap to real hardware. Radar’s propagation physics — wave reflection inside a metal-and-glass cabin, cross-section variation across materials — is harder to model to the same fidelity as a camera’s optical chain, and there is no widely agreed method for measuring that sim-to-real gap in automotive radar. Camera modalities close faster than radar does, and a programme that schedules both on the same maturity timeline is scheduling against physics it cannot change.

Internal consistency is not correctness. A shared scene state guarantees that four modalities agree with each other. It says nothing about whether any one of them agrees with the sensor that will actually ship. Each sensor model still has to be validated against measured output from real hardware. That sensor model validation step is a methodology requirement, not an optional check, and it is where automotive sensor-modeling experience actually matters, rather than being a credential attached to a pitch.

One error source removed leaves the rest untouched. Removing modality misalignment removes one specific source of error from a fusion stack. It does not fix a fusion architecture that is the wrong shape for the problem, does not rebalance a class distribution that is still skewed, and does not replace the engineering judgment of deciding how modalities should be weighted against each other. That work is unaffected by where the training data came from.

Synchronization, Not Just Coverage

The standard case for synthetic data in-cabin is coverage: filling in the rare classes and edge cases that real-world collection under-samples by construction. That case is true, and it is not specific to any one vendor. Every company in this category makes some version of it.

The multi-sensor case made across this piece is different in kind. Simultaneity is not a matter of collection budget or campaign scale that a larger real-world programme could eventually buy its way toward. A synchronized rig buys temporal and spatial co-registration once, for one sensor configuration, and the purchase expires the moment that configuration changes. The ontological requirement was never for sale at all, because the modalities were never observing one defined physical state to begin with.

And simultaneity only counts if every modality is physically correct at that same instant. Four renders that line up on a timestamp but were reconstructed from one another are still four approximations. Synchronization without physical accuracy is coincidence, not a fusion dataset.

Enjoyed this article?

Tap to add us as a Preferred Source on Google to see more content like this.

About Anyverse

With over 25 years of heritage in physics-based simulation and a 2008 Academy Technical Achievement Award, Anyverse is a leading provider of high-fidelity synthetic data and validation tools for advanced Perception AI.

While Anyverse remains a premier partner for the Automotive market, specializing in In-Cabin Monitoring systems and ADAS, its technology is increasingly adopted across Defence, Robotics, and Industry 4.0. Leveraging a proprietary physics-grounded engine, procedural synthetic data generation, and next-gen Generative AI, Anyverse produces datasets with unmatched realism and sensor-level accuracy (including RGB, NIR, Radar, LiDAR, and Thermal).

Its unified validation framework helps partners ensure compliance with specific regulations like Euro NCAP, as well as broader safety standards for autonomous systems. By bridging the gap between simulation and reality, Anyverse accelerates development, reduces costs, and ensures trustworthy perception for vehicles, robots, and intelligent machines.