Table of Contents
Table of Contents
When 98% in simulation doesn't survive the factory floor
A bin-picking policy in simulation clears 98% of grasp attempts. Deployed on the real line, it drops below 70% in its first week. Nothing about the geometry changed. The CAD models were the same. The controller was the same. What changed was the light hitting a slightly specular part, and a depth camera returning something the policy had never seen.
The diagnosis matters more than the number. The policy didn’t fail at manipulation — the grasp planning, the contact model, the controller all performed as designed. It failed at seeing. The dynamics were fine. The photons were wrong.
This distinction gets lost because “physics-based simulation” is used in robotics to mean one thing: rigid-body dynamics — contact, friction, inertia, articulation, the physics that determines whether a planned motion executes correctly. That layer is mature and well-served by modern engines.
There is a second physics that gets far less attention: light transport, material response, and the electro-optical behavior of the sensor, which determine what the robot’s perception system actually receives. This layer is routinely approximated rather than modeled — and failures logged as “manipulation problems” often originate here, in what the camera or LiDAR returned, not what the arm did.
The sim-to-real gap is a fidelity problem before it is a volume problem. Rendering more scenes through the same approximate sensor model doesn’t close it — it produces more training data with the same systematic error baked in.
And the fidelity problem is about to stop being a purely engineering concern. From 20 January 2027, European robotics vendors will have to produce evidence that they closed it, not assurances, and we’ll come back to what counts as evidence.
How robotics teams use synthetic data today
Meeting that standard starts with knowing what’s already working. Synthetic data already carries real weight in robotics pipelines. Four use cases come up repeatedly:
- Manipulation and grasp-pose estimation — training perception and policy models to identify graspable points and orientation before contact
- Warehouse and logistics navigation — mapping, obstacle detection, and path planning across mobile platforms
- Legged and humanoid locomotion — terrain interaction and balance policies trained at a scale no lab could stage physically
- Human-robot collaboration — perception in shared workspaces where a cobot and a person occupy the same square meter
Most teams doing this work already have a simulation environment they trust. Isaac Sim, Gazebo, and MuJoCo are genuinely excellent at what they were built for – robot dynamics, contact physics, and control-policy learning at scale, with rendering good enough to keep a policy grounded in something resembling the real world.
That’s not a criticism to walk back later. It’s a scope boundary, confirmed rather than asserted: a recent survey, A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI (arXiv:2505.01458), contrasts PyBullet’s rasterization-based rendering against Isaac Sim’s ray tracing, and notes that MuJoCo’s OpenGL rasterization “lacks hardware-accelerated real-time ray tracing, restricting visual fidelity and widening the perception sim-to-real gap.” Isaac Sim is the exception — its RTX ray-traced rendering is why the same survey treats it differently. But it’s worth being precise about what that buys, because ray tracing is not the same thing as physical accuracy. Ray tracing solves geometric light transport, usually in RGB tristimulus. It is not spectral rendering, and it models neither sensor response, nor the ISP chain, nor LiDAR return physics. Isaac Sim’s rendering is the best of this group and still sits on the visually-plausible side of the line drawn in the next section.
That line is the one that matters here, because perception models ask more of a rendering layer than a control policy ever did: they need to match what a real sensor outputs, not what a scene plausibly looks like to a human eye.
That gap between plausible and accurate is measurable, and the evidence is uncomfortable for anyone reaching for more scenes as the fix. A 2024 benchmark, SplatSim, replaced a simulator’s mesh rendering with 3D Gaussian Splatting – a technique that reconstructs photorealistic scenes from real captured images, with no light-transport modeling involved, and closed most of the distance to policies trained on real-world data — by changing the rendering layer and nothing else. The policies were trained by behavior cloning, a setting where rendering fidelity is supposed to matter less — which makes the effect more striking, not less.
SplatSim — arXiv:2409.10161
A careful reader will spot the trap in that sentence: if a technique with no physical light modeling closed most of the gap, doesn’t that undercut the case for physics-based rendering specifically? The honest answer is no, and the reason matters for the rest of this article. SplatSim reconstructs a scene that was already captured. It’s a powerful way to reproduce what you have. It is not a way to generate what you don’t have and can’t collect:
- It cannot re-light a scene under conditions that weren’t captured
- It cannot swap a material and predict how a different surface would behave under the same sensor
- It cannot render through a sensor configuration that was never physically present during capture
- It cannot produce parametric, reproducible ground truth — the scene is whatever was recorded, not a controlled variable
That last point is the one worth remembering. Generating what you don’t have and can’t collect is exactly what the certification argument later in this piece depends on. SplatSim and physics-based rendering are answering the same underlying question — what does the sensor actually receive? — from opposite directions.
Two meanings of "physics-based" in robotics simulation
Robotics simulation already does “physics-based” well — just not the physics that determines whether a robot can see. Two distinct physical models sit inside every simulation pipeline, and conflating them is where a lot of confusion starts:
Physics of motion | Physics of sensing | |
Covers | Rigid-body dynamics, contact models, friction, articulation, inertia | Spectral light transport, material BSDFs, participating media, lens and aperture behavior, sensor response, noise |
Determines | Whether the robot’s action transfers | Whether the robot’s perception transfers |
Current state | Solved to a useful degree by mature engines | Routinely approximated |
This is also where generative AI belongs in the picture, and where it doesn’t. Generative models are genuinely useful for multiplying variety — layouts, human appearance, clutter, texture diversity — layered on top of a physically accurate base. What they can’t do is replace that base, because a generative model doesn’t model the sensor.
Decomposing the sim-to-real gap: perception and action dynamics
The same survey groups sim-to-real discrepancies into two categories, and the split is worth building on directly:
Sim-to-real gap. The performance difference between a policy evaluated in simulation and the same policy deployed on physical hardware — the gap between what a model learned to do and what it can actually do outside the simulator.
Perception gap. The divergence between the sensor data a model was trained on and the sensor data it receives in deployment. It originates in rendering approximation, missing sensor-response modeling, unmodeled optical phenomena, and material inaccuracy — every point where a simulated scene diverges from what a real sensor would actually capture.
Action-dynamics gap. The divergence between simulated and real physical interaction — contact, friction, actuation, and the countless small mechanical realities a dynamics engine approximates rather than measures.
A concrete case for the second gap: a grasping policy trained against a fixed friction coefficient learns a grip that only holds at that exact value. The real gripper’s pads wear over time, and a policy that was never shown a friction distribution wider than the simulator’s default fails on contact with a part whose surface friction differs from what it trained on.
Domain randomization is often reached here, and it’s worth being precise about what it buys: robustness to noise, not robustness to reality, if the base distribution it randomizes over is wrong to begin with. Widen that distribution far enough to cover the uncertainty, and the model learns an invariance that discards the signal it needed. The real tradeoff is coverage against learnability — domain randomization mitigates not knowing the real distribution. Model that distribution correctly, and you need markedly less of it.
The action-dynamics gap has had roughly a decade of dedicated robotics engineering behind it — contact solvers, system identification, most of the sim-to-real literature. The perception gap, by comparison, is usually treated as a rendering setting to tune rather than a variable to engineer. It’s the more tractable of the two, and currently the more under-served.
Sensor simulation: modeling what the sensor receives, not what the scene looks like
A camera does not photograph a scene. It integrates incoming radiance over a spectral response curve, an aperture, and an exposure interval, then runs the result through an image signal processor before a single pixel reaches a training set. Simulating the scene and simulating the sensor are different tasks — and most robotics pipelines only do the first one.
RGB, arm-mounted: manipulation
What actually shapes the image a manipulation model trains on:
- Spectral sensitivity and quantum efficiency
- Exposure and dynamic range
- Rolling shutter behavior under motion
- Lens distortion and vignetting
- The full ISP chain — demosaicing, white balance, tone mapping, denoise, compression
A perception model trained on ISP-free renders has never seen the artifacts it will be fed in production. The Reality Gap in Robotics: Challenges, Solutions, and Best Practices (arXiv:2510.20808) documents the mechanism: simulators using simplified OpenGL rendering with idealized pinhole camera models omit lens flares, chromatic aberration, flying pixels and rolling shutter distortion — and the features a model learns on that data transfer poorly to real deployment. The ISP chain sits on top of that list, not inside it, which is the point: every one of these is a property of the sensor, not of the scene.
LiDAR, mobile base: navigation
Navigation models depend on:
- Beam divergence
- Multi-return behavior
- Intensity as a function of material and incidence angle
- Retroreflective surfaces — safety vests, floor markers, trailer tape — producing blooming and dropouts
- Behavior in dust or steam
Structured light and ToF: bin picking
Two failure modes decide whether a bin-picking system works outside a lab: multipath interference in concave metal bins, where the sensor receives light reflected off multiple surfaces instead of one, and ambient IR contention near sunlit loading doors, where sunlight swamps the sensor’s own emitted signal. The most consequential version of both: a shiny part returns nothing, and the depth image has a hole exactly where the grasp point is.
Ground truth as a first-class output
A physically modeled pipeline doesn’t just render an image — it produces pixel-accurate depth, semantic and instance segmentation, surface normals, material IDs, 6-DoF pose, and per-object visibility, consistently and by construction. Real-world data doesn’t come with any of this attached; it’s hand-labeled, expensive, and imprecise wherever a human annotator has to guess. For training, that ground truth accelerates iteration. For validation work — where the next section picks up — the ground truth isn’t a convenience. It’s the evidence.
The point of all this isn’t making images look more convincing. The determinant of real-world transfer is not how convincing an image looks to a human — it’s whether the model’s input distribution matches what the deployed sensor actually produces.
Perceiving people: where approximated sensor models fail first
For a warehouse AMR or a hospital robot, people are the nominal operating condition of the workspace — not an edge case handled separately from everything else.
That’s worth stating plainly, because it changes what “coverage” has to mean. What makes human perception genuinely hard at the sensor level:
- Heavy occlusion — a person is rarely fully visible; carts, shelving, and other people constantly interrupt the sensor’s view
- Unpredictable, non-rigid motion — unlike a rigid object, a human body doesn’t move in ways a dynamics model can fully anticipate
- Enormous appearance variance — clothing, PPE, skin tone, mobility aids, carried objects
- Proximity — people appear precisely at the ranges where depth sensors are least reliable, not at the comfortable mid-range distances most benchmarks are built around
Perception failure here is a safety failure, not a task failure. A missed detection at three meters is a line in a dataset evaluation. A missed detection at half a meter is an incident report.
This is exactly where domain randomization without physical grounding is least defensible. Randomizing appearance can’t manufacture a correct sensor response on a high-visibility vest — the retroreflective return from that fabric is a physical phenomenon, not a nuisance parameter a wider distribution can average away. Get it wrong and the vest either disappears from the point cloud or blooms into a false obstacle, at the exact proximity where either outcome is an incident rather than a missed detection.
Generative AI has a real role here, and it’s a specific one: generate human appearance and pose diversity, then render that diversity through a physically accurate sensor model — not the other way around. Diversity upstream, physical accuracy downstream. Reverse that order and the result is data that looks varied and is uniformly wrong at the sensor level — clothing textures and skin tones that were never actually run through a spectral response curve or a LiDAR return model.
This same problem class — human presence, pose, and occupancy detection under severe occlusion and extreme appearance variance — is one we’ve already worked through at production scale in a different safety-critical domain: automotive in-cabin monitoring, validated against real sensor hardware for years, under a regulatory regime that already demanded exactly this kind of evidence.
Training data is becoming regulatory evidence
For a decade, a robotics company’s obligation was to build a system that works. The obligation arriving now is to demonstrate that it works — to a third party, with documented evidence, before it ships.
The scope question comes first. The Machinery Regulation’s Annex I Part A sends two categories through mandatory third-party conformity assessment, effective 20 January 2027: “safety components with fully or partially self-evolving behaviour using machine learning approaches ensuring safety functions,” and the machinery that embeds them. The operative word is self-evolving. A perception model that’s trained, frozen, and shipped is a different case, on its face, from one that keeps adapting after deployment — and exactly where that line falls isn’t settled. The harmonized standards meant to clarify it are still being finalized.
Layered on top of that is a second timeline, and it moves in the opposite direction. Regulation (EU) 2026/1744 — the Digital Omnibus on AI, in force since 27 July 2026 — pushed the AI Act’s high-risk obligations for AI embedded in Annex I regulated products from 2 August 2027 to 2 August 2028. That deferral still governs most of those products: medical devices, toys and the rest remain fully in scope on the later date.
Machinery is the exception. The same regulation carves Machinery Regulation products out of the AI Act’s direct high-risk applicability altogether, and gives the Commission power to add AI-specific health and safety requirements through delegated acts under the Machinery Regulation instead. For robotics, that closes one route and opens another — and the one it opens is still empty. As of this writing, those delegated acts haven’t been adopted.
One settled requirement, dated: 20 January 2027. One route closed: direct AI Act applicability to machinery. One route open but unwritten: the delegated acts that will carry the AI-specific requirements, with no date attached. A team that gets the first one wrong has no time left to build an evidence base from nothing.
The relevant part of the European picture:
- Machinery Regulation — third-party conformity assessment for safety-critical machinery with self-evolving, ML-based behavior; since the 2026 Digital Omnibus, also the route through which AI-specific requirements apply to machinery, rather than the AI Act directly
- ISO 10218-1:2025 / ISO 10218-2:2025 — industrial and collaborative robot safety; the February 2025 revision folded the collaborative-application requirements formerly in ISO/TS 15066 directly into the series
- ISO 3691-4 — driverless industrial trucks
- ISO 13849 — safety-related parts of control systems
The 2025 ISO 10218 revision made a parallel move worth noting: functional safety requirements that used to be implied are now explicit, tied to formal robot classifications — the same shift from implicit to checkable that this article is arguing synthetic data evidence needs to make. The demand common to all of these instruments is the same: evidence that the system was assessed against a defined scenario set, including hazardous ones — a question of performance limitations in the absence of any fault, in much the same spirit as automotive’s ISO 21448 (SOTIF), applied to a different class of machine.
The collection problem is where this gets difficult in practice. The scenarios that matter most for an assessment are precisely the ones that can’t be staged: a person entering the path of a 500 kg AMR at closing speed, a child-sized occlusion emerging from behind a pallet, a perception failure under a specific combination of low sun angle and wet floor. None of this can be collected at scale. Most of it can’t be collected safely at all. Real-world collection isn’t a slow path to this kind of evidence. It isn’t a path.
So the evidence has to be synthetic. But not any synthetic data qualifies. Three properties determine whether it’s admissible:
- Physical correctness. If the simulated sensor output doesn’t correspond to what the real sensor produces, the test proves nothing about the real system. Passing an assessment against inaccurate data is worse than failing outright — it certifies a false result.
- Determinism. Same scene definition, same seed, same sensor model, same output, every run, on any machine. A stochastic or unreproducible pipeline can’t support a regression campaign, can’t isolate the variable that caused a failure, and can’t be re-run for an assessor eighteen months later. It’s the requirement almost nobody in synthetic data talks about, and the one that decides whether a test campaign becomes a technical record or stays a one-off result.
- Traceability. Every generated frame carries its full parameterization — geometry, materials, illumination, sensor configuration, ground truth. The scenario set becomes a specification, not an artifact. That’s what turns a folder of images into documented evidence.
This reframes physics-based synthetic data from a training convenience into validation infrastructure. It’s also why the market is likely to separate along a line that has nothing to do with rendering quality: a renderer optimized for plausibility can help a team train. Only a pipeline that’s physically correct, deterministic, and traceable can help a team certify.
It’s also where generative AI’s limit becomes explicit rather than implied. A generative model that can’t explain why it produced a given output can’t produce auditable evidence. That’s not a criticism of generative methods — it’s a statement about where in the pipeline they belong, and validation isn’t it.
Where physics-based simulation is the wrong tool
None of the preceding argument holds if it’s read as “always use this.” Four limits worth being direct about:
- It’s more expensive per frame. For coarse layout learning or high-level navigation pre-training, appearance-level realism is often entirely sufficient, and considerably cheaper.
- It doesn’t replace a dynamics engine. Contact, articulation, and control still need one. The two layers are complementary — the same point made earlier about Isaac Sim, Gazebo, and MuJoCo: rendering fidelity is an addition to that stack, not a rival to it.
- It doesn’t remove the need for real data. It changes the ratio — targeting real-world collection at validation and calibration rather than bulk coverage, and making the real data that does get collected more useful.
- Physical accuracy is only as good as its inputs. Wrong material measurements and wrong sensor characterization produce confidently wrong data. The measurement and characterization work behind a physically accurate pipeline is real work, not a switch that gets flipped on.
Requirements for a robotics synthetic data pipeline
Taken together, the argument so far points to a specific set of engineering requirements — not a buyer’s checklist, but a consequence of taking the perception gap as seriously as the action-dynamics gap already gets taken:
- Sensor models characterized against real hardware, not parameterized by eye — and the ability to show that characterization, not just assert it.
- Coverage across both task families and across embodiments. Manipulation and navigation have different sensor suites, and most teams eventually need both — with sensor mounting, motion and vibration modeled per platform, not assumed constant across arms, mobile bases, legged robots and humanoids.
- Interoperability with the existing robotics stack. Complementary, not rip-and-replace. A pipeline that requires abandoning the dynamics environment a team already trusts will not get adopted, regardless of its rendering quality.
- Parametric control over physically grounded variability — the ability to vary one physical property and hold the rest fixed. That’s what separates a controlled experiment from a batch of images, and it’s why generative capability belongs upstream of the sensor model: one component of a variety strategy, not a substitute for the physical layer underneath it.
- Determinism and full traceability as pipeline properties, not optional features. The previous section covers why.
- Validation treated as a first-class capability, distinct from training. Most pipelines were designed for one and retrofitted for the other, and it shows.
That combination is narrower than it sounds — narrow enough that it says something larger than a pipeline checklist can capture.
Two questions every robotics team will have to answer
Physical AI is moving out of demonstrations and into places where people work and live. That transition is a perception problem far more than it is a control problem — and for a decade, the field’s investment has run the other way.
Of the two gaps this piece has walked through, rendering and sensor fidelity is the more tractable lever, and it remains the more neglected one. The tools to close it already exist. Treating it as a priority is the part the field hasn’t caught up on yet.
Which leaves two questions every robotics team building toward deployment will eventually have to answer:
Does your simulated sensor data match what your real sensor produces?
Can you prove it, reproducibly, to someone who wasn’t there?
The market will separate along the second question, not the first. Approximating reality is enough to build a demo. Modeling it is what it takes to ship a certified system into a European warehouse.
Physics-based synthetic data is infrastructure — for training, and for the evidence record. Generative AI is a force multiplier on top of it.
Building toward that standard? Talk to us about what a physically accurate, deterministic pipeline looks like for your sensor stack.
