The sim-to-real problem in manipulation is well-studied in the household robotics setting: you train in a simulated kitchen or living room and you want your policy to transfer to the real version of that scene. The techniques developed for that setting, domain randomization over textures, lighting colors, and object sizes, carry over to industrial settings in principle but miss most of what actually causes sim-to-real failure in a factory environment.
Factory floors are not noisy households. They are noisy in specific, structured ways that require specific adaptations. This post describes the sim-to-real gap as we have encountered it on real factory floors, not as it is typically framed in the academic literature, and the domain randomization strategy we use to address it.
What Is Actually Different in a Factory
Three factors drive most of the sim-to-real gap we encounter in industrial settings, and they are not the three factors most domain randomization papers focus on.
The first is metal surface optical properties. Machined metal surfaces are specular, anisotropic reflectors. A brushed aluminum bracket does not look the same from two different camera angles, and neither does it look the same under the same camera angle when the overhead fluorescent lights are at different intensities (which happens with energy management systems that modulate lighting levels). Standard texture randomization in simulation applies random 2D texture images to object surfaces, which bears no resemblance to the actual reflectance behavior of metal. Getting this right in simulation requires a BRDF model for metal surfaces, not texture randomization.
The second is part clutter density. Academic bin-picking benchmarks typically feature bins with a handful of parts arranged with enough space between them that the depth sensor can get a clean return on each part. Real parts bins in production settings are packed to a much higher density, with parts touching and overlapping such that the depth sensor sees continuous merged surfaces rather than separated individual objects. Policies trained on low-density simulated bins transfer poorly to high-density real bins because the instance segmentation problem changes qualitatively.
The third is floor and bin vibration. Nearby machinery couples vibration into the parts bin through the factory floor. For a structured-light sensor operating at 10-20 Hz, part vibration at 5-15 Hz creates motion blur in the depth map that does not exist in simulation. The parts are not quite where the depth map says they are at grasp time.
BRDF Randomization for Metal Surfaces
We model metal surfaces in simulation using a Cook-Torrance microfacet BRDF, which parameterizes metal reflectance by roughness (the distribution of microfacet surface normal angles), Fresnel reflectance at normal incidence (determined by the metal's refractive index and extinction coefficient), and a specular/diffuse balance term. For our target part materials (machined aluminum, turned steel, zinc-plated steel, anodized aluminum), we measured real BRDF parameters from physical samples using a gonioreflectometer and defined realistic parameter ranges for each material class.
During training, for each rendered frame we sample BRDF parameters from the per-material-class ranges and render with those parameters. The result is a training distribution that includes the full range of metal surface appearances from nearly diffuse (sandblasted or anodized surfaces) to highly specular (mirror-polished stainless). The depth sensor simulation then applies the reflectance model when computing simulated structured-light returns, producing simulated specular noise patterns that match what we see on real parts.
This is more expensive to render than simple texture randomization: ray-tracing the BRDF accurately requires path tracing rather than rasterization, and path-traced renders at sufficient sample count take 3-5x longer than rasterized renders. We use rasterization with an approximated BRDF for most training data generation and reserve full path tracing for a validation sample set used to check that the approximation does not introduce systematic bias.
Density Curriculum for Bin Clutter
Training immediately on high-density clutter does not work well: the model has not yet learned to identify individual part surfaces, so the high overlap and occlusion in a crowded bin provides a learning signal that is too ambiguous to make progress from. We use a density curriculum that starts training with low-density bins (2-4 parts per bin) and progressively increases the density over the course of training as the model's part-instance segmentation quality improves.
The density schedule we use is not derived from theory, it is empirical: we found that increasing density too quickly stalled training at a local minimum where the model learned to target the center-of-mass of the entire bin contents rather than individual part grasp points. The schedule that worked was a three-phase progression: single-part bins for the first 20% of training, 2-6 part bins for the middle 50%, and realistic production-density bins (8-15 parts) for the final 30%. The validation set always uses production-density bins.
Part interaction dynamics also change at high density. Parts support each other in ways that affect which grasps are physically accessible: a part buried under two others has most of its valid grasp candidates blocked. In simulation, we run the grasp candidate set through a kinematic interference check that removes candidates where the gripper approach path would collide with neighboring parts. This check is necessary at high density but adds significant computation to the training data generation pipeline.
Vibration Modeling and Temporal Integration
We model bin vibration in simulation as a superposition of sinusoidal displacement components with frequencies sampled from a range calibrated to real factory floor vibration measurements (5-20 Hz, amplitude 0.3-2 mm depending on the proximity to heavy machinery). During training data generation, we render multiple frames of the scene with slightly different part positions corresponding to different vibration phases, and we present these as a temporal sequence to the model rather than a single frame.
The model then uses temporal integration (attention over a 3-frame window at the sensor's acquisition rate) to estimate the time-averaged part position, which is more stable than the instantaneous position at any single frame. The temporal integration adds some latency to the pose estimate but substantially reduces the variance in the position estimate for vibrating parts.
We are not claiming this solves the vibration problem completely. High-amplitude vibration from very nearby heavy presses or stamping machines remains difficult. In those cases, we recommend gating the vision acquisition to a low-vibration phase of the press cycle, which the automation controller can synchronize via its existing process monitoring. We treat this as an integration detail rather than a model problem.
Residual Gap and What We Are Not Saying
After all of the above, there remains a measurable residual gap between simulation performance and real-factory performance. On our validation set of controlled real-factory scenes, the grasp success rate is consistently 8-12 percentage points lower than on simulation runs with matching part sets. We believe this residual gap has three main contributions: systematic errors in our BRDF model that do not capture all real metal surface conditions, depth sensor artifacts that our noise model does not reproduce accurately, and contact dynamics during grasp execution that simulation underestimates.
We are not claiming that our domain randomization strategy eliminates the sim-to-real gap. We are claiming that it reduces it to a level where real-factory performance is high enough for practical use in the scenarios we have tested, and that the remaining gap is systematic enough to be correctable with a small amount of real-world fine-tuning data if needed.
Sim-to-real transfer is not a solved problem in industrial manipulation. The specific techniques that work in household robotics (random texture mapping, color jitter, random lighting colors) are necessary but not sufficient. The domain knowledge that matters is the physical optics of the materials involved, the mechanical dynamics of the specific industrial setting, and the density and interaction regime of the actual parts being handled. Getting these right in simulation is harder than it looks from the academic literature, and it is worth being honest about how much work remains.