A standard pick cell perceives the world as a single depth image or point cloud covering the entire working volume: parts bin, fixture, surrounding structure, and sometimes the robot's own body. The grasp policy trained against this kind of input learns implicitly to route around background clutter and find the target objects, but it does so by encoding the scene as a whole. Changing the layout of the parts, adding new part types to the bin, or running the same policy on a cell with different background structure introduces distribution shift that the model was not explicitly trained to handle.
Object-centric representations decompose this global scene observation into per-object representations before the grasp policy ever sees them. Each part in the scene gets its own representation, extracted independently of what else is in the field of view. The policy then operates on a single object representation, not the whole scene. This decomposition has an important consequence: the policy can generalize to novel parts and novel scene configurations without retraining, because the per-object representation abstracts away scene-level context.
This post covers how we approach the scene decomposition problem for industrial parts bins, why getting it right in the presence of occlusion and part contact is harder than it looks, and where the current approach has limits that we have not resolved.
The Scene Decomposition Problem in Industrial Settings
Industrial parts bins are genuinely hostile to object segmentation. Parts are often made of the same material, sit in physical contact with each other, and have surface geometry that repeats across multiple instances. A bin of M5 hex bolts contains dozens of nearly identical objects, each in a different orientation, with surfaces that reflect structured light in ways that produce missing data regions. A bin containing mixed bearings and connectors has objects with surface normals distributed across every direction, which means depth sensor coverage is uneven and part-to-part boundaries are often visible only at grazing angles.
Common segmentation approaches from general-purpose robotics research, including instance segmentation from color images and Euclidean clustering on point clouds, both struggle in this setting for different reasons. Color-based instance segmentation requires color contrast between objects that metallic parts don't provide. Euclidean clustering breaks down when parts are in contact, because the point cloud returns from two touching parts form a single connected region with no gap between them.
We addressed this by formulating the decomposition as a local geometry grouping problem rather than a global segmentation problem. Instead of trying to identify complete object boundaries, we identify local surface patches that are consistent with belonging to a single rigid body, and then merge patches based on geometric compatibility. Two patches are compatible if their local surface normals and curvature estimates are consistent with coming from the same rigid surface, and if their relative position is within the dimensional envelope of the expected part class.
The dimensional envelope constraint is where part class information enters the pipeline. If we know we are in a bin of bearings with outer diameter in a certain range, we can set tight bounds on the spatial extent of any single object's patch cluster. Patches that would require merging into a cluster larger than the maximum expected part dimension are rejected and re-assigned. This is a form of geometric prior that makes the decomposition tractable for known part families, at the cost of requiring that prior to be specified. For a factory running a known set of part types, this cost is acceptable. For a fully open-world setting, it would need to be replaced with something more general.
Representing Each Object for the Grasp Policy
Once the scene is decomposed into per-object patch clusters, each cluster is processed into a fixed-format object representation suitable for the grasp policy. We encode the spatial distribution of the patch cluster (its bounding geometry and principal axes), a histogram of local surface curvature types across the cluster, and a patch-level token sequence derived from the same tokenization described in our earlier post on patch transformers.
The patch-level tokens preserve the local geometry detail that the grasp policy needs for contact planning. The bounding geometry provides the global pose context. Together they give the policy both the "what does this surface feel like to contact" information and the "where is this object relative to the gripper" information, which are the two things the policy needs to plan a stable approach.
The representation is deliberately fixed-length. Variable-length representations require padding or masking strategies when processed by the policy network, and those strategies introduce subtle artifacts in the attention patterns when the distribution of sequence lengths at inference time differs from training. Using a fixed-length representation with a canonical ordering of patch tokens, based on spatial position within the cluster, gives the policy a consistent input format across objects of very different size and shape.
Zero-Shot Transfer: What It Actually Means Here
When we say the system handles zero-shot pick and place for novel parts, we mean specifically: given a part whose mesh was not present in the training data, the system can compute an object-centric representation of that part from depth sensor data and use it with the existing grasp policy to find valid grasp candidates, without any re-training or re-programming step.
What this does not mean: the system handles parts from shape families not represented in training with equal fidelity to parts from well-represented families. A part from a shape family that has few or no examples in training will produce an object representation that falls outside the distribution the policy was trained on, and the policy's output on that representation will be less reliable. The system provides a confidence estimate on its grasp candidates, and candidates generated for out-of-distribution object representations should carry lower confidence.
We ran an internal evaluation in which we introduced three part families that were not present in our training data: a set of injection-molded plastic clips (a shape family with few direct analogs in our training data, which is dominated by metal parts), a set of small PCB standoffs, and a set of rubber o-ring seals. The metal standoffs achieved grasp success rates comparable to in-distribution parts in our standard evaluation setup. The plastic clips and o-rings were harder: the plastic clips have a shape geometry profile unlike our metal part training distribution, and the o-rings have a surface normal distribution that is nearly degenerate (a thin torus). Both categories generated low-confidence outputs that correctly flagged the need for fallback handling.
The Occlusion Problem
Partial occlusion is the most significant open problem in our current object-centric decomposition. When one part partially occludes another in the depth sensor view, the visible portion of the lower part may be too small for reliable patch clustering. The decomposition either fails to identify the partially visible object as a distinct entity, or produces an incomplete object representation that underestimates the part's actual extent and yields incorrect grasp candidates.
The standard approach in the literature is to complete the occluded object's shape using a learned shape completion model. We have explored this and have a preliminary shape completion module in our pipeline. It improves decomposition quality for partially visible parts when the visible fraction is above roughly 30-40% of the total surface area. Below that threshold, shape completion produces high-variance outputs that are not reliable enough to trust for grasp planning.
Practically speaking, the cell layout and bin fill level largely determine how much occlusion is present. A bin that is too full produces heavy inter-part occlusion; a bin at moderate fill with deliberate spatial organization reduces it. Many factories already manage bin fill as part of their cell operation. For the remainder, the system needs to signal when occlusion is causing unreliable decomposition, so a human or an automated destacking mechanism can intervene. We surface this signal through the per-object confidence scores rather than attempting to hide it.
Where This Fits in the Broader System
Object-centric decomposition is the perception front-end for the grasp policy, not the policy itself. Better decomposition directly improves grasp policy performance for novel parts by giving the policy more accurate and consistent input representations. But decomposition quality is not the only factor: the policy's training distribution, its handling of uncertainty, and the downstream motion planning and execution all matter.
The piece we find most practically significant is the consistency it provides across scene configurations. A cell running the same policy in two factories with different bin layouts, different surrounding structure, and different overhead lighting should produce consistent grasp behavior because the policy only ever sees the per-object representation, not the scene. Empirically this is mostly true with the caveat that calibration differences between sensor setups affect the quality of the depth data, which affects the patch extraction before the policy ever sees anything. Getting scene-level variability down requires clean sensor calibration as a prerequisite, which is obvious but worth stating clearly.