When a factory robot trained on one part family encounters a new SKU, it typically does one of two things: it attempts a grasp using a position from its programmed library and succeeds by luck, or it fails and someone calls an integrator. The second case is far more common when the new part differs structurally from what the robot has seen. This is not a calibration problem. It is a generalization failure, and fixing it at the policy level is the core of what we have been working on.
This post describes the approach we took to build grasp policies that transfer across object shape families. It covers the shape family hypothesis, how we tokenize point clouds to expose shape features useful for grasp planning, the contrastive training objective that drives cross-family transfer, and the failure modes we have not yet solved.
Shape Families as a Useful Inductive Bias
Industrial parts do not vary arbitrarily. They cluster into shape families defined by dominant surface geometry: cylindrical shafts and bushings, prismatic brackets and housings, planar connectors and plates, compound bodies with flanges or steps. This structure is not accidental. Parts are constrained by manufacturing processes and functional requirements, which means the geometry space is sparse and highly organized compared to the general object space that household robotics benchmarks explore.
We tested this hypothesis by clustering our CAD mesh library using principal curvature features computed on the full mesh surface. Of roughly 1,800 parts drawn from automotive and electronics assembly contexts, about 85% clustered into eight stable geometry families with clear centroid separation. The remaining 15% were compound or atypical parts that sat near cluster boundaries. This distribution told us that a family-structured representation had real traction: most of the variation we care about is within-family, not cross-family.
The practical implication: if a model learns grasp-relevant geometry features that are consistent within families, it should generalize to novel within-family members reasonably well. Cross-family transfer is harder and requires a different kind of abstraction, which we address through the contrastive objective described below.
Patch Tokenization for Grasp-Relevant Geometry
We represent incoming depth sensor data as a point cloud and tokenize it into local patches using Farthest Point Sampling (FPS) followed by ball-query neighborhood grouping. Each patch captures a local surface neighborhood around a sampled center point. The patch encoder computes per-point features (local surface normals, curvature estimates, relative position within the patch) and aggregates them into a fixed-length patch token via a small PointNet-style MLP.
Patch size is a meaningful design parameter. Setting the ball-query radius too large averages away the fingertip-scale geometry detail that determines contact stability, particularly for parts with small chamfers, steps, or recesses where antipodal contact points must be placed carefully. Setting it too small produces noise-dominated patches from the sparse depth returns common with structured-light sensors scanning metal surfaces at oblique angles. Through ablation on our validation set, we settled on a radius that covers roughly 8-12 mm of surface at typical sensor distances of 400-700 mm, which captures the geometry scale relevant to gripper contact while tolerating sensor noise at the patch boundaries.
One thing we are careful not to overstate: patch tokenization solves the representation problem for within-family generalization. It does not solve the global pose estimation problem for parts with rotational symmetry. A cylindrical shaft looks the same from any azimuthal angle, which means the model must reason about grasp axis orientation without a complete pose estimate. We handle this with a pose-conditioned grasp head that outputs a distribution over grasp candidates, not a single deterministic pose.
Contrastive Training Across Shape Families
The contrastive objective pulls together patch-level representations from parts belonging to the same shape family while pushing apart representations from different families. Crucially, the contrastive signal operates at the patch level, not the object level. This distinction matters: if we pulled together whole-object representations within a family, the model would learn object identity features (this looks like bearing X) rather than the local geometry features that transfer to novel instances (this patch has the surface curvature profile of a cylindrical contact surface).
During training, we sample triplets: an anchor patch from a source part, a positive patch from a different part in the same shape family with similar local geometry, and a negative patch from a different shape family. The encoder is trained jointly with the grasp prediction head so the contrastive supervision shapes the representation space while the task supervision keeps the learned features grounded in actual grasp utility.
The result is a latent space where parts within a family cluster tightly around geometry that predicts stable grasp outcomes. Novel parts in the same family land close to that cluster, and the grasp head generalizes across the cluster. Parts at family boundaries get a softer, blended representation, which the model handles with higher uncertainty in the output distribution.
A Concrete Test Case
We ran a targeted evaluation on a set of turned aluminum components: shaft inserts, bushing sleeves, stepped shafts, and one thin-wall sleeve variant. None of these meshes appeared in the training data. The cylindrical family centroid was well-covered in training by similar but distinct parts.
The model found stable antipodal grasp candidates for four of the five variants on the first attempt. The failure was the thin-wall sleeve. The inner diameter of the sleeve was narrow enough that the point cloud from our structured-light sensor returned almost no points on the inner surface, leaving the local geometry around the rim ambiguous. The model assigned high uncertainty to that region and proposed grasps on the outer surface that had inadequate force closure for the part's mass distribution at speed.
This failure was informative rather than surprising: it was predictable from the patch density statistics before attempting the grasp. A sparse patch with high curvature variance and low point count signals unreliable geometry, and we now propagate that uncertainty into a grasp confidence score that the system uses to flag parts requiring fallback handling. We are not claiming the model handles all parts; we are claiming it provides a principled signal about where it is and is not confident.
Where We Are Pushing Next
Compound-geometry parts remain the hardest case. A cylindrical shaft with a prismatic mounting flange does not belong cleanly to either the cylindrical or prismatic family. We currently handle this with soft family membership, assigning a weighted combination of family embeddings based on the proportion of the point cloud that fits each family's patch statistics. The grasp head then operates on the blended representation. This works adequately for parts with well-separated compound regions, but it breaks down for parts where the compound geometry is interleaved, such as a spline gear where the tooth profile creates a periodic alternation between cylindrical and planar surface patches.
The deeper open question is whether shape families are the right level of abstraction at all, or whether we should be working directly in the space of grasp-relevant geometric primitives (contact edges, flat contact patches, cylindrical contact surfaces) decoupled from part-level taxonomy. We have early experiments in that direction that look promising, and we expect to publish results later this year.
What we have now is not a general solution to novel-object grasping across the full range of industrial parts. It is a solution that works reliably within well-represented shape families and provides calibrated uncertainty at family boundaries. For most flexible pick cells handling parts within a known geometry range, that is sufficient to eliminate the re-teaching cycle for new SKUs. The edge cases are real and worth being honest about.