Applying transformer architectures to 3D point clouds is not new. PointNet++ introduced hierarchical local feature aggregation in 2017, and Point Transformer variants have since pushed state-of-the-art on standard 3D benchmarks like ModelNet40 and ScanObjectNN. What is less documented is what it actually takes to get these architectures to work reliably on depth sensor input from industrial parts in a factory environment, where lighting is inconsistent, metal surfaces cause specular returns and sensor noise, and the parts you encounter at deployment time were not in your training set.
This post is mostly about what we changed and why, written for readers who have already read the standard point cloud transformer literature and want to understand the industrial-specific adaptations.
Starting Point: Standard Point Transformer Architecture
Our baseline was a Point Transformer architecture with vector self-attention: for each point, attention weights are computed from position-encoded relative coordinates and feature differences, and the attended features are aggregated into a new per-point representation. Transition down and up layers with FPS and k-nearest neighbor grouping provide the hierarchical structure needed to capture multi-scale geometry.
On our own training split of CAD-derived synthetic depth renders this worked well, reaching grasp success rates consistent with what the published literature reports on similar part families. When we moved to real sensor data from an Azure Kinect mounted above a parts bin on a factory floor, performance dropped substantially. The architecture itself was not the issue. The input distribution was.
Sensor Noise Is Not Additive Gaussian on Metal
Most published work treats depth sensor noise as approximately Gaussian with some standard deviation. On non-metal surfaces in controlled lighting that is a reasonable model. On machined aluminum and steel in factory lighting, it is not. We see three distinct noise types that require different handling.
First, specular noise: polished metal surfaces reflect structured-light patterns in ways that cause misregistration between the projected and captured patterns. The result is not random noise around the true surface but systematic errors that are correlated along the sensor's illumination direction. These show up as stripes or bands of incorrect depth, often concentrated on curved surfaces where the reflection angle changes rapidly. Standard Gaussian augmentation during training does not help the model learn to handle these patterns.
Second, missing data regions: at oblique angles to highly reflective surfaces, the sensor returns no data at all. In a point cloud, this appears as holes in the surface reconstruction. For a part sitting at an arbitrary pose in a bin, some portion of the part's surface will always be in a missing-data region relative to any single camera viewpoint. A model trained on dense synthetic point clouds will have never seen this pattern.
Third, edge fringing: at depth discontinuities, structured-light sensors produce a characteristic fringe of incorrect depth estimates that smears the boundary between the part surface and the bin floor. This creates phantom geometry near part edges that can corrupt contact-point estimation if the model treats it as real surface data.
We address all three with targeted augmentation during training and a pre-processing step at inference time. For specular noise, we add correlated stripe noise in the sensor's illumination-direction frame during training, not isotropic Gaussian. For missing data, we drop random surface patches during training using a visibility simulation that approximates the geometry of oblique-angle sensor gaps. For edge fringing, we apply a normal-consistency filter at inference that flags boundary points with high normal vector variance as unreliable and removes them from the point set before encoding.
Positional Encoding for Grasping, Not Classification
Standard point transformer positional encodings represent the absolute or relative 3D coordinates of points within a local neighborhood. This works well for classification tasks where the goal is a global shape descriptor. For grasp pose prediction, we care about something different: the relationship between candidate contact points and the object's principal axes, which requires understanding geometry at the scale of gripper finger separation (typically 30-80 mm for industrial grippers on small parts).
We modified the positional encoding to include the local surface normal direction and an estimate of the local principal curvature, computed from the neighborhood point set. The attention mechanism can then directly attend over surface orientation features, not just spatial positions. This change had a measurable effect on grasp pose estimation accuracy, particularly for parts with flat contact surfaces where the key constraint is aligning the gripper approach direction with the surface normal.
We are not claiming this is the only way to solve this problem. An alternative is to compute grasp-relevant geometric descriptors (point pair features, FPFH, similar) and concatenate them with the learned features. We tried this and found the soft attention over normal-augmented positional encodings generalized better to novel part geometries, while the hand-crafted descriptor approach worked slightly better on seen parts and worse on unseen ones.
Attention Radius and the Token Count Tradeoff
Industrial manipulation on small-to-medium parts (roughly 20-300 mm characteristic size) operates in a regime where the relevant geometry is neither as small as tabletop household objects nor as large as scene-level point clouds. The practical consequence is that standard token counts used in published models are often either too coarse or too fine.
At 512 tokens with typical ball-query radii from classification benchmarks, we were averaging over too much surface area at the patch level, losing the edge and chamfer detail that determines stable contact geometry. At 2048 tokens, inference latency was 180-220 ms per frame on our deployment hardware, which was too slow for reactive grasp replanning at the rates we wanted. We settled on 1024 tokens with a smaller ball-query radius, which brought inference to 95-110 ms on an NVIDIA Orin NX while preserving enough geometry resolution for our part size range. This is not a universal number, it is specific to our gripper geometry and target part size distribution.
Multi-View Fusion Without a Full 3D Reconstruction
Single-view depth estimation is fundamentally limited for parts that are self-occluding from any single viewpoint. For bin-picking scenarios where parts are touching or overlapping, a single overhead camera misses the geometry between touching surfaces. The standard solution is multi-view stereo reconstruction, but building a dense 3D mesh per grasp attempt adds latency and computational cost we wanted to avoid.
Our current approach uses two camera viewpoints (one overhead, one at a 35-degree offset angle) and fuses the resulting point clouds at the feature level rather than in 3D space. Each point cloud is encoded separately through the patch encoder, and the patch tokens from both views are concatenated and passed through a cross-attention layer that can resolve correspondences between overlapping regions. This is cheaper than a full multi-view reconstruction and handles the most common cases of part-to-part occlusion in a flat bin layout. It does not handle cases where a part is fully occluded from both viewpoints simultaneously, which we treat as a failure mode requiring bin agitation.
What We Have Not Solved
Semi-transparent parts, such as certain plastic components with partial light transmission, confuse structured-light sensors in ways that our noise augmentation does not cover. We have not encountered this much in our target manufacturing contexts, which are primarily metal and rigid plastic, but it is a known gap.
Very small parts below about 15 mm characteristic size push against the resolution limits of the sensors we work with. At that scale, the point cloud density is low enough that patch tokenization becomes unreliable, and we are closer to the estimation regime where contact point placement requires sub-millimeter precision. We are not currently solving this case.
The architecture choices described here are what we have learned works in the industrial setting we are targeting. They are not the final form of the model. We will publish updated results as we iterate on the architecture and as we gather more real-factory evaluation data.