Before we could publish results for our model, we needed an evaluation protocol we could stand behind. The benchmarks we found in the manipulation literature fell into two groups: household object datasets (YCB, EGAD, GraspNet-1B) oriented toward domestic robotics, and laboratory pick-and-place setups designed for tabletop manipulation at academic robot speeds. Neither reflects the distribution of parts or the performance requirements of industrial pick cells.
So we built our own. RMB-2K is 2,000 industrial part meshes across 20 part categories, each with associated grasp annotations, evaluation protocols for both grasp success prediction and actual physical grasp execution in simulation, and a standard evaluation harness that can be run against any model that takes a point cloud and outputs grasp poses. This post is about the design choices that went into it and what we got wrong the first time.
Why Existing Benchmarks Do Not Transfer to Industrial Settings
The YCB object set is the workhorse of tabletop manipulation evaluation. It is well-studied, physically available for purchase as real test objects, and has been used in hundreds of papers. It is also composed of household objects: soup cans, mugs, mustard bottles, toy blocks. The geometry taxonomy of YCB is oriented toward the visual appearance and graspability of daily-life objects, and the grasp success criteria are oriented toward human hand-scale grasps at low payload requirements.
Industrial parts differ from household objects in ways that matter for grasp evaluation. First, many industrial parts are symmetric: cylindrical bearings, hex bolts, flanged connectors. Symmetry creates grasp redundancy (there are many equivalent grasp poses for a perfectly cylindrical shaft) which requires a different success criterion than the one used for evaluating grasps on asymmetric objects. Second, many industrial parts have tight grip geometries: parts designed to be assembled by specific fixtures have surfaces that a parallel-jaw gripper can only contact in specific configurations. Third, industrial parts are typically metal or rigid plastic with high surface stiffness, which means the grasp contact mechanics are very different from the soft-deformable contact on silicone grippers grasping soft household objects.
GraspNet-1B is larger and more recent, and does include some industrial-looking objects. But it remains scene-oriented (cluttered tabletop scenes) rather than bin-oriented (parts in a bin at industrial density), and its grasp quality metric is based on grasp wrench space analysis with no regard for part-specific assembly constraints.
Part Category Selection
We defined 20 part categories based on our own analysis of what types of parts appear most frequently in flexible pick cells across our target industries: automotive sub-assembly, electronics connector assembly, precision machined components, and fastener handling. The categories are:
Cylindrical bodies (shafts, pins, rods, spacers), flanged bodies (flanged bushings, flanged connectors, flange plates), prismatic bodies (brackets, blocks, housings, covers), flat plate forms (shim washers, connector plates, cover plates), threaded fasteners (bolts, screws, studs), hex socket and Torx fasteners, bearings (ball bearings, roller bearings, needle bearings), ring and clip forms (snap rings, retaining rings, circlips), connector bodies (rectangular connector housings, terminal blocks), L-bracket and angle forms, T-shaped bodies (T-brackets, cross-connectors), compound stepped forms (stepped shafts, shoulder bolts), slotted and keyed forms (keyed shafts, slotted collars), gear forms (spur gears, worm gears), thin-wall cylinders (bushings, thin tubes), irregular prismatic forms (non-standard brackets), cast body forms (cast housing halves, irregular cast parts), sheet metal forms (stamped brackets, formed clips), knurled surface forms (knurled knobs, grip nuts), and hollow and tube forms (hollow cylinders, tube sections).
Each category has between 80 and 120 meshes in the full benchmark set. The range is intentional: some categories (like cylindrical bodies) have very high within-category variation and need more samples to probe generalization; others (like knurled surfaces) have lower variation and 80 samples is sufficient to evaluate the relevant grasp challenge.
Mesh Sources and Quality Control
CAD-accurate part meshes are not easy to obtain at scale for industrial parts. Most parts in production are designed in proprietary CAD systems (CATIA, SolidWorks, NX) and the mesh exports are licensed to the manufacturers, not publicly available. We built our mesh library from three sources: open-source CAD repositories (GrabCAD and similar sites where designers share individual part files), procedurally generated parametric parts using SolidPython and similar tools, and a small number of meshes we created ourselves for part categories not well-represented in the open sources.
All meshes went through a quality control pipeline: watertightness check (no open boundaries), manifold topology verification, resolution normalization (re-meshed to consistent polygon density of approximately 10K-50K triangles depending on part size), and a visual review step where we rejected meshes with obvious geometric artifacts or unrealistic manufacturing features. About 30% of the original candidate meshes were rejected at quality control, mostly procedurally generated meshes with non-physical features like zero-thickness walls.
Grasp Annotation and Evaluation Protocol
For each mesh, we generate a grasp annotation set using a force-closure analysis: we sample candidate grasp poses from a 6-DOF pose space discretized at 5-degree rotation intervals and 2mm position intervals, compute the quality metrics (grasp wrench space volume, Ferrari-Canny metric, antipodal quality) for each candidate with our target gripper geometry (parallel-jaw gripper with 80mm max opening, 5mm finger tip width), and label each candidate as valid (FCC metric above threshold) or invalid. The per-mesh annotation set typically contains 400-2000 valid grasp candidates depending on part geometry.
The evaluation protocol has two levels. Level 1 is grasp success prediction: given a partial-view point cloud of the part (rendered from a random camera viewpoint), does the model predict grasp poses that overlap with the annotated valid set? This is fast to run and fully simulation-based. Level 2 is simulated grasp execution: we load the part mesh and predicted grasp pose into PyBullet, execute the grasp, and measure whether the part is lifted 10 cm off the surface without dropping. Level 2 is more expensive but captures dynamics that Level 1 misses, particularly for grasps near the boundary of the valid set where small positional errors matter.
We do not require physical robot execution for the benchmark. Physical execution is expensive, slow, and subject to hardware variation that makes cross-site comparisons unreliable. The benchmark is designed for simulation-based evaluation, with physical validation as an optional complement rather than a requirement.
What the First Version Got Wrong
Our initial part category definition included "connector bodies" as a single category. After running our first model evaluation, we found that the connector body category had dramatically higher score variance than any other category. When we dug in, the root cause was that we had mixed two very different grasp regimes: connectors with large housing bodies that are straightforward to grasp, and connectors with dense pin arrays that are effectively ungrasplable by a parallel-jaw gripper without a specialized compliant fingertip. Combining them inflated the apparent difficulty of the category and made our results uninterpretable.
We split the connector category into "large-body connector housings" and "fine-pitch connector blocks" and introduced a per-category gripper compatibility flag. Models are now evaluated separately on gripper-compatible and gripper-challenged parts, and we report results broken down by this flag. The gripper-challenged category is included in the benchmark to characterize model behavior at the edge of gripper capability, but it is not counted in the primary success-rate metric.
We also initially weighted all 20 part categories equally in the aggregate metric. This created a distorted view of industrial relevance: threaded fasteners are extremely common in practice but were getting the same weight as irregular prismatic forms, which are rare. We revised the aggregate metric to use frequency-weighted averaging based on our assessment of relative part category prevalence in flexible manufacturing cells, with the weighting documented in the benchmark specification so others can adjust it for their own target contexts.
Release and Use
RMB-2K is available on request for research purposes. The mesh library is released under a license that permits non-commercial research use. The evaluation harness is open source. We are in discussions about extending the benchmark with additional part categories and a physical robot evaluation track, which we would run at a partner facility rather than trying to replicate at every lab that wants to use the benchmark.
We built this benchmark because we needed it, not as a contribution in itself. If it turns out to be useful to other groups working on industrial manipulation, that is a good outcome. If other groups find it has the same blindspots we found in existing benchmarks, we want to know: feedback on the benchmark design is welcome at [email protected].