Bolt Detection and Position Estimation Using Domain Randomization

Current robot automation efforts of aircraft fuselage layer joining are very inefficient; specifically, in detecting fasteners. Present method of localizing fasteners is manually feeding the locations to the robot which is an expensive task for the number of fasteners needed for joining.

We propose using machine learning to identify and locate the position of bolts on a work piece without prior knowledge of their location. The approach is to generate synthetic data using the technique of domain randomization to use in machine learning to identify bolts and their location. The rationale is that with enough variability in the synthetic images, real-world images will appear as another instance of the synthetic data; this process will allow us to work in varying surrounding conditions. With this method we create synthetic images with several instances of the workspace varying lighting, occlusions, textures, and noise and feed it to a neural network which will return a location of the bolt.

In testing the accuracy of this proof-of-concept, a test bed will consist of workpiece embedded with bolts with known locations. Our developed detector will be given real images of the testbed and will (1) detect the number bolts in the image, and (2) estimate the position of each detected object. On our three test sets we can detect bolts with at least 85% accuracy, while for estimating position our method is able to predict 75% of the bolts under 1.27cm error for the first two test sets and on the third test set predict 75% bolts under 2.54cm.