A Low-cost Automated Fastener Painting Method Based on Machine Vision

. Fastener painting of aerospace structures, in particular legacy products, relies heavily on the skill or rather craftsmanship of the human operator. This process is time-consuming while automated operations with industrial robots can be a more efficient solution. Spray painting robots have been widely used in industry, however, they are not suitable for painting fasteners individually because it will cause a significant waste of materials. Thus, it is essential to develop proper tools and automated methods to replace manual work in order to reduce the cost and improve product quality. This research topic has been receiving more and more attention from both academia and industry. In this article, we present a low-cost and flexible solution for automated fastener painting using a painting dabber and machine vision. A specific nozzle for the dabber is designed to apply paint on fasteners. The system locates the fasteners on aerospace structures with a Cognex camera, then painting is done by the robot with off-line programming. Experimental results show the effectiveness and practicality of automated painting system developed in this paper.


Introduction
Painting is widely used in industry, especially in automotive and aerospace manufacturing. It is an indispensable procedure for engineering machinery products in the process of surface manufacturing, which has various effects such as protection, beautification, water avoidance, antifouling, hidden and so on [1]. As intelligent equipment, painting robots can improve the efficiency and quality of painting operations, meanwhile reducing the labor costs [2]. However, painting robots are usually used in traditional spray painting that applies a coating (paint, ink, varnish, etc.) through the air onto a surface. They are often programmed to follow a specific path and to spray material precisely and consistently, as shown in Figure 1a. The main drawback of a spraying robot is that it is only suitable for surface painting. For the case of fastener painting where paint only needs to be only applied on several discrete regions on a surface, it is not competent anymore.
Meanwhile, traditional painting robots are usually programmed using a teach pendant which, although it is an intuitive programming approach and doesn't require high programming skills, it does suffer from the reliance on the human programmer to make changes to the robot path. This manual teaching process often takes a long time, and during this stage, the robot cannot be used in the production. Moreover, workers are exposed to toxic and hazardous environments. Therefore, this type of automation is not cost effective for aircraft manufacturing where the geometry of the parts and the required operations vary considerably.
In this paper, a low-cost automated fastener painting system is presented. The traditional fastener painting process is usually carried by operators using brushes to paint every fastener on the product, as shown in Figure 1b. This is a highly repetitive task and the manual process is time-consuming and labourious. Moreover, the product quality is based on the operator's skills, experience, and knowledge of the production facilities. The operators must work in a hazardous environment. All such disadvantages will restrict the efficiency of the manual painting process. The proposed system emulates the human operation but in a fully automated manner. A painting dabber is used to apply paint instead of brushes which are unsuitable for an industrial robot to use. A machine vision system is implemented on the industrial robot to determine the position of the fasteners on the structure.

Related Work
One of the challenges in fastener painting is to locate each fastener quickly and accurately. In this perspective, good sensing ability is essential to provide robots with an increased level of autonomy. Vision systems are highly suited for providing the robot's environment information as well as for the recognition of objects and their features, providing information about their position and pose, allowing it to more reliably act in an unconstrained cell. Furthermore, to ensure any automated process is carried out effectively after the manual operation, vision can be used for in-line visual quality inspection to confirm that the assembly's sub-components are mounted within design specification. Machine vision has been used widely in manufacturing and production, in a number of applications [3]. Indeed, the state of the art presents various solutions that use different techniques (both 2D and 3D) such as monocular vision [4], CCD cameras [5] or laser range-imaging [6]. Many types of vision systems and range sensors are available today, but the design and integration of them needs to take consideration of many issues such as; accuracy, speed, reliability, and cost, as well as environmental operating constraints such as; lighting, temperature, and humidity.
Besides good sensing abilities, it is necessary to develop systems that can reduce the time and difficulty associated with tool path programming. In this respect, offline programming software (OLP) [7] has been developed in recent years, both as commercially available products (such as KUKA.Sim for KUKA [8], RobotStudio for ABB [9] and MotoSim for Motoman [10]) as well as in academic research field (for example ROBOMO [11] and Robott [12]). Such software has the ability to model and to present graphically the robot and other equipment, generate programs and then simulate the robot behavior for a given task. However, most software still relies on the teach pendant for robot motion generation, such as tag creation and trajectory planning. Therefore, this paper proposed an automated OLP approach with the help of external sensors. It is low-cost, efficient and sufficient for the use in the manufacturing operations of small and medium-sized enterprises (SMEs).
The remaining content of this paper is organized as follow: Section 3 describes the machine vision-based fastener painting system, including the system overview and detailed techniques. Experimental results carried on an ABB industrial robot are presented in Section 4. Finally, this work is concluded in Section 5.

3
The Automated Fastener Painting System

System overview
The automation of the fastener painting has been thought as a partially automated process where the human operator presents the parts to be operated by the robot, after having performed other manual operations on it previously. Each part is provided by an RFID tag that is scanned before presenting the part to the robot, and that allows uploading from a database the corresponding part information. After the manual operation, the part will be transferred to the painting robot by a conveyor. As the part will be placed randomly on the conveyor by the operator, painting robot must rely on external sensors to detect the exact position and orientation of the part in front of it. Meanwhile, specific painting end-effector will need to be designed as well for different fastener models in a real-world application. The experimental setup of the painting cell. Figure 2 shows the designed fastener painting workbench and the physical setup used for the experiments of the developed system. The robot used in this paper for implementation is the ABB IRB-14000 (also known as YuMi). It is a dual-arm manipulator platform which is designed for small parts assembly. It has only a 0.5 kg payload but this is adequate for small fastener painting. Compared with the large-sized industrial robot, YuMi is cheaper and the manufacturer specified accuracy of 0.02 mm is more than sufficient for painting applications. Also, YuMi is equipped with a built-in safety system, such as padded arms and light-weight design. Therefore, YuMi is quite suitable for working alongside human operators without the requirement for fully enclosed guarding. Meanwhile, YuMi has a built-in machine vision camera module in its right hand. The vision module contains a Cognex AE3 In-Sight 2D camera and provides powerful and reliable vision and identification tools. Table 1 details the camera specification.

Painting process
To emulate the human's skill in the fastener painting process, a painting dabber is used to apply paint on the fasteners, as shown in Figure 3. The painting dabber is composed of a bottle which contains the paint and a nozzle which is used to apply paint. During the usage, a piece of foam is placed inside the nozzle to control the flow of the paint.
The foam will soak up the paint and store it. When the dabber contacts the fastener, the paint will be squeezed out and then applied evenly on the fastener. Nozzles of different sizes can be substituted with respect to different fasteners. Painting dabber is easy to obtain, therefore it is suitable for the proposed low-cost scenario. According to the working principle of the painting dabber, the robotic fastener painting process is designed as follows: Step 1: Locate each fastener with the machine vision system.
Step 2: The robot grabs the dabber and moves to the position that is 20 mm above the located fastener.
Step 3: The robot goes down and keeps the fastener is entirely covered by in the nozzle.
Step 4: Rotate the dabber clockwise by 180 degrees and then rotate 180 degrees counter-clockwise.
Step 5: The robot goes up and moves on to painting the next fastener. The system configuration of vision-guided programming is shown in Figure 4(a). A hybrid position/vision control platform is developed to control the robot motion. The position control is used to maintain the tool orientation. Vision sensing is used to locate the fasteners on the part. The part image is captured by a single camera. Then the image is processed using the PatMax [13] algorithm for the 2D (x and y) coordinates. Figure  4(b) shows an example of the image processing results. The depth coordinate (z) cannot be obtained from the camera. However, it is able to be detected by the contact between the painting dabber and the fastener, according to the variation of the robot's motor torque.

System calibration
One of the most important steps of using vision system is the camera-robot calibration.
The calibration consists of two basic steps. First the camera calibration which converts the image pixels to mm, and second the camera to robot calibration which relates the camera coordinates to a robot frame. Camera calibration is the process of estimating the parameters of a pinhole camera model approximating the camera that produced a given photograph. For 2D cameras, a checkerboard with fiducial is usually used for calibration. The calibration process will find the origin in the camera frame Ƒ which is used to determine the position of the fastener in Ƒ . We denote this position as . Then this position needs to be transformed to the robot frame Ƒ . Finally, we will get the fastener's position in the robot coordinate .

Singularity avoidance
During the movement from the initial robot position to the painting position (the position where the fastener is), the robot may pass through the singularity pose. It is because the fastener is a symmetrical object, the orientation from the vision system may vary from each result although the fastener's position never changes. As shown in Figure 5, two camera results give the same position of the object but completely different orientation. As the motion path is planned by the robot's own controller, it can provide the shortest solution but it cannot avoid the singularity. For example, in Figure 5(b), the robot's wrist joint (joint 6) has to rotate a large angle during the movement. Singularity may occur in this case. Thus, to avoid the singularity, we can simply fix the orientation value that comes from the vision result as zero. Consequently, the robot's wrist joint will not rotate during Step 2. If the initial position of joint 6 is programmed properly (for example, 6 = 5°), the singularity will be avoided in the whole painting process. The joint value is set approximately to zero so that the joint boundary of the 6 th joint will not be exceeded during the rotation in Step 4 in Paragraph 3.2. Another advantage of fixed orientation is that the execution time is largely reduced. For multi fasteners, the robot doesn't need to rotate to the exact results coming from the camera as the orientation result of each fastener can be completely different (It can be clearly seen from Figure 4(b)).

Experimental Results
To evaluate the proposed painting system, a test part with five fasteners was used in the experiment. Software running on the robot is written by the ABB RAPID programming language. Experiments were conducted with the set-up as shown in Figure 2(b). Experimental results are illustrated in Figure 6. From the results, we can see that the fasteners can be successfully painted with the proposed system. It was observed that the vision accuracy could be affected by the ambient light. 5 camera acquisitions of the part were made at the shortest possible time span one after the other at four different times during the day (10 am, 1 pm, 2 pm, 5 pm) in order to test the camera with different levels of light and different light directions. The average errors between the camera results and the real positions are measured for all cases. Meanwhile, we also calculated the rate that the system can successfully recognize all fasteners. Table 2 shows the accuracy and the success rate of each condition. The minimum error is 1 mm in the morning and in the late afternoon while the maximum error is 4.5 mm at 5 pm. The light conditions in the workshops were partially controllable because of a skylight in the ceiling. To achieve greater accuracy, it has been decided to leave the available neon lights always turned on.

Conclusion and Future Work
In this paper, an automated fastener painting system was presented. It is based on a simple machine vision system and a painting dabber to emulate the skilled human operation. An ABB YuMi robot with built-in vision sensor is deployed , which reduces the total system cost. The proposed system is currently able to locate and paint fasteners on flat surfaces. This platform could enable SMEs to cost efficiently automate fastener painting or similar processes such as small part sorting and sealing. The preliminary experimental results presented in this paper validate the effectiveness of the proposed approach.
In future work, we will consider the integration of more sensors, such as the force/torque sensor. Such sensors are able to provide the essential contact feedback information, which can be used to compensate for the error from the camera. For 3D curved surfaces a low-cost 3D sensor (such as the Kinect RGB-D sensor) could be added. To fully evaluate the proposed system, more experiments with standard aerospace structures are required with additional quality control measures to enable process certification.