Teleoperated, In Situ Repair of an Aeroengine: Overcoming the Internet Latency Hurdle

There is a substantial financial incentive for in situ repair of industrial assets. However, the need for highly trained mechanics to travel to the location of a repair often results in inconveniently long downtimes. The emergence of robots capable of replicating human interventions on industrial equipment can be coupled with remote-control strategies to reduce the response time from several days to a few hours.

Abstract-There is a substantial financial incentive for in-situ repair of industrial assets. However, the need for highly trained mechanics to travel to the location of a repair often results in inconveniently long downtimes. The emergence of robots capable of replicating human interventions on industrial equipment can be coupled with remote control strategies to reduce the response time from several days to a few hours. This work outlines the design and remote control strategy for a novel robotic system to carry out repairs on aeroengine compressors in-situ via the internet. A high-level control computer serves as an interface with the skilled operator. A low-level controller receives instruction packets from the high-level controller via the internet and uses them to determine the necessary movements to carry out a machining operation. The robot, comprising a combination of rotary, prismatic and flexible (continuum) joints, was designed to replicate the degree of freedom of hand-held tools. Sensors and encoders on the robot enable the low-level controller to independently detect faults and stop all motion despite the high latency of internet communications. The remote control system was tested by machining stress relief features on eleven compressor blades with a median RMS error of 0.064 mm between the desired and measured blends. A successful demonstration on a production aeroengine shows the capability of the system.

I. INTRODUCTION
D OWNTIME is expensive. Every hour counts when giving maintenance to high value assets, particularly in industries such as power stations for air, land and sea. Repair tasks in complex installations such as gas turbines often require expert technicians with very specific sets of skills travelling around the world to address these maintenance needs. One such repair task is the in-situ removal of stress concentrating defects on gas turbine compressor blades that could be caused by foreign object damage. This repair is currently carried out using slender grinding and polishing tools that are inserted into the compressor though inspection ports. While it is possible to make a robot with the necessary degrees of freedom to replicate the movements of the technician, it is essential that such a robot be controlled by a person with the required skill and experience. The ability to carry out these operations remotely would significantly reduce downtime, but worldwide remote control comes with a number of challenges. To address this gap in capabilities, this work presents a novel system architecture for remotely repairing high-value industrial assets such as aeroengine compressor blades.
In-situ maintenance technicians currently employ a complex customised set of tooling for visualisation, measurement, surface preparation and material removal; often, these are tools specialised for a single type of intervention. These operations are currently performed manually and have limited repeatability and precision. A robotic solution for such inspection and repair tasks would be a breakthrough for high-value asset industries with worldwide operations (e.g. aerospace, energy), opening the possibility for some level of automation of these interventions and eliminating human errors during manipulation of tools. Adding to this the ability to control such an instrument remotely under the supervision of a trained expert would dramatically reduce the time taken to address complex tasks, from several days to a few hours.
The use of robots to automate high-skill tasks and reduce downtime is growing year on year. In [1], Dong et al. demonstrate the use of a long hyper-redundant continuum robot to address in-situ inspection and repair tasks on the intermediate pressure compression stages of an aeroengine. Previous work has also demonstrated the use of track-based robots to address in-situ a number of processes on hydropower equipment, from post-weld heat treatment to grinding [2], [3].
On the other hand, tele-operation of in-situ robotic systems has been explored widely in the field of minimally invasive surgery. Robots with high precision and flexible articulation [4]- [6] are used to operate on patients through a single point of access, reducing the recovery time and allowing for potential remote operation by an expert surgeon anywhere in the world. A well-known, currently in-use system in robotic surgery named DaVinci [7] consists of multiple surgical arms with cutting, grasping and observation end effectors. This device relies on a direct wired connection with the operator-level control system to ensure near-synchronous movement between the surgeon's hand and the robot arm The application considered here is the realization of a stress relief feature on a gas turbine compressor blade by navigating into the compressor via a borescope inspection port, positioning sensors in the required location for measurement of defects, analysing measured data, and removing the required material as a stress relief feature on a dented blade (a procedure known as boreblending). Currently, in-situ boreblending is carried out using manual tools and instruments [8], but a recently developed robotic system (described in later sections) has the capability to replicate the movement of expert hands, take 3D measurements and remove material in the constrained environment, thus enabling a step change in the technology. This article presents a novel strategy for remote control of in-situ robots with high-latency network communication between operator and end effector, and the approach is demonstrated using a recently developed

II. ROBOTIC BOREBLENDING HARDWARE
A boreblending robot was designed to be able to replicate the motion of a skilled mechanic inside an aeroengine inspection port. Borescope ports consist of a series of concentric narrow holes, usually smaller than 10mm. Manual blending tools can be manipulated from outside the engine casing by rotating and inserting them along the axis of the hole (i and ii in Fig. 1) and to a smaller extent, by tilting and pushing the probe to the limits permitted by the inner and outer casing hole geometry (Fig. 1). Articulated probes allow for a tool to be moved towards the affected area (iii), and rotation of the engine shaft (iv) can change the position of the blades relative to the access port.
Manual tools do not change articulation angle during operation and instead rely on using small tilt and push movements to ensure consistent parallel cuts are produced. However, the degrees of freedom required to follow a curved compressor blade can also be achieved with active articulation of the tip. The distance between the axis of the borescope holes and the edges of the adjacent blades is between 21.5mm and 26.7mm for the specific example selected for demonstration, and this defines the available space for articulation of the joints and end effector.
The robot presented in Fig. 2 is composed of a cylindrical outer frame that houses actuators, sensors and a 8.5mm diameter insertion tube and grinding end effector with five degrees of freedom. The outer frame (2) is rigidly attached to the gas turbine casing for the repair intervention. An FHA-8C-50 rotary servo motor (1) (Harmonic Drive, 6, 553, 600p/rev encoder resolution) mounted to the outer frame turns an inner frame about the axis of the borescope port (i). The inner frame (3) consists of three linear rails and a L-220.50SG linear actuator (Physik Instrumente, 0.022µm design resolution, 0.1µm minimum increment) that slides the insertion tube and relevant actuators along inspection port 0.022µm design resolution, 0.1µm minimum increment) (4) are used to drive stainless steel tendons that bend a flexible actuated tip (two tendons for each of ✓ 2 , ✓ 3 and ✓ 4 in Fig. 4) and tilt the end effector (iii); six LCM201 load cells (5) (Omega Engineering, ±100N ) monitor cable tensions.
The insertion tube (7) houses six tendons (6), an air conduct and a Mentor Visual iQ Video Borescope stereo camera (GE Digital Solutions) (8) with a 440k pixel sensor capable of taking 3D measurements of the affected area. The camera is rigidly attached to the insertion tube. Hence, the orientation is determined by the rotation stage and any images or measurements are in the reference frame of the robot.
The flexible segments of the robot were designed to achieve the necessary degree of freedom and to reach the required positions in the compressor. Two main factors, flexibility and buckling, were considered. Flexibility is the maximum bending capability of the flexible segments, which affects the working volume of the end effector. Buckling is the unwanted collapse of the backbone elements, and is one of the major considerations when designing flexible structures [1], [9]. In the design, the total flexible joint length (i.e. the sum of the length of each individual joint) is required to be large enough to allow a sufficient bending angle, but the length of each individual joint must not exceed the buckling threshold since the critical buckling load is inversely proportional to the length of the flexible structure. The critical buckling load is the maximum force that can be applied along the axial direction of the flexible structure before it reaches the stability boundary of its shape.
Three key parameters can be changed to adjust the flexibility and buckling: the backbone diameter D, the total length L of the flexible joint and the number of flexible segment n. The following design process was developed to determine D, L and n: First, a maximum bending angle of 100 in the direction of articulation iii was defined based on the compressor geometry to cover the whole possible repair area. Super-elastic Nitinol rods were selected for the backbone (based on previous experience with the material [1]), and different diameters (0.8mm, 1.0mm and 1.2mm) were considered for the design with a target to keep the strain below 8% (to prevent plastic deformation). To avoid buckling, the flexible joint is divided into multiple shorter segments (Fig. 3a). Given that the total tension of the cables is expected to be around 180N (based on initial worst-case-scenario calculations), each flexible joint was analysed to be able to withstand no less than 200N in the axial direction before buckling. Finally, the length of all the flexible segments are required to be less than 15mm, due to the geometry constraints of the working environment in the engine.
The critical buckling load P cr can be calculated using the Euler buckling theory: where E is the Young's modulus of the material (41GP a for Nitinol), I is the second moment of area, L fr is the flexible rod length and K is an effective length factor which depends on the conditions of end support of the rod (in continuum segments it is assumed to be K = 2 as a worst-case scenario). Hence, two flexible segments were employed in the direction of articulation iii (each with flexible joint length 4mm and joint diameter 0.8mm) and one segment in the orthogonal direction (flexible joint length 1.2mm and joint diameter 0.8mm) as shown in Fig.2. Each of the flex degrees of freedom is actuated by two opposed tendons, which are loaded with as much as 180 N when fully flexed. The end effector (10) consists of a miniature air-driven spindle that has been incorporated in the last stage of the flexible segment, can hold friction grip tools of 1.6mm shaft diameter (such as a 1.8mm diameter cylindrical electroplated diamond grinding tool) and achieve speeds of over 250, 000 RPM.
The high speed and low torque nature of miniature air-driven spindles means that the contact forces between the end effector and the target blade never exceed 1.5 N (stall observed at forces around 1.3 N). The low reaction force during grinding means that it is appropriate to use tendon-driven flexible structures, as the tension in the tendons is more than enough to achieve sufficient stiffness and stability.
The forward kinematics adapts the Denavit-Hartenberg (D-H) convention mechanism to find the angular and displacement values and to calculate the Cartesian coordinates of the end effector.
As the length of the compliant joints is short, it was assumed that the backbone bends into a circular arc when loaded (the assumption was verified experimentally using motion capture equipment). Changes in backbone length due to compresson or flexion were also assumed to be negligible. Like in [10], each backbone is described as two revolving joints in the D-H matrix, linked by a rod of varying length. It can be shown that the straight-line distance d between the two ends of an arc of length a (a curved backbone) at flex angle ✓ is given by:  Table I, where a 1 . . . a 7 are the lengths of the respective sections and is the travel of the insertion linear actuator along ii. The cable displacement calculations assume a straight line between guide hole centres in the bent configuration.
The inverse kinematics uses the Jacobian pseudo-inverse method [14] to find the optimal solution for each joint in multiple iterations. For efficiency, an initial guess is made based on the general direction of the desired position. An average of 25 iterations is expected to arrive at an appropriate solution with error ✏ < 0.1µm (smaller than the effective resolution of the actuators), and therefore a limit is set at 100 attempts before the desired position is deemed out of reach.
A turning tool (Olympus) is a powerful geared stepper motor that is attached to the gas turbine rotor (via a gearbox) and used to rotate the shaft to find the blade requiring repairs and position the blade in the correct position for machining Fig. 5. Inverse kinematics calculation flowchart relative to the access port (iv in Fig. 2); its movements are synchronised with those of the boreblending robot.
III. REMOTE CONTROL SYSTEM ARCHITECTURE The proposed overall system architecture consists of a high-level controller (HLC) with a versatile user interface, a secure network communication protocol, and a self-sufficient and fail-safe low-level controller (LLC) for commanding the robot's mechatronics.
The network communication is managed using a dedicated protocol developed in this work, named compact-RUDP (reliable user datagram protocol). This combines data and live video transfer across the global network and ensures that data transfer is reliable, secure and with priority given to the data control packets. It uses the real-time transport protocol (RTP) to establish the data communication on the web and the session initiation (SIP)/ user datagram protocol (UDP) for registering and identifying the transmitter and receiver transport. An application called OnSight Connect (developed by Librestream [11]) is used to manage the data communication and video/picture streaming at both ends.
Depending on the distance and internet service quality, round trip delays in network communications are often greater than 50ms [12] and can, in extreme cases, exceed 350ms [13]. While delays of 50ms may seem appropriately small when compared to human reaction times, the cumulative nature of the latency means that during a chain of commands, the effective round trip delay between operator action and feedback may exceed the practical limits of remote control. For example, the mean round trip time for two computers running OnSight Connect, one connected to a fibre optic broadband provider (UK-based) and the other connected to the 3G mobile network (UK-based), was measured to be 74ms (standard deviation of 9ms), but after the necessary sequences of automated interactions (e.g. handshakes, DNS look-ups), the effective time between operator action and feedback was measured to be 968ms (standard deviation of 57ms). Thus it is reasonable to expect that this would increase significantly across greater distances on a global scale. Hence, "direct" remote control where the operator commands the position of the various degrees of freedom of the robot and closes the control loop is not a viable option with the current setup.
Thus the proposed solution is to use a HLC to define a machining path using operator inputs, and to send a list of end effector position commands to a LLC for execution, with the ability to request interruptions and further measurements. The LLC interprets the task-space position commands and calculates the required actuator positions using the inverse kinematics of the robotic system. Awareness of the status of the robot, the state of the sensors and the network connection allows the LLC to autonomously send the HLC error notifications, status updates and, in the case of loss of signal, retreat the robot into a safe position and await further instructions. By separating the path generation tasks from the low-level control signals, the latency can be arbitrarily high without impairing the performance of the robot.
The proposed HLC is a computer (Fig. 6) in a remote location from the site of the repair and connected to  the internet. The purpose of the HLC is to serve as a high-level user interface for the expert operator. Hence, all the information that would be available to the operator if they were carrying out the repair in person must be represented in this interface. A number of windows across a few monitors are used to display information relating to the in-situ repair performed by the robot according to the stage in the work flow. For the boreblending application (Fig. 7), these include: • Live video stream from external cameras mounted at the site of the repair. This stream allows the operator to oversee installation of the robotic instrument on the engine inspection port and see the repair environment. • Live video stream from robot's on-board cameras (B).
Once the robot is installed, this stream is used to oversee the tip navigation and aid in positioning the rotor correctly to acquire measurements of the affected area. • Console for commanding the robot to move to inspection, measurement and repair positions. A similar interface can be used to remotely control the engine turning tool and move the rotor until the target blade is in the desired position. • 3D point cloud display of the affected area (A). This console allows the operator to view and re-orient the object, select key measurements and obtain information about the size and shape of the defect. This display is also used to select the points marking the bounds of the defect used to generate the machining path -in the Fig. 7 example, the two points where the defect meets the blade edge and the deepest part of the defect (red dots on A). • Console for machining path generation (C). This allows the operator to define the geometry (e.g. aspect ratio, size and shape) of the stress relief feature depending on the level of damage on the blade. • Console for inquiring on the status of the robot's subsystems, network connection and actuators. This allows the operator to verify that all motors and sensors are working correctly before starting the repair task. • Live simulation of the robot and the repair site environment (D). A rendered 3D model of the environment and the robot's live position gives the operator an internal view that would otherwise be impossible, allowing them to safely navigate the robot. • Display of the current machining operation. This console shows the progress and current status of the machining task and allows the operator to interrupt the blend and take new measurements to verify the robot is performing adequately and the rotor blade has not moved away from the grinding tool. • Live sound from repair site. The operator can communicate with on-site ground crew and hear the grinding tool as it removes material, allowing them to sense changes in machining parameters, tool wear and stalling. Microphones can be mounted on the actuation pack and worn by the technical staff.
The software running on the HLC (Fig. 8) interprets the operator instructions and converts them into data packets containing information about the size, shape and position of the tool path, as well as machining parameters such as depth of cut and surface finish requirements. The data packets are then sent to the LLC to process the operation.
The proposed LLC is a portable computer with internet access located at the site of the repair and connected to the boreblending robot and other hardware required for the repair operation. The LLC has a minimalistic user interface with a console for inquiring about the status of the robot's • A turning tool controller (Olympus) that provides positioning of the compressor rotor • Two C-843 (P.I.) precision linear actuator controllers that drive the robot's tendon stages and insertion actuator • A Gold DC Whistle (ELMO Motion Control) servo drive controller that drives the robot's rotation stage • A M C-iQ stereo vision measurement system (GE) that uses optical metrology to produce a point cloud of the area of interest • An iNET-400 data acquisition system (Omega Engineering) that monitors the cable tensions for the flexible joints • A micro controller (ATMEL) that operates a pneumatic valve to supply the air-spindle-driven grinding tool • An external camera for visualisation of the repair environment The software running on the LLC (Fig. 8) receives the size, shape, position and depth of cut parameters from the HLC and calculates the necessary actuator positions using the Jacobian pseudo-inverse method [14] for inverse kinematics. The trajectory is offset by the tool diameter (similar to the calculation of g-code) and discretised to a list of positions with increments of length 10µm. The inverse kinematics calculations return a joint-space configuration that can be converted into tendon actuator displacement using a modified version of (2). The LLC can navigate the robot to the correct inspection, measurement and machining positions using stored safe-approach trajectories based on the environment geometry.
The LLC is intended to be self-diagnosing and self-protecting. Information from the load cells on tendon stages is used to detect unplanned collisions with the environment (abnormally large loads) or tendon failures (abnormally low loads) and report these back to the operator via the HLC computer. Should a collision occur, the LLC commands the actuators to retreat to the previous safe position and informs the operator. Should an unexpected tendon failure occur, the LLC commands the other tendons and stages to return to a known safe position (assuming the damaged stage is at angle 0 ) to prevent a chain reaction failure. Communication failure with sensors, valves or actuators is informed to the HLC and addressed on a case by case basis. Finally in the event of a network failure, the LLC saves the last position and action in memory and retreats to a position that is safe to be withdrawn from the engine; if connection is restored the operator can choose to continue the blend.
IV. PROPOSED SYSTEM WORKFLOW The remote boreblending process (Fig. 9) starts by initializing the communication between the HLC and LLC using the Onsight Connect application. Using voice communication and site cameras, the operator instructs technical staff on site to mount the robot to the appropriate inspection port and attach the engine turning tool to the gearbox. It should be noted that features such as the spacing and angle of bolts that attach the robot to the engine (the bolts used to plug the borescope access port during flight) are unique for specific ports. Concentric locating features on the robot ensure alignment with the port. It is therefore impossible to accidentally attach the instrument to the incorrect port or in the wrong direction.
Once the robot is attached, the operator can instruct the robot to move an inspection position (with the on-board camera within the gas stream) and use the turning tool console to move the engine shaft until the defective blade is in the correct position in the field of view. The stereo vision measurement system is then commanded to take a measurement of the defect and generate a point cloud, which is used by the operator to define the shape, size and position of the blend.
The robot is then sent the machining data and instructions over the internet to start removing material in small increments. The operator oversees the process in real time (with some latency) using the simulator screen on the HLC to reproduce the movement of the robot, and listens to the grinding tool with the audio stream. Any problems with the robot's tendons or hardware connections are independently handled and reported to the HLC. After some initial time, the operator can pause the operation and instruct the robot to take another measurement, ensuring no tool problems or unexpected misalignment have occurred. Misalignment can be a result of environmental factors such as wind and temperature changes moving the rotors and blades, or it could be caused to a smaller extent by the contact and vibration of the grinding tool. Misalignment can be identified by overlaying consecutive images from the on-board cameras. If a problem is identified, the operator can make corrections to the desired path based on the new point cloud and send new instructions to the LLC. If, based on experience, the operator identifies issues with surface quality or tool wear, the depth of cut can be modified for each pass to cut more or less aggressively. Once the blend is complete, the robot automatically moves to the inspection position to measure the repair for approval. If satisfied, the operator orders the robot to retract from the compressor and communicates with ground staff to remove it from the engine.

V. PROCESS VALIDATION
The system was demonstrated first using a representative test rig in a lab environment and then on an appropriate gas turbine compressor.
To study the repeatability of the system, eleven compressor blades were intentionally damaged on the leading edge with similar defects and mounted on a mock-up compressor stage (Fig. 10). The robot was inserted into the borescope port and the defects were scanned using the embedded stereo measurement camera to identify the target area from the point cloud. Key elements of each point cloud were selected in the HLC user interface to generate a machining path with a pre-defined aspect ratio and shape. The operation was carried out in thirty passes with a maximum depth of cut of 32µm.
Once the blends were completed, each blade was scanned using a structured light 3D scanner (Artec Spider) with a quoted accuracy of 0.05mm. The point clouds were compared to nominal simulated machining curves using the iterative closest point registration method. The results show strong correlation between the point clouds and the nominal curves, Fig. 10. Experimental setup (top) and results (bottom) of repeatability study on compressor test rig with a median RMSE of 64µm and a standard deviation between blades of 32µm (Fig. 10). Consistent dimensions were observed for all blades, confirming the repeatability of measurement, manual point selection and machining. Finally, the system's capabilities were demonstrated on a production aeroengine (Fig. 11) with a previously identified defect. The HLC was connected to the internet using a WiFi Fig. 11. Remote boreblending demonstration carried out on a training aeroengine: A. setup and establishing of internet connection, B. ground support attaching robot to aeroengine, C. robot machining compressor blade and D. finished stress relief blend router, while the LLC was connected using a 3G mobile network device (A). This configuration of internet devices was chosen as a realistic scenario with high likelihood of inconsistent network delay and command latency.
The robot was inserted through a borescope port (B) and manoeuvred for inspection; measurements of the defect were made using the built-in probe and the point cloud was analysed to fit the correct scallop curve. The robot successfully machined the blade (C), demonstrating that despite the likely variability between the rotor position in a lab setting and a real scenario, the point cloud selection methodology accurately aligns the curve with the blade edge. The robot was then retracted to the borescope port to be removed from the engine. After inspection by a trained professional, the shape, size and finish of the scallop (D) were deemed adequate for aerospace standards (the accuracy of the blend was comparable to that observed during the lab trials), demonstrating the practical application of remote robotic boreblending.
While the example scenario chosen for demonstration is fairly specific, the approach is scalable to other in-situ repair tasks usually carried out manually by highly specialised technicians, and could be adapted to suit the maintenance needs of industrial assets such as turbines and compressors or key parts of hazardous hardware such as nuclear reactors.

VI. CONCLUSIONS
This work presents a novel strategy to remotely control in-situ repair instruments for use in high-value industrial assets such as aeroengines. Due to the high latency of world-wide network communications, a conventional remote control system using human inputs and direct feedback is not feasible. Hence, the proposed system architecture uses a high level control computer to display the necessary information to a skilled operator and send commands to a self-sufficient low level control computer connected to the robot.
An operator can use the HLC to view video streams from the repair site, to identify the defect that needs to be repaired on a point cloud, to oversee the repair operation with a rendered simulation of the environment and verify the status of various subsystems. The LLC receives position and machining commands and independently carries out the repair. The LLC uses sensors to identify unexpected problems and retreat to a safe location before alerting the operator. The network communication uses a custom protocol named compact-RUDP that combines reliable data and media transfer.
The system was put to the test by machining a number of stress relief scallops on compressor blades with a shape RMS error of 0.064mm. The target defect was manually selected from point cloud data on the HLC and the shape, size and position of the blend was streamed to the LLC for machining. The demonstration showed the ability of the remote control system to produce consistent results despite human influence when defining the cut.
A successful demonstration was carried out on a production jet engine. A stress-concentrating defect was removed remotely under the supervision of a qualified technician, showing the capability of the system to eliminate the need to travel for in-situ repair and hence dramatically reduce the downtime caused by the operation.
This work presents an approach to tackle remote control tasks for long-distance in-situ repair which is scalable and flexible. The resulting system architecture has been shown to provide value to industries with high-value assets and lays the foundations for remote in-situ repair.