Deep Dynamic Layout Optimisation of Photogrammetry Camera Position based on Digital Twin

Wang, Likun; Wang, Zi; Kendall, Peter; Gumma, Kevin; Turner, Alison; Ratchev, Svetan

Abstract

The photogrammetry system has been widely used in industrial manufacturing applications, such as high-precision assembly, reverse engineering and additive manufacturing. In order to meet the demand of the product variety and short product lifecycle, the factory facilities including photogrammetry devices, should be relocated in response to rapid change in mechanical structure and hardware integration. Nevertheless, the camera position of the photogrammetry system is difficult to select to guarantee an optimal field of view (FoV) coverage of retro-reflective targets during the whole production horizon. Especially in a reconfigurable manufacturing work cell, scaling and calibration of a photogrammetry system requires professional skills and these would cost tremendous labour for rapid configuration each time. In this paper, we propose a novel deep optimisation framework for the photogrammetry camera position for the dynamic layout design based on digital twin. The optimisation framework follows an effective coarse-to-fine procedure to evaluate the FoV visibility over the target frame. In addition, the deep Q-learning algorithm is utilised to find the maximum FoV coverage and avoid collision. Three experiments are implemented to verify the application feasibility of the proposed deep camera position optimisation framework. Note to Practitioners-Large-volume in-process metrology is an essential element in flexible manufacturing systems. Quality of large-volume measurement relies heavily on target visibility within its field of view. In a compact industrial robotic cell, this is extremely challenging as the robot would take the primary position and causing view blockage throughout its operation. This makes the simultaneous monitoring of robot head and the work piece key feature extremely difficult. Manual trial-and-error positioning approach is lengthy and requires high level of expertise, due to both safety and spatial concerns. We approached this problem by simulating the camera's view in a digital twin environment and maximising the target visibility throughout the full operation cycle. The generic framework can provide guidance in metrology setup within automated manufacturing environment, accelerate the system commissioning time, remove dependency of skill level and expand the capability for flexible/reconfigurable manufacturing systems. Although V-STARS camera are used in this application, the framework can be applied for other types of vision systems that requires field of view.

Citation

Wang, L., Wang, Z., Kendall, P., Gumma, K., Turner, A., & Ratchev, S. (2024). Deep Dynamic Layout Optimisation of Photogrammetry Camera Position based on Digital Twin. IEEE Transactions on Automation Science and Engineering, 21(4), 6176-6189. https://doi.org/10.1109/TASE.2023.3323088

• Journal Article Type: Article
• Acceptance Date: Oct 6, 2023
• Online Publication Date: Oct 18, 2023
• Publication Date: 2024-10
• Deposit Date: Oct 18, 2023
• Publicly Available Date: Oct 18, 2023
• Journal: IEEE Transactions on Automation Science and Engineering
• Print ISSN: 1545-5955
• Electronic ISSN: 1558-3783
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Volume: 21
• Issue: 4
• Pages: 6176-6189
• DOI: https://doi.org/10.1109/TASE.2023.3323088
• Keywords: Photogrammetry system; Field of view; Dynamic layout optimisation; Deep reinforcement learning; Digital twin
• Public URL: https://nottingham-repository.worktribe.com/output/26221108
• Publisher URL: https://ieeexplore.ieee.org/document/10287916