Real-Time Thermal Imaging Using Augmented Reality and Accelerated 3D Models

We present an innovative method to allow real-time visualisation of temperature fields on power electronic circuits using augmented reality. A real-time time-domain simulation incorporating the Block Arnoldi model order reduction techniques is implemented in the Unity game engine. Traditional methods run too slowly whereas the developed system allows an example model with 9232 thermal nodes to be dynamically simulated in real-time at a speed fast enough to allow for a smooth augmented reality visualisation. The aim is to show the feasibility of the technology and that it could be applied to visualising electromagnetic fields in future work.


I. INTRODUCTION
Augmented reality (AR) describes the method of overlaying computer-generated 3D graphics onto a real-time video stream of the real-world environment. This is useful as it can enhance the user's perception of reality by providing informative feedback during interaction with the real-world environment. AR is already widely implemented in other fields and has shown promising results. For example in the medical discipline, Endosight, a novel needle guidance system is used to assist surgeons in interventional procedures [1]. In the automotive and aerospace industries, a range of prototypes have already demonstrated the viability of AR technology, one such example being a truck fuse placement solution which has improved the Mercedes truck production process [2]. There is therefore potential for AR to benefit power electronic system design.
This work aims to integrate AR within a virtual prototyping (VP) workflow in the context of power electronic system design and demonstrates the feasibility of this application of AR by working through a simple thermal example as a proof of concept. A Power Electronics VP tool which is in development is already able to perform multi-domain simulations of power electronic system designs to rapidly predict quantities such as This work was supported by the Engineering and Physical Sciences Research Council, through grant EP/R513283/1 under project reference 2117247 and grant EP/R004390/1. 3D temperature and electromagnetic fields [3]. The fast Multiphysics models in this tool are able to run in real-time and combining the models with measurements and AR allows realtime visualisation of temperature and electromagnetic fields on a live video stream. In this work, the visualisation is of surface temperature on a typical power electronic circuit with surface temperature chosen for the visualisation so that the results can be validated using an Infra-Red imaging camera. The ultimate aim of the work is to extend this to other effects that cannot be visualised with conventional tools such as cross-sectional temperature plots and electromagnetic fields. The proposed system must be able to react in real-time to changes in power dissipation within the device and update the temperature field on the AR visualisation. To accomplish this, there needs to be a voltage and current sensing circuit to send the supplied power value to the AR application, which will then use this as an input to the time-domain simulation for the imported model. An overview of this proposed system can be seen in Fig. 1.
To allow smooth visualisations to happen in real-time, a framerate of at least 20 frames-per-second should be a minimum target (based on 24fps being the accepted standard framerate for cinema). This means that during run-time, each frame should take a maximum of 50ms to complete all necessary calculations and draw calls. Since typical numerical methods that are able to predict field quantities (e.g. the finite-difference method for thermal simulation) are unable to achieve this framerate on a desktop PC with typical specifications, we have used Model Order Reduction (MOR) techniques to reduce the time taken per time-step (for a model with 9232 thermal nodes) from 233ms without MOR to 0.22ms using MOR. This reduced-order model will remain more computationally efficient regardless of the hardware specifications and even when the complexity of the mesh is increased.

II. IMPLEMENTATION
A basic circuit was designed in order to demonstrate the steps required to model the device in the virtual prototyping tool and then visualise its thermal performance in the AR application. The design is made up of two MBR20200CTG Dual Schottky Rectifiers connected in parallel with a power  supply providing a constant DC current, the waste energy being dissipated through the attached heatsink. A real-time AR system has been developed comprising of a dual-channel voltage and current measurement circuit which sends a stream of the real-time readings to the AR application, developed in Unity [4], to use as an input. The two voltage and current measurement channels in the developed circuit allow the power input to each device to be measured independently.
In the schematic (Fig. 2) it shows the two devices connected in parallel. Each MBR20200CTG contains two diodes within the package which are connected together in parallel for the purpose of this example circuit. The application loads a simulation model exported from the VP design tool in matrix form to perform a time-domain simu-lation, solves for temperature using a time-domain simulation, converts the numerical temperature values to a corresponding colour-map, and visualises it on a 3D model of the heatsink. Using ARToolKit [5], an open-source AR framework, the resulting 3D model is overlaid onto a live webcam stream.

A. 3D Model Generation
To begin with, the model geometry and electrical properties are defined in a virtual prototyping design tool under development [3]. This tool allows time-domain electro-thermal simulation of power electronic systems using Model Order Reduction accelerated 3D models. The software takes geometry and material properties as an input, creates a mesh, and then applies a numerical method (the finite-difference method for thermal conduction) to produce a large system of ODEs in matrix form that describe the systems behaviour.
The material properties for the device and heatsink model as defined in the power electronics virtual prototyping tool are given in Table I. Values for the thermal boundary conditions, interface and solder layers were initially estimated, then refined in iterations of the design after observing the effect on the thermal properties of the model and comparing with experimental data from initial tests. The interface materials are very thin layers of material (a thickness of 0.1mm in this example) used to model the non-ideal thermal interface between the devices and heatsink. One possible reason for the difference between the interface layers for the two devices is the screw torque.
The number of sub-divisions and the coarseness of the mesh are adjusted in the software settings to achieve a compromise In order to reduce the calculation time per step, this large system of ODEs is converted to a much smaller system of ODEs using Krylov Subspace based Model Order Reduction techniques, specifically the Block Arnoldi method [6]. This initial pre-processing step takes up 3236ms in this example, but is done only once at the start. When the model size is doubled, it is only the reduced-order model generation time that increases while the execution time remains the same. In this case, the time per step when using the reduced-order model is 0.22ms which is over 1000 times faster per time step than the non-reduced order model. The reduced-order model is much smaller than the full model, only 30 unknowns in this example, which is small enough to run in real-time and it is still able to accurately predict temperature across the entire component. As shown in Fig. 3, matrices M, A, B and C are very large matrices that come from the Finite Difference Method and the respective reduced-order matrices Mr, Ar, Br and Cr are generated through the Arnoldi process which are much smaller, yet when solution y is calculated using the reducedorder matrices at each thermal mesh node, it still yields the same final result when expanded. It is the use of this algorithm that allows a dynamic thermal simulation to be run in real-time and visualised in AR.

B. Exporting Model to AR Application
Once the simulation has been designed to the specification of the user, the model is exported from the virtual prototyping design tool as two separate text files, one containing the geometry data defining the model, and one containing the reducedorder simulation data. For the purposes of this example, the reduced-order matrices were exported from within the VP tool so that the time-consuming order reduction algorithm is only performed once prior to exporting the simulation data rather than at every start-up of the AR application. These files are then parsed in Unity to extract the relevant geometry and simulation data in order to rebuild the model.

C. Real-Time AR Simulation
At every frame, the solution to the reduced order system of equations is evaluated and expanded to get numerical values for the complete temperature field, i.e. a temperature value at each of the 9232 mesh nodes (Fig. 3). A single calculation step alone takes 0.12ms on average to execute in the Unity-based AR application and 0.22ms in the VP tool. To draw the results onto the heatsink, this solution is then mapped to the same colour scheme as the thermal imaging camera and onto the mesh. The live camera frame is then used to detect the position and rotation of the markers around the physical heatsink and therefore to orient the correct transform with which to draw the textured heatsink model. This draw call takes an average of 0.15ms. The combined calculation and drawing per time-step takes 0.27ms in the AR application. This is far lower than the 50ms maximum required for a smooth visualisation at greater than 20fps.

III. VALIDATION A. Testing Procedure
For the purposes of IR imaging, the heatsink and devices were painted black. The devices were then connected to the measurement circuit and to the power supply as shown in Fig. 4. The microcontroller on the measurement circuit was then connected by USB to a computer which was running the Unity AR Application. Also connected to this computer was the webcam affixed above the thermal imaging camera, to stream the live view of the heatsink (Fig. 5). For initial testing, the power electronic circuit was subjected to step change in power dissipation, a total heating current was passed though both semiconductor devices which was set to 0A when t < 0 and 22.5A when t ≥ 0. The voltage and current in each device was measured independently using the two channels in the measurement circuit so that accurate power dissipation values are obtained. These measured values were verified using a voltmeter and current probe to confirm that the measurement circuit readings are accurate.

B. Thermal Waveforms
A spot IR measurement was made at the centre of each device and an equivalent reading was recorded from the AR model. The same was done to obtain the waveforms for two points on the heatsink surface. From these spot markers (Fig. 6), the temperature waveforms were extracted and these are shown in Fig. 7 indicating a very similar heating transient between the simulated and experimental results. A slight difference can be seen between the plotted device temperatures and this is further highlighted in Fig. 8. For the majority of the heating transient, there is less than 5% error. At around 10s, the error in both devices is higher. This suggests that the error could be due to inaccuracies in the thermal interface model that couples the devices to the heatsink. These values were estimated from an initial steady-state calibration test. Work is ongoing to improve the performance of the thermal model but for the purposes of this proof-of-concept, these results are sufficient to illustrate the capability of the real-time AR system.

C. Thermal Spatial Plots
Experimental and simulated temperature plots for the heatsink are shown to agree across the startup transient in Fig. 9. The same temperature scale is used in both cases and the heatsink contours can be seen to match. As mentioned, the slight error in the estimation of the device-heatsink interface layer has resulted in the simulated devices heating slower when t = 100 and heating faster when t = 1400.

D. Performance Analysis
As can be seen in Fig. 10, across a range of model sizes, the reduced-order model significantly outperforms the nonreduced-order model, even when taking into consideration the extra overhead introduced by the pre-processing step.
The advantages of the MOR techniques become exceedingly apparent when using larger models as the time per step remains around 0.1ms when using MOR, whereas for the non-reduced model, the time per step increases exponentially with model size. Comparing the average time per step in the developed Unity AR application with the VP design tool (Fig. 11) shows that both developed tools are able to calculate the solution to the reduced-order model in far less than 50ms. There is no clear relationship between model size and the simulation time for the reduced-order model. The reason for this is that increasing the model size will only affect the model generation time which is irrelevant during a real-time simulation where this pre-processing step can be carried out before the simulation is started. Fig. 12 confirms that the average frames-per-second during a time-domain simulation is consistently above 30fps regardless of the model size and that the AR simulation should run in real-time. The time per step is independent of the model size and therefore, the framerate remains constant and is high enough for a smooth visualisation.

IV. CONCLUSION
The structure for a proof-of-concept tool to allow the realtime visualisation of the temperature fields on power electronic circuits using augmented reality has been presented. The feasibility of this process for surface thermal visualisation has been demonstrated and verified using a thermal imaging camera. A video showcasing the resulting AR simulation has been prepared [7]. We are now in the process of extending this procedure to visualising electromagnetic fields. There is also the possibility to add the ability to measure the junction temperature of semiconductor devices in future work but there would be no way to experimentally validate this.