Creating a Digital Twin VR-Environment

DUKE is a collaboration project between Lapland University of Applied Sciences (Lapland UAS) and Rovaniemi Municipal Federation of Education (REDU). The idea of DUKE is to examine how VR solutions can be implemented in educational purposes while leveraging renewable energy. Our main pilot is a digital twin of an educational district heating plant located in Rovaniemi and our secondary pilot is a digital twin of a customer grade heat pump.

Digital twins allow us to visualize the environment with great immersion, this enables us to implement accurate functionality to various devices and systems. High immersion helps users to subconsciously connect the virtual object to the real-life object, thus seeing a virtual lever would have the same relationship as seeing a real lever; it should be interactable.

DUKE’s customer grade heat pump was made in cooperation with a project partner. Our partner wanted to incorporate VR solutions alongside their user manual. This way, the user can visualize more accurately basic procedures and maintenance work for the heat pump. Our DUKE engineers tackled this project by working closely with the heat pump specialists. The specialists taught us how to do maintenance on the heat pump and we, in turn, translated that knowledge into a VR simulation. The development process fundamentally revolved around an iterative workflow, where we had assessment and review meetings back and forth. In doing so, we ensure that the 3D-recreation was accurate and at a satisfactory quality.

Creating the digital twin VR-environment

A digital twin is a 3D virtual recreation of an object or a real-world environment. The definition ‘digital twin’ can be perceived as an instruction regarding how closely or realistically the virtual recreation represents its real-world counterpart. Here in Frostbit Software Lab, we aim to create our digital twins to the extent where the 3D-recreation is highly accurate and unmistakable when comparing the 3D-version to its reference.

The workflow and subsequently the challenges we face depends largely on the use case scenario. The VR simulation hosts a physics simulation system in addition to the high-fidelity 3D-visual representation of the environment. For this reason, we chose to use Unity for its lightweight and adaptable nature to implement our real time VR simulation. Unity allows us to easily manipulate and optimize the VR simulation to work with customer grade computers and high-end computers for a higher fidelity experience.

We kicked off the visual implementation process by taking reference images on-site. The images are captured from multiple angles in order to get an understanding of the object’s shape and contour. Moreover, it is exceedingly useful in figuring out the object’s material properties, such as how shiny it is, what type of metal it is and what kind of reflections it produces.

The 3D-modeling process uses blueprints in tandem with the reference images to get accurate measurements for a high-detail recreation. A keen observer may still find some discrepancies when comparing the screenshot image to the reference image. This is due to readjustments, some parts of the district heating plant were deemed obsolete for the VR simulation and therefore had to be cut or modified.

The texturing process uses the multi-angle images to figure out what type of material would be the most accurate to its real-world version. We use HDRi-images to create realistic lighting to the 3D-scene. HDRi’s are essentially 360-degree panoramic images that contain a vast amount of lighting data. Ultimately, we are able to create believable environments by following real-world lighting conditions while incorporating physical based rendering techniques.

VR-Implementation in a game-engine

The first step our developers need to make is to choose an ecosystem, in other words, which development pipeline to use. Locking into a development pipeline is essential in guaranteeing efficient feature implementation down the line. The ecosystem in this context consists of choosing which manufacturers VR-gear to use, what game-engine and development toolkit to use. Our DUKE engineers ended up on using Unity game-engine for its lightweight and adaptable nature.

Our DUKE engineers chose to use SteamVR software development kit for its wide VR-gear support. This SDK provides accessible script blueprints, that we can tailor for our own usage. For example, the SDK provides a basic functionality for object interaction and player movement. This way, more time can be allocated to polish more advanced features such as the dynamic interaction between different objects while a mathematic simulation is running in the background.

The mathematic simulation is built with modularity in mind. Each part of the simulation is broken down to individual pieces. This grants us the customizability to assign the simulation scripts to the 3D-objects that would logically handle the same physics in the real world. For example, the scripts that handle pressure loss are assigned to the 3D-objects that look like valves or pipes. One of the challenges we faced when working with a VR solution was that real-life dimensions did not translate well in a VR environment. EU-standard sized doors seemed too narrow, standardized knobs, bolts and valves seemed too small and ultimately play-testers reported having a difficult time maneuvering in the VR environment overall. A silver lining needed to be struck in order to satisfy the digital twin -nature of the project and the ease-of-use side for the players.

The current DUKE heat pump build contains four scenarios: “Tutorial”, “Water filter”, “Pressure adjustment” and “Water leakage”. Each scenario provides information on how to operate the customer grade heat pump and how to perform basic maintenance.