مبادرة HMI Dojo: تطوير التوأم الرقمي في Unreal Engine 5

Imagine tracking work progress on a construction site not only with your eyes but remotely, whilst being far from the area. How much more convenient would it be to follow the machine’s surroundings not only via windows but with an extended view on a tablet inside the cockpit? Or try to estimate the savings that virtual simulations can offer instead of trying new approaches in real life.

This is the promise of the digital twin concept, a key area of the new HMI initiative run in Spyrosoft’s R&D programme. The project seeks to develop DigiTwin, a digital counterpart to a mini excavator, that replicates the physical machine’s functionality in the virtual realm using Unreal Engine 5 technology. In this article, we walk you through the journey of creating the DigiTwin, its current features and potential enhancements, keeping in mind how such a concept could shape the future of heavy machinery or الزراعة القطاعات.

ما هو تطوير التوأم الرقمي؟

A digital twin development allows the creation of a digital counterpart to a real-world device like a vehicle, construction machinery, or even an entire entity such as a factory, shop, or field. This digital replica mirrors the object’s physical state and allows monitoring of the functions, simulating scenarios or conducting analysis without interfering with reality.

Digital twin supports decision-making for changes in physical facilities. For example, having a virtual copy of a supermarket, you can run scenarios for various cash registers and self-checkout arrangements and then introduce the one with the highest capacity.

تطوير مفهوم التوأم الرقمي في مشروع البحث والتطوير DigiTwin

يُعد DigiTwin مشروعًا محددًا تم تنفيذه كـمبادرة Dojo – برنامج البحث والتطوير الداخلي لدى Spyrosoft. It represents a digital twin of a particular machine, in this case, a mini excavator (hence the name – digger twin). The goal is to track the device’s operation in real-time and interact with it via a dedicated application.

The idea behind the initiative was to provide support for heavy or agricultural machinery as the concept could further be transferred to other devices. In the case of an excavator’s digital twin development, the project fulfils two roles:

• يدعم DigiTwin المشغّل في التحكم بموقع الآلة بشكل أفضل من خلال العرض المرئي على جهاز لوحي داخل قمرة القيادة،
• يتيح ذلك لمدير البناء أو المشرف مراقبة المعلومات الفورية حول الجهاز والوصول إليها، حتى في منطقة جغرافية مختلفة.

Although DigiTwin is still in progress (that is a constant state for R&D projects), several feature enhancements and upgrade ideas are still in the pipeline. Plans include controlling the machine from a tablet, tracking the excavator’s surroundings or supporting the operator, i.e., by transferring construction work parameters to the application and monitoring the level of reaching a target such as the depth or shape of an excavated trench.

DigiTwin كمبادرة Dojo

The initiative aims not merely to create a product but to apply new solutions and pave the way in laboratory conditions before realising a potential project in this area. That’s why the project has been launched as part of our internal R&D, Dojo HMI.

تشمل الأهداف التي حددناها عند إطلاق المبادرة ما يلي:

• توسيع عرض HMI من خلال خدمات التوأم الرقمي،
• تعزيز ظهور Spyrosoft في سوق الزراعة والآلات الثقيلة،
• توسيع الكفاءات التقنية في Unreal Engine 5،
• توسيع قدرات التصميم ثلاثي الأبعاد،
• تعميق المعرفة بخدمات Microsoft Azure IoT.

كيف يعمل DigiTwin؟ التقنيات المستخدمة

يعمل DigiTwin من خلال الاتصال بنموذج مادي لحفارة. يتم توجيه الآلة عبر وحدة تحكم، وهي مزوّدة بأجهزة استشعار مثبّتة تجمع بيانات عن حركتها وموقعها.

Then, data is transmitted to the IoT Hub in the Microsoft Azure cloud. The cloud serves as a connection point between the physical model and its digital twin, represented in the application running on Unreal Engine 5.

The Unreal Engine 5 application receives data from the cloud and mirrors the operation of the excavator in real-time, creating its dynamic 3D model. The person supervising the work may therefore have the physical machine out of sight but see its movements on the digital replica.

الأجهزة والنموذج المادي للحفارة

As none of us had a regular-sized machine in their garage, we bought a metal electric control digger EC160E Volvo Double-E-Hobby which is an exact copy of a 16-tonne excavator scaled in 1:14. As an operator’s tablet, we chose the iiyama ProLite T2736MSC-B1 27″ touch monitor.

أجهزة الاستشعار ووحدات التحكم

As the main microcontroller, we’ve picked the ESP32-S3 running FreeRTOS. It is responsible for sensor data gathering and communication with the Microsoft Azure cloud. The measurements themselves are done with four LSM6DSO IMUs from STMicroelectronics – three as accelerometers for the digger joint (movable arm segment) absolute angles and one as the gyro for the cabin rotation. All the MCU-sensors communication is based on the I²C protocol.

النمذجة ثلاثية الأبعاد في Blender و3ds Max

The 3D model of the excavator for the DigiTwin project was created using Blender and 3ds Max. Both these platforms are well adapted for work with Unreal Engine. Although we faced some difficulties while importing data into Unreal Engine 5, eventually we came up with a refined method. You can read more about the case in the article الرسوم المتحركة الهيكلية في Unreal Engine 5. كيف تستورد نموذجاً ثلاثي الأبعاد؟.

دور Unreal Engine 5

Unreal Engine is a game engine that enables the creation and rendering of advanced, realistic 3D objects and environments. Although primarily known for its usage in computer game production, it also finds application in other areas such as film, animation, architecture, engineering, and medicine.

اخترنا Unreal Engine 5 لمشروع DigiTwin لعدة أسباب:

• this game engine already has mechanisms that simplify the creation of solutions like digital twins. It allows for the rendering of advanced 3D graphics in real-time, which is crucial for dynamically replicating the machine in the application;
• with the growing role of 3D graphics across various industries, game engine manufacturers started paying attention to other markets. We decided to test if Unreal Engine 5 is a good fit for the construction and agriculture sectors;
• the choice of Unreal Engine was also motivated by a desire for team competence development. We wanted to use this project to enhance our skills, showcase the technology’s capabilities, and offer clients solutions based on the engine.

معالجة البيانات في مركز إنترنت الأشياء من مايكروسوفت أزور

As public clouds are one of the most convenient and available hosting and computing solutions, we turned to Microsoft Azure for that project. In the previous Dojo initiative, we got closer to Amazon Web Services, so this time, we agreed that we would get to know the second largest cloud service provider and eventually compare the two platforms.

Also, Azure cloud offers a wide range of services and tools related to the Internet of Things – IoT Hub – which is crucial for a project that involves collecting and transmitting data from sensors. We wanted to test how this module would work with Unreal Engine and how it would operate in practice.

خطط التطوير المستقبلية

As mentioned earlier, the DigiTwin project aims to deepen competencies rather than releasing a market-ready product. The MVP version, created over several weeks, includes “just” a replication of the physical device’s movement in Unreal Engine.

ومع ذلك، لا يزال لدينا العديد من التحسينات والأفكار للميزات في قائمة المهام المتراكمة، بما في ذلك:

• working on the surroundings. We plan to refine the environment in which the machine operates to make it more realistic and mimic the actual surroundings, such as the construction area with blockades or terrain inclines;
• من وجهة نظر المشغّل. تغطي الرسوم المتحركة الحالية المنظر الخارجي، ونعتزم إضافة المنظور الداخلي ليتمكن المشغّل من الحصول على رؤية داعمة على الجهاز اللوحي؛
• expansion to augmented and virtual reality: it is possible to extend the product to augmented and virtual reality capabilities – adding the option of using AR/VR glasses to interact with the machine;
• تطوير عملية الإنتاج. نهدف إلى تطوير البنية التحتية لعملية إنتاج البرمجيات بأكملها، بما في ذلك أنظمة CI/CD واختبارات Unreal Engine؛
• اتصال مباشر بالجهاز يتجاوز السحابة. من بين خططنا مفهوم ربط الحفارة بتطبيق Unreal عبر micro-ROS؛
• الحضور في الفعاليات والمعارض المتخصصة. وبعد ظهورنا الأول في لقاء Learn and Chat المحلي في شتشيتسين وفعالية Car HMI Europe في برلين، نعتزم عرض DigiTwin في فعاليات لاحقة.

النتائج حتى الآن

Thanks to the work on DigiTwin, our HMI team was able to deepen its knowledge and experience in areas such as Unreal Engine 5, Azure, IoT, and the creation of digital twins. Now we will utilise the knowledge to develop the initiative further and launch commercial projects. What’s more, our stand at Car HMI received an extremely positive reception and encouraging feedback from the event’s participants. And still, more is yet to come.

>> هل أنت مهتم ببرمجيات التوأم الرقمي؟ اعرف المزيد حول هذا الحل هنا

تعرف على DigiTwin والخبراء الداعمين خلال يوم الاكتشاف 2023 من PRN

في 16 نوفمبر 2023، سنكون حاضرين في فعالية Discovery Day 2023 التي تنظمها شبكة Pittsburgh Robotics Network في مركز David L. Lawrence للمؤتمرات.

If you would like to check the DigiTwin in action, talk about project implementation or learn about the HMI opportunities for your company, reach us at the event or leave a message via the form below.