A study of an Architecture of Digital Twin Ship with Mixed Reality

Article information

J Navig Port Res. 2022;46(5):458-470
Publication date (electronic) : 2022 October 31
doi : https://doi.org/10.5394/KINPR.2022.46.5.458
*Surveyor, Korean Register
**Senior Surveyor, Korean Register
General Manager, Korean Register
Corresponding author, janghs@krs.co.kr 070)8799-8395
*ejlee@krs.co.kr 070)8799-8596
Received 2022 October 5; Revised 2022 October 11; Accepted 2022 October 25.

Abstract

As the 4th industrial revolution progresses, the application of several cutting-edge technologies such as the Internet of Things, big data, and mixed reality (MR) in relation to autonomous ships is being considered in the maritime logistics field. The aim of this study was to apply the concept of a digital twin model based on Human Machine Interaction (HMI) including a digital twin model and the role of an operator to a ship. The role of the digital twin is divided into information provision, support, decision, and implementation. The role of the operator is divided into operation, decision-making, supervision, and standby. The system constituting the ship was investigated. The digital twin system that could be applied to the ship was also investigated. The cloud-based digital twin system architecture that could apply investigated applications was divided into ship data collection (part 1), cloud system (part 2), analysis system/application (part 3), and MR/mobile system (part 4). A Mixed Reality device HoloLens was used as an HMI equipment to perform a simulation test of a digital twin system of an 8 m battery-based electric propulsion ship.

1. Introduction

The International Maritime Organization reviewed the relevant instruments for the introduction of new technologies and started to develop the MASS Code for the operation of autonomous ships.(IMO, 2022) Currently, the government is expressing its will to support the development of new technologies and financial support for the shipbuilding and offshore industry by carrying out the national R&D Projects for the revival of the main industry in accordance with the paradigm shift in the shipbuilding and offshore industry. The Korean New Deal promotes the development and demonstration of core technologies for ship intelligence, and the new maritime and fishery industry innovation strategy is also promoting the preemptive securing of core technologies for intelligent ships, such as intelligent navigation and automatic berthing.

In the field of maritime transport, discussions on the concept of autonomous ships are ongoing. In the IMO, the concept of autonomous ships has been classified into four and used temporarily. However, it is difficult to establish the concept of an autonomous ship because the technology level of the autonomous system that can be applied differs depending on the onboard systems.

As the latest technology is applied to the industry, terms such as IIoT, digital twin, cyber-physical system, and autonomous system are used to inform the application of this technology, but it is difficult to know the exact level of technology and it is difficult to clearly distinguish between the terms because there are many common features. Many papers review different aspects of digital twin, providing many definitions and descriptions of digital twins.(Fuller et al., 2020; Wilhelm et al., 2021)

Existing papers focus on a specific technology, this paper aims to apply digital twin technology to ships and build an architecture that users can use. Chapter 2, discusses the concept of digital twin applicable to ships by reviewing the recently published papers. Chapter 3 describes the technical analysis of the vessel system that can be configured as a digital twin. In Chapter 4, a user-based digital twin system architecture that can apply the applications investigated in Chapters 2–3 was constructed. Afterward, a cloud-based prototype digital twin which configured as ship data collection (part 1), cloud system (part 2), analysis system/application (part 3), and AR/mobile system (part 4) was connected and tested on an 8-meter battery-powered electric propulsion vessel.

2. Concept of the Digital twin ship

2.1 General definition of the Digital twin

As the term of digital twin suggests, a digital twin is a digital model of a real object. Cutting-edge IT technology has developed dramatically since the 4th industrial revolution and many researches and developments are being conducted in various fields such as manufacturing, city, robot, medical care, and logistics. Before the digital twin concept appeared, a Cyber-Physical System (CPS) was used as a similar concept, and in a broad sense, the Industrial Internet of Things (IIoT) system was also used as a concept similar to the digital twin.

Since the term of digital twin is used in a broad sense and can be interpreted differently by stakeholders, a number of papers have been published that analyzed digital twin technology in various industries and presented the concept. Although a number of papers define digital twins in various aspects, it is commonly agreed that it is essential to transfer the data flow between real and digital objects in both directions.(Tao et al., 2019) An integrated digital system with a physical system can be divided into digital model, digital shadow, and digital twin.(Kritzinger et al., 2018)

Due to the characteristics of ships, in most ships except for coastal ships, it is expected that the seafarer will be on board or will be maintained for a considerable period of time in the operation of the ship either directly or indirectly through remote control. Human Machine Interaction based digital twin model including the ship operator should be applied.(Wilhelm et al., 2021) Therefore, it is necessary to identify the way digital twins support ship operators by each of the ship’s constituent systems, and it is necessary to classify digital twins and the tasks performed by seafarers.

2.2 Technical level definition of Digital twin ship

A ship is composed of multiple systems, and seafarers directly control the systems of the ship by manipulating. Autonomous systems are expected to gradually replace ship seafarers in implementing certain tasks, but when and at what level the replacement would happen will vary depending on the nature of the task and role of seafarers. Thus, for large ships where many autonomous systems of varied levels are operated, clearly defined the role of systems and seafarers, and their interaction must be handled with great importance. In this paper, the technical level of system was defined in consideration of the roles of system and the roles of seafarers.(Thomas B. Sheridan et al., 2014; Wilhelm et al., 2021)

The role of system consists of data collection support, decision and implementation. The level of automation increases as we move from data collection to implementation. The stages of decision and implementation can be seen as the roles to be played by autonomous systems. Role of system can be defined as follows:

  • Information(Data collection): The system collects and stores data on task and relevant status through sensors and provides seafarer with information in a summary

  • Support: Based on collected data, analysis and prediction is carried out to provide information required by the operator in the decision-making process. Analytics and prediction are based on technology such as big data and AI.

  • Decision: Measure to be taken by system is determined through optimization based on data analysis and simulation.

  • Implementation: Autonomous system controls onboard systems based on determined measures.

The level of automation applied to the system and ship operation strategy may change the role of seafarer. Largely, a seafarer’s role is divided into operation, decision, supervision and stand-by, with the role of seafarer on for each task decreasing in the same order. The definition of the role at each stage is as follows:

  • Operation: Conventional role of seafarer regardless of system automation. Seafarer will carry out overall tasks related to onboard system operation with the support or information from the system.

  • Decision: The seafarer exercises overall control of the system based on support and decision by the system. An autonomous system provides seafarers with strategy and insight into the decision.

  • Supervision: The seafarer verifies the system’s decision and implementation rather than making decisions on system operation. A system under operation may be shut down or decision-making authority may be withdrawn if necessary.

  • Stand-by: The role of a seafarer is replaced by the system.

3. Architecture of the digital twin ship

3.1 The architecture for Digital twin ship

A ship is a very complex system composed of various systems. Many technical documents on the concept of digital twin for detailed systems have been published, but the concept of digital twin for ships has not yet been published. ISO 23247 series are standard documents for the digital twin framework of manufacturing published by the International Organization for Standardization. This paper classified the digital twin architecture into 4 areas, and added an operator area in consideration of the ship operation characteristics. In this chapter, each area of the digital twin was identified through the analysis of technical documents and the interaction between the digital twin and the operator was reviewed.

3.2 Information collection area on physical objects

In the information collection area, the ship components synchronize data to applications, cloud systems, stakeholders, and etc, while storing information through various sensors of physical objects. In this area, it is necessary to identify which components in the ship are the physical objects of the digital twin ship.

The technical level applied to each system may differ depending on ship composition and the nature of seafarer task. Thus, to define the level of autonomous technology by system, ship compositions are divided as specified in rows 1 and 2 of Table 2, considering ship structure and function, and seafarer task. At the category level, onboard systems are divided by physical location and function into navigation, berthing (including anchoring and mooring), steering, propulsion, electric power system, hull, cargo storage, loading/discharging, emergency response, security and habitability. Key functions and seafarer tasks a reconsidered for each item to derive the low-level category.

Level of technology applied to individual onboard system

1) O: Operation, D: Decision making, Su: Supervision, St: Stand-by 2) In: Information, S: Support, D: Decision, I: Implementation

3.3 Analysis area for digital tiwn

The analysis area is a crucial area that analyzes the data collected by the ship and provides a model or information required for the application area. This area is divided into analytics technology and infrastructures for data analysis. The first is data analysis technology. The term data analytics focuses on collecting, analyzing and storing data to gain greater insight.

Machine learning and deep learning are representative data analysis algorithms, and deep learning is a subclass of machine learning. Machine learning is characterized by generating algorithms that allow computers to act through data learning without developers directly programming. In conjunction with the system that automatically collects data, the system takes over the repetitive tasks previously performed by seafarers. It is generally used for systems related to maintenance.(Coraddu et al., 2014; Uyanık et al., 2020) Deep learning learns more data and provides higher accuracy to machine learning that can scale up the size of neural networks. It has shown innovative performance improvement in fields that have previously shown limitations, such as image recognition and natural language processing.

The second is the analytics infrastructure. Analytics technology requires continuous learning and analysis on new data. Most systems in maintenance do not require real-time response, and the method of learning and reasoning on board is not efficient in terms of maritime communication environment or cost. Therefore, strategic use of a onshore infrastructure is required depending on the application.

3.4 Application area

The application area is a service provided to users through data analysis. Applications are generally developed and operated based on data collection and analysis systems.

The main applications of the first column of Table1 were classified into navigation, maintenance, energy management system, automation, and seafarer health in consideration of the functions of the system and the task of the seafarer. Navigation, maintenance, and energy management systems are fields where data-based systems, like big data and artificial intelligence systems, are applied. The systems of other ships were classified as an automated system, and systems related to the safety of seafarers were classified as crew health. In the second column, the functions proposed in the technical documents was classified. It was necessary to classify the functions into the existing automation system and the data-based system. General automation functions are classified into monitoring, optimal control and control, and the data based functions were explicitly written in the system name.

The functions of main application of DT

3.5 Area of interaction with stakeholders

Although digital twin technology has reached a considerable technical level and has reached the level of unmanned technology in some fields, since ships are operated at sea, an operation strategy through interaction between seafarer and system is required rather than relying on the system. Recently, the International Maritime Organization has been discussing how to consider remote operators as seafarers through the MASS code. As a new entity is introduced into the maritime system, it is necessary to clarify the role of the seafarers and the role of the digital twin.

3.6 Review of the level of technology applied to individual onboard system

Table 2 is a summary of our analysis on common applications by the onboard system and their technical level based on a review of technical materials. For the classification of onboard systems, column 1 and 2 referred to the Autonomous ship Guideline published by Korean Register. In column 3, which is the main application, according to the onboard systems, Table 2 was used to classify them, and in columns 4 and 5, these were classified according to the roles of seafarers and onboard systems presented in Chapter 2.2. In the case the level of autonomous technology was unclear in the technical material, expert opinion was sought.

In the case of navigation and berthing (including anchoring and mooring) systems, the system is capable of deciding on the navigation route, steering and controlling propulsion under controlled conditions. The system for recognizing the external environment collects and analyzes data on the external environment to support the system in planning movement within port, collision avoidance and navigation. As for technology applied to navigation, the role of system includes informing, support, decision making and implementation, and that of seafarer include stand-by, and decision making or supervision depending on the ship operation environment.

As for propulsion and hull, systems are mainly used to support seafarer in terms of maintenance and repair. In terms of maintenance and repair, system provides information and support, while seafarer engages in operation based on system support. The operation has traditionally been and still remains the role of seafarer. Although the system is capable of analyzing the state of a facility, it would not be able to predict a failure in an accurate manner and execute the repair itself in the near future. Therefore, the mentioned role of seafarer will remain.

As for the power system, technology for energy management and maintenance and repair are mainly applied and used. For energy management, system carries out all four roles while seafarer serves as a supervisor that participates in decision-making process when necessary. For maintenance and repair, system provides information need in making decisions, and seafarer engages in operation. Given the various advantages in maintenance and repair, battery-based electric propulsion systems have been applied to MASS for short-distance coastal operations in many cases.

Currently, cargo storage, loading/unloading and housing system are categorized as automation system, and engage in repetitive tasks based on embedded system. For automation, system plays the role of information collection and implementation of tasks. When making decisions, system relies on pre-determined values by seafarer rather than data analysis. The role of seafarer in such case is to supervise the decision-making process if necessary. For maintenance and repair, system monitors the condition and provides analysis results to the seafarer as support. Seafarer carries out tasks based on support from the system.

In the case of emergency response and security, monitoring technology utilizing sensors such as cameras and data analytics are being used for fire detection, search of distress and detection of infiltration by system. To ensure safe operation of MASS, there are currently efforts to develop technology that enables the system to respond actively after detection. However, it is mainly the role of seafarer to engage in operation for incident follow-up. In terms of escape, the system identifies risk areas on board, and supports seafarer and passengers to escape through the safest route.

4. Development of prototype application

In Chapters 2 and 3, we performed a technical analysis on which vessel system can be configured as a digital twin and what information can be provided to the seafarer. To apply the vessel’s application function investigated in Chapter 3 to the digital twin system, it is necessary to set up a development environment that connects the physical vessel and the digital twin system. Therefore in Chapter 4, a four-part digital twin architecture was constructed and tested. In Part 1 as a physical object, an 8-meter-sized battery-based electric propulsion ship (Figure. 3) was prepared for this study. In Part 2, the Microsoft Azure cloud system (Figure 2) was implemented, and in Part 3, the vessel’s monitoring system was applied. In Part4, HoloLens was used to configure and test MR-based real-time monitoring systems shown in Figure 5 and 6.

Fig 3.

Physical object of the electric propulsion vessel

Fig 2.

Cloud-based digital twin system architecture

Fig 5.

Application operations with MR

Fig 6.

Ship monitoring system with MR

4.1 Architecture

Based on the digital twin architecture analyzed in Chapter 3, a digital twin architecture suitable for electric powered ships was constructed. The digital twin architecture of electric propulsion ship is divided into 4 parts as shown in Figure 1. As a physical model of digital twin, Part 1 is equipped with ship IoT system that can collect data inside and outside the ship and marine communication facilities for the linkage with land. Part 2 is a digital model on the cloud that implements digital twin model by receiving data on ship. Part 3 is a MR system that can visually check information on ship and information on cloud. Part 4 is an application that can provide the optimal solution through digital model. ML model, real-time monitoring, battery monitoring, recognition of the surrounding environment and maintenance are included.

Fig 1.

Digital twin diagram

4.2 Digital model (Part 1)

It is necessary to secure 3D CAD files on the ship and equipment to visualize 3D model of the physical vessel. For equipment, CAD file that is used in the design stage was used, and direct modeling was made for the hull model. After that, it is necessary to convert model files and do optimization work of model to upload the files to application and operate it in the user device. In this study, Unity program was used to develop an application and Pixyz program was used for optimization. Table3 shows the results of implementing Tessellation and Decimate after performing Remove hole, Hidden Removal and Merge are by using Pixyz are as follows.

Number of polygons comparison

Model completed with optimization and uploaded to the application is shown in Figure 4. HoloLens limits the number of polygons in one scene by less than 100,000 for the smooth operation of the application.(J. Lee et al., 2017) However, the number of polygon in the entire model that performed optimization was 182,312 which was not satisfied with the limits. In particular, the number of polygon of motor was 138,637 which accounted for the largest portion. Therefore, additional modeling work is necessary for the weight reduction of the motor model in the future.

Fig 4.

Drivetrain 3D model

4.3 Network (PART 1 – PART 2)

The electric propulsion ship and onshore center communication system consists of one network using Virtual Private Network. In addition, it is composed in parallel with Wifi and LTE in consideration of the unstable communication conditions of the state of sea. It is necessary to connect to Azure Cloud through network on the ship to upload real-time data. Therefore test was performed by connecting prototype server with the network on the ship.

To test the network connection, a packet internet groper was sent to each server to check the response and response speed. Test results confirmed that the respond speed of the data was about 24ms on average. Then, WebSocket was used to test the connection between the server and the application and the server and the cloud, and the results confirmed that it successfully received real-time data through the local IP and port number.

The minimum data transmission cycle of the data transmission system is 1 ms, but it was confirmed that the bottleneck phenomenon occurs if the reception cycle gets shorter. This issue will be analyzed in the future by using the performance profile of the program.

4.4 Cloud system (PART 2)

Azure cloud system was used to visualize the data of digital twin system and implement the function of analysis and prediction. The internal cloud system was divided into five sectors which received data from the ship to be transmitted to users, as shown in Figure 2. The major functions of each sector are as follows.

4.4.1 Ingest and Device Management

IoT Hub collects and manages real-time data of the ship received from the server and the number of connected devices and the number of messages used each day can be checked. Collected data is transmitted to Digital twin sector through message routing.

4.4.2 Digital twins

As a digital replica of an actual object, it draws improved results through control and simulation analysis. It collects, processes and analyzes the data and integrates them with AI service to analyze the patterns.

4.4.3 Serving storage

As a data storage system, it can save the message and file data transmitted to the ship and result values transmitted to the digital twin system. It was designed in SQL in consideration of data structure extracted from the ship and Cosmos DB was used for a function that requires quick response of the database.

4.4.4 Operational rules and insights (PART3)

It belongs to AI service of the digital twin system which can manage model learning and the distribution by using Azure Machine Learning. It is scheduled to be applied to the battery failure diagnosis system and power quality system. It is planned to conduct additional linkage tests in the future.

4.4.5 Solution network

It is responsible for the communication function to transmit data from Azure Digital Twin and Azure Machine Learning system to user application. SignalR communication is a real-time web communication technology using WebScoket method which automatically processes connection between clients and services. It has the advantage of fast data transmission speed.

4.5 Implementation (PART4)

HoloLens is a Microsoft-developed mixed reality-based device that provides a blend of physical and digital worlds. In this study, we developed a prototype digital twin application of cloud-based electric propulsion vessel using HoloLens as the HMI device. On the left of Figure 4, a physical object of an electric propulsion vessel is shown, and a digital twin model is placed on the right. Figures 4 and 5 show the operation of the application using HoloLens, in which a dashboard operating real-time data of the vessel is implemented. As a result, it was confirmed that users can see real-time information of the ship through the application by receiving real-time data to the cloud using the communication network on the ship. Users can operate the application by using Mixed Reality based HoloLens and mobile devices as well as 360-degree view and current state of the ship by moving the position. In addition, they can interface with the application through screen touch and hand tracking.

5. Conclusion and future plans

In this paper we summarized the concept of HMI-based digital twin that can consider interaction between digital twin and crews or operators before developing digital twin system of the ship. And then we investigated the roles and application technologies of each system after classifying the system that is made up of the ship to analyze the ship system. The ship system was subdivided into Navigation & Communication, Berthing, Steering Propulsion, Power System, Hull, Cargo storage, Loading/Discharging, Emergency response, Security and Habitability depending on physical locations and functions, and the roles of operators were divided into operation, decision, supervision and stand-by. The roles of digital twin were divided into information, support, decision and implementation. Application corresponding to each system was classified into navigation, maintenance, energy management, automation and seafarer health to specifically present the digital twin of the ship.

For this research, 8-meter-sized battery-based electric propulsion vessel was prepared, and cloud-based prototype digital twin system was implemented by using the azure cloud system. With the digital twin architecture constructed in this paper, an interworking test was conducted for the maintenance (monitoring) function among the user-based digital twin application and the actual vessel. Digital model is imported to the development program through weight lightening of CAD files and conversion of the file format. And then it was made to transmit and send data to the cloud by using ship communication system to receive real-time data. Users can use HoloLens device as HMI device and see real-time sensor data of the vessel through the developed application.

It is planned to apply and test several function of digital twin algorithms to the configured cloud-based digital twin system.

Acknowledgements

This research was a part of the project titled ‘Development of Smart Port-Autonomous Ships Linkage Technology’, funded by the Ministry of Oceans and Fisheries, Korea

References

1. Ai B, Jia M, Xu H, Xu J, Wen Z, Li B, Zhang D. 2021;Coverage path planning for maritime search and rescue using reinforcement learning. Ocean Engineering 241:110098. https://doi.org/10.1016/J.OCEANENG.2021.110098 .
2. Alexeev VA, Walsh JE, Ivanov V. v, -, al, Dobricic S, Russo S, Lagunov A, Podorojnyak N. 2020;Building a Monitoring System for Bunkering Operations in the Arctic. IOP Conference Series: Earth and Environmental Science 459(4):042052. https://doi.org/10.1088/1755-1315/459/4/042052.
3. Beko M, Ivosević Š, Dlabač T. 2021. Application of Sensors in the Corrosion Monitoring of the Ship’s Structural Parts. 2021 25th International Conference on Information Technology, IT 2021 https://doi.org/10.1109/IT51528.2021.9390091.
4. Cai DL. 2019;Analysis of Steering Angle Control of Ship Variable Frequency Hydraulic Steering Gear Based on Labview. Journal of Applied Science and Engineering 22(1):49–56. https://doi.org/10.6180/JASE.201903_22(1).0006.
5. Choi HT, Park J, Choi J, Kang M, Lee Y, Jung J, Kim J, Kweon H, Kim J, Yoon KJ, Kim H, Park ST. 2021;Design and Preliminary Results of Novel Situational Awareness Systemfor Autonomous Ship based on Artificial Intelligence Techniques. Journal of Institute of Control, Robotics and Systems 27(8):556–564. https://doi.org/10.5302/J.ICROS.2021.21.0063.
6. Choi J, Yang CS. 2017;Smart Escape Support System for Passenger Ship : Active Dynamic Signage & Real-time Escape Routing. Proceedings of the Korean Institute of Navigation and Port Research Conference :79–85.
7. Coraddu A, Lim S, Oneto L, Pazouki K, Norman R, Murphy AJ. 2019. A novelty detection approach to diagnosing hull and propeller fouling. Ocean Engineering 17665–73. https://doi.org/10.1016/J.OCEANENG.2019.01.054 .
8. Coraddu A, Oneto L, Ghio A, Savio S, Anguita D, Figari M. 2014;Machine learning approaches for improving condition-based maintenance of naval propulsion plants. Proceedings of the Institution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment 230(1):136–153. https://doi.org/10.1177/1475090214540874.
9. Fuller A, Fan Z, Day C, Barlow C. 2020. Digital Twin: Enabling Technologies, Challenges and Open Research. IEEE Access 8108952–108971. https://doi.org/10.1109/ACCESS.2020.2998358.
10. Gaber M, El-Banna SH, El-Dabah M, Hamad MS. 2021. Intelligent Energy Management System for an all-electric ship based on adaptive neuro-fuzzy inference system. Energy Reports 77989–7998. https://doi.org/10.1016/J.EGYR.2021.06.054.
11. Guan W, Peng H, Zhang X, Sun H. 2022;Ship Steering Adaptive CGS Control Based on EKF Identification Method. Journal of Marine Science and Engineering 2022 10(2):294. https://doi.org/10.3390/JMSE10020294.
12. IMO. 2022;MARITIME SAFETY COMMITTEE 105th session Agenda item 7 - DEVELOPMENT OF A GOAL-BASED INSTRUMENT FOR MARITIME AUTONOMOUS SURFACE SHIPS (MASS). :81.
13. Ji R, Yang Y, Niu J. 2021. Ship Fire Detection Methods Based on Machine Learning. Proceedings - 2021 International Conference on Information Science, Parallel and Distributed Systems, ISPDS 2021 231–234. https://doi.org/10.1109/ISPDS54097.2021.00052.
14. Kawakami T. 2016;Communication systems for safety and security of ships. Journal of KONES 23(1):153–160. https://doi.org/10.5604/12314005.1213547.
15. Kritzinger W, Karner M, Traar G, Henjes J, Sihn W. 2018;Digital Twin in manufacturing: A categorical literature review and classification. 51(11):1016–1022. https://doi.org/10.1016/j.ifacol.2018.08.474 .
16. Lakshmi E, Priya M, Achari VS. 2021;An overview on the treatment of ballast water in ships. Ocean & Coastal Management 199:105296. https://doi.org/10.1016/J.OCECOAMAN.2020.105296.
17. Lee ES, Mokashi AJ, Moon SY, Kim GS. 2019;The Maturity of Automatic Identification Systems (AIS) and Its Implications for Innovation. Journal of Marine Science and Engineering 2019;7(9):287. https://doi.org/10.3390/JMSE7090287.
18. Lee JH, Kong KJ, Jung BK. 2018;Performance analysis of dynamic positioning system with loss of propulsion power of T/S NARA. Journal of the Korean Society of Fisheries and Ocean Technology 54(2):181–187. https://doi.org/10.3796/KSFOT.2018.54.2.181.
19. Lee J, Kim S, Kim L, Kang J, Lee S, Kwon S. 2017;A Study on Virtual Studio Application using Microsoft HoloLens. International Journal of Advanced Smart Convergence 6(4):80–87. https://doi.org/10.7236/IJASC.2017.6.4.12.
20. Lee JM, Chung WJ, Lim DJ, Choi KS. 2019;A Study on the Development of Hall Effect Sensor for Hydraulic Locking Alarm in Ship’s Steering Gear. Journal of the Korean Society of Manufacturing Process Engineers 18(1):116–121. https://doi.org/10.14775/KSMPE.2019.18.1.116.
21. Li C, Wang D. 2018;Knowledge-Based Engineering-based method for containership lashing bridge optimization design and structural improvement with functionally graded thickness plates. Proceedings of the Institution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment 2019;233(3):760–778.
22. Lin X, Wang S, Sun Z, Zhang M. 2021. YOLO-SD: A Real-Time Crew Safety Detection and Early Warning Approach. Journal of Advanced Transportation 21–12. https://doi.org/10.1155/2021/7534739.
23. Roberts GW, Hancock C, Klug F, Lienhart W, Zuzek N, Ligt H. 2019;Deflection Monitoring and frequency response of a Ship using GPS and Fibre Optic based sensors. Joint International Symposium on Deformation Monitoring
24. Ma Y, Maqsood A, Oslebo D, Corzine K. 2021. Comparative analysis of data driven fault detection using wavelet and fourier transform for DC pulsed power load in the all-electric ship. Conference Proceedings - IEEE Applied Power Electronics Conference and Exposition - APEC 1343–1347. https://doi.org/10.1109/APEC42165.2021.9487090.
25. Mihai V, Rusu L, Alavi AH, Vafai K. 2021;An Overview of the Ship Ventilation Systems and Measures to Avoid the Spread of Diseases. Inventions 2021 6(55) https://doi.org/10.3390/INVENTIONS6030055.
26. Miyauchi Y, Sawada R, Akimoto Y, Umeda N, Maki A. 2022;Optimization on planning of trajectory and control of autonomous berthing and unberthing for the realistic port geometry. Ocean Engineering 245:110390. https://doi.org/10.1016/J.OCEANENG.2021.110390.
27. Pan J, Hou W, Wang L, Lv H, Zhang J, Huang J. 2022;Sealing state monitoring of watertight door using fiber Bragg grating sensor with arc-shaped structure. Optical Engineering 61(4):047102. https://doi.org/10.1117/1.OE.61.4.047102.
28. Parsons MG, Kotinis M. 2011;Trim and Draft Control Capability of the Variable Buoyancy Ship. Journal of Ship Production and Design 27(3):118–126. https://doi.org/10.5957/JSPD.2011.27.3.118.
29. Perabo F, Park D, Zadeh MK, Smogeli O, Jamt L. 2020. Digital Twin Modelling of Ship Power and Propulsion Systems: Application of the Open Simulation Platform (OSP). IEEE International Symposium on Industrial Electronics 1265–1270. https://doi.org/10.1109/ISIE45063.2020.9152218.
30. Poggi L, Gaggero T, Gaiotti M, Ravina E, Rizzo CM. 2020. Recent developments in remote inspections of ship structures. International Journal of Naval Architecture and Ocean Engineering 12881–891. https://doi.org/10.1016/J.IJNAOE.2020.09.001.
31. Santiago Caamaño L, Galeazzi R, Nielsen UD, Míguez González M, Díaz Casás V. 2019;Real-time detection of transverse stability changes in fishing vessels. Ocean Engineering 189:106369. https://doi.org/10.1016/J.OCEANENG.2019.106369.
32. Shen H, Liu C, Yin Y, Sun X. 2020;Automatic Stowage of Bulk Carrier Based on Ship Floating-State Control. Journal of Ship Research 64(03):298–312. https://doi.org/10.5957/JOSR.11170070.
33. Sheridan TB, Parasuraman R. 2005. HumanAutomation Interaction. Reviews of Human Factors and Ergonomics 189–129. https://doi.org/10.1518/155723405783703082.
34. Simon N, Bordoy J, Hoflinger F, Wendeberg J, Schink M, Tannhauser R, Reindl L, Schindelhauer C. 2015. Indoor localization system for emergency responders with ultra low-power radio landmarks. Conference Record - IEEE Instrumentation and Measurement Technology Conference 309–314. https://doi.org/10.1109/I2MTC.2015.7151285.
35. Song BS, Lim KH, Lee BH, Cho DJ, Lee HJ, Jang JH, Chang KH. 2019;SMARTNavigation over Pilot LTE-Maritime: Deployment and Coexistence with PS-LTE. IEEE Communications Magazine 57(9):126–131. https://doi.org/10.1109/MCOM.2019.1800260.
36. Su S, Ouyang M, Li T, Wang Q. 2021;Application of ship electric propulsion based on internet of things system and electronic system. Microprocessors and Microsystems 81:103748. https://doi.org/10.1016/J.MICPRO.2020.103748.
37. IMO. 2022;Sub-Committee on Pollution Prevention and Response (PPR 9). :59.
38. Tam JH. 2020;Overview of performing shoreto-ship and ship-to-ship compatibility studies for LNG bunker vessels. Journal of Marine Engineering & Technology 21(5):257–270. https://doi.org/10.1080/20464177.2020.1827489.
39. Tao F, Sui F, Liu A, Qi Q, Zhang M, Song B, Guo Z, Lu SCY, Nee AYC. 2019;Digital twin-driven product design framework. International Journal of Production Research. International Journal of Production Research 57(12):3935–3953. https://doi.org/10.1080/00207543.2018.1443229.
40. Tsang YP, Choy KL, Wu CH, Ho GTS, Lam HY, Koo PS. 2017;An IoT-based cargo monitoring system for enhancing operational effectiveness under a cold chain environment. International Journal of Engineering Business Management :9.
41. Tusher HM, Munim ZH, Notteboom TE, Kim TE, Nazir S. 2022;Cyber security risk assessment in autonomous shipping. Maritime Economics & Logistics 2022 24(2):208–22. https://doi.org/10.1057/S41278-022-00214-0.
42. Uyanık T, Karatuğ Ç, Arslanoğlu Y. 2020. Machine learning approach to ship fuel consumption: A case of container vessel. Transportation Research Part D: Transport and Environment 84102389. https://doi.org/10.1016/J.TRD.2020.102389.
43. Vaneeckhaute C, Fazli A. 2020. Management of ship-generated food waste and sewage on the Baltic Sea: A review. Waste Management 10212–20. https://doi.org/10.1016/J.WASMAN.2019.10.030.
44. Vassalos D. 2020;The role of damaged ship dynamics in addressing the risk of flooding. Ship and Offshore Structures 17(2):279–303. https://doi.org/10.1080/17445302.2020.1827639.
45. Wilhelm J, Petzoldt C, Beinke T, Freitag M. 2021;Review of Digital Twin-based Interaction in Smart Manufacturing: Enabling Cyber-Physical Systems for Human-Machine Interaction. International Journal of Computer Integrated Manufacturing 34(10):1031–1048. https://doi.org/10.1080/0951192X.2021.1963482.
46. Xue S, Wu S, Tang Q, Liu S, Liu B. 2021;Research on torsional vibration monitoring system of ship power shafting. IOP Conference Series: Materials Science and Engineering 1207(1):012007. https://doi.org/10.1088/1757-899X/1207/1/012007.
47. Yang C, Huang F. 2016;An overview of simulation-based hydrodynamic design of ship hull forms. Journal of Hydrodynamics 28(6):947–960. https://doi.org/10.1016/S1001-6058(16)60696-0.
48. Yu YH, Baloch SR. 2019;Real time fault detection and diagnosis system for onboard engine room. Journal of Advanced Marine Engineering and Technology 43(10):810–815. https://doi.org/10.5916/JKOSME.2019.43.10.810.
49. Zhang L, Mou J, Chen P, Li M. 2021;Path Planning for Autonomous Ships: A Hybrid Approach Based on Improved APF and Modified VO Methods. Journal of Marine Science and Engineering 2021 9(7):761. https://doi.org/10.3390/JMSE9070761.
50. Zhang X, Wang C, Jiang L, An L, Yang R. 2021. Collision-avoidance navigation systems for Maritime Autonomous Surface Ships: A state of the art survey. Ocean Engineering 235109380. https://doi.org/10.1016/J.OCEANENG.2021.109380.
51. Zhu G, Guo X, Yi Y, Tan W, Ji C. 2020. Experiment and simulation research of evolution process for LNG leakage and diffusion. Journal of Loss Prevention in the Process Industries 64104041. https://doi.org/10.1016/J.JLP.2019.104041 .

Article information Continued

Fig 1.

Digital twin diagram

Fig 2.

Cloud-based digital twin system architecture

Fig 3.

Physical object of the electric propulsion vessel

Fig 4.

Drivetrain 3D model

Fig 5.

Application operations with MR

Fig 6.

Ship monitoring system with MR

Table 1.

The functions of main application of DT

Main application Functions
Navigation Situation awareness, Communication, Risk analysis, Control
Maintenance Monitoring, Failure analysis and prediction, Failure removal
Energy management Monitoring, Optimization, Power quality
Automation Monitoring, Optimization, System control
Seafarer health Monitoring

Table 2.

Level of technology applied to individual onboard system

1) O: Operation, D: Decision making, Su: Supervision, St: Stand-by 2) In: Information, S: Support, D: Decision, I: Implementation

System Classification Main application Seafarer Role1) System Role2)
Category Sub-Category O D Su St In S D I
Navigation & Communication Position identification (E. S. Lee et al., 2019) Navigation (Situation awareness)
Environment recognition (H. T. Choi et al., 2021) Navigation (Situation awareness)
Internal/external communication (Song et al., 2019) Automation (Monitoring)
Collision avoidance (Short term) (X. Zhang et al., 2021) Navigation (Control)
Voyage plan (Long term) (L. Zhang et al., 2021) Energy Management (Optimization)
Berthing (Including anchoring and mooring) Position maintaining (LEE et al., 2018) Navigation (Control)
Navigation within ports (Miyauchi et al., 2022) Navigation (Control, Situation awareness)
Steering Steering power generation (Lee et al., 2019) Automation (Control, monitoring)
Rudder angle control (Cai, 2019) Automation (Control)
Rudder angle monitoring (Guan et al., 2022) Automation (Monitoring)
Propulsion Main engine (Mechanical) (Yu & Baloch, 2019) Maintenance (Analysis, Prediction)
Energy Management (Condition monitoring)
Electric propulsion motor (Su et al., 2021) Maintenance (Analysis, Prediction)
Propulsion power transmission (Xue et al., 2021) Maintenance (Analysis, Prediction)
Power System Generator (Perabo et al., 2020) Maintenance (Analysis, Prediction)
Power distribution (AC, DC) (Ma et al., 2021) Energy Management (Power quality)
Sources (Battery, Fuel cell, etc.) (Gaber et al., 2021) Maintenance (Analysis, Prediction)
Hull Lines of hull (Yang & Huang, 2016) Maintenance (Monitoring, Analysis)
Stability (Santiago Caamaño et al., 2019) Automation (Control, Monitoring)
Drainage (Lakshmi et al., 2021) Automation (Control, Optimization)
Weight distribution (Shen et al., 2020) Automation (Control, Monitoring)
Hull Hull fouling (Coraddu et al., 2019) Maintenance (Monitoring)
Hull robustness (Maciej Serda et al., 2019) Maintenance (Monitoring)
Water-tightness (Pan et al., 2022) Maintenance (Monitoring)
Corrosion control (Beko et al., 2021) Maintenance (Monitoring)
Structural inspection & maintenance (Poggi et al., 2020) Maintenance (Monitoring)
Cargo storage Storage space (Tsang et al., 2017) Automation (Monitoring, Control)
Cargo securing (Li & Wang, 2018) Automation (Control, Optimization)
Cargo management (Tsang et al., 2017) Automation (Monitoring, Control)
Cargo safety (Zhu et al., 2020) Automation (Monitoring)
Loading/Discharging Cargo transfer (Tam, 2020) Automation (Monitoring, Control)
Draft management (Parsons & Kotinis, 2011) Automation (Monitoring, Control)
Cargo management (Alexeev et al., 2020) Automation (Monitoring)
Emergency response Fire system (Ji et al., 2021) Automation (Monitoring, Control)
Flooding (Vassalos, 2020) Automation (Monitoring, Control)
Escape (Choi & Yang, 2017) Automation (Optimization)
Search and rescue (Ai et al., 2021) Automation (Monitoring)
Security Cyber security (Tusher et al., 2022) Automation (Monitoring)
Physical security (Kawakami, 2016) Automation (Monitoring)
Habitability Accommodation space (Mihai et al., 2021) Automation (Monitoring)
Food and drink (Vaneeckhaute & Fazli, 2020) Automation (Monitoring)
Working space (Simon et al., 2015) Automation (Monitoring)
Sanitation (IMO PPR 9, 2022) Seafarer health (Monitoring)
Human safety (Lin et al., 2021) Seafarer health (Monitoring)

Table 3.

Number of polygons comparison

Model name Original model Weight reduction model
Ship 904 904
Motor 803,529 138,637
Battery Control Unit 79,216 17,897
Cluster 52,573 8,731
Trottle 77,709 15,943
Total Count 1,013,931 182,312