†Corresponding author, 1. Introduction
The International Maritime Organization reviewed the relevant instruments for the introduction of new technologies and started to develop the MASS Code for the operation of autonomous ships.(IMO, 2022) Currently, the government is expressing its will to support the development of new technologies and financial support for the shipbuilding and offshore industry by carrying out the national R&D Projects for the revival of the main industry in accordance with the paradigm shift in the shipbuilding and offshore industry. The Korean New Deal promotes the development and demonstration of core technologies for ship intelligence, and the new maritime and fishery industry innovation strategy is also promoting the preemptive securing of core technologies for intelligent ships, such as intelligent navigation and automatic berthing.
In the field of maritime transport, discussions on the concept of autonomous ships are ongoing. In the IMO, the concept of autonomous ships has been classified into four and used temporarily. However, it is difficult to establish the concept of an autonomous ship because the technology level of the autonomous system that can be applied differs depending on the onboard systems.
As the latest technology is applied to the industry, terms such as IIoT, digital twin, cyber-physical system, and autonomous system are used to inform the application of this technology, but it is difficult to know the exact level of technology and it is difficult to clearly distinguish between the terms because there are many common features. Many papers review different aspects of digital twin, providing many definitions and descriptions of digital twins.(Fuller et al., 2020; Wilhelm et al., 2021)
Existing papers focus on a specific technology, this paper aims to apply digital twin technology to ships and build an architecture that users can use. Chapter 2, discusses the concept of digital twin applicable to ships by reviewing the recently published papers. Chapter 3 describes the technical analysis of the vessel system that can be configured as a digital twin. In Chapter 4, a user-based digital twin system architecture that can apply the applications investigated in Chapters 2-3 was constructed. Afterward, a cloud-based prototype digital twin which configured as ship data collection (part 1), cloud system (part 2), analysis system/application (part 3), and AR/mobile system (part 4) was connected and tested on an 8-meter battery-powered electric propulsion vessel.
2. Concept of the Digital twin ship
2.1 General definition of the Digital twin
As the term of digital twin suggests, a digital twin is a digital model of a real object. Cutting-edge IT technology has developed dramatically since the 4th industrial revolution and many researches and developments are being conducted in various fields such as manufacturing, city, robot, medical care, and logistics. Before the digital twin concept appeared, a Cyber-Physical System (CPS) was used as a similar concept, and in a broad sense, the Industrial Internet of Things (IIoT) system was also used as a concept similar to the digital twin.
Since the term of digital twin is used in a broad sense and can be interpreted differently by stakeholders, a number of papers have been published that analyzed digital twin technology in various industries and presented the concept. Although a number of papers define digital twins in various aspects, it is commonly agreed that it is essential to transfer the data flow between real and digital objects in both directions.(Tao et al., 2019) An integrated digital system with a physical system can be divided into digital model, digital shadow, and digital twin.(Kritzinger et al., 2018)
Due to the characteristics of ships, in most ships except for coastal ships, it is expected that the seafarer will be on board or will be maintained for a considerable period of time in the operation of the ship either directly or indirectly through remote control. Human Machine Interaction based digital twin model including the ship operator should be applied.(Wilhelm et al., 2021) Therefore, it is necessary to identify the way digital twins support ship operators by each of the ship’s constituent systems, and it is necessary to classify digital twins and the tasks performed by seafarers.
2.2 Technical level definition of Digital twin ship
A ship is composed of multiple systems, and seafarers directly control the systems of the ship by manipulating. Autonomous systems are expected to gradually replace ship seafarers in implementing certain tasks, but when and at what level the replacement would happen will vary depending on the nature of the task and role of seafarers. Thus, for large ships where many autonomous systems of varied levels are operated, clearly defined the role of systems and seafarers, and their interaction must be handled with great importance. In this paper, the technical level of system was defined in consideration of the roles of system and the roles of seafarers.(Thomas B. Sheridan et al., 2014; Wilhelm et al., 2021)
The role of system consists of data collection support, decision and implementation. The level of automation increases as we move from data collection to implementation. The stages of decision and implementation can be seen as the roles to be played by autonomous systems. Role of system can be defined as follows:
Information(Data collection): The system collects and stores data on task and relevant status through sensors and provides seafarer with information in a summary
Support: Based on collected data, analysis and prediction is carried out to provide information required by the operator in the decision-making process. Analytics and prediction are based on technology such as big data and AI.
Decision: Measure to be taken by system is determined through optimization based on data analysis and simulation.
Implementation: Autonomous system controls onboard systems based on determined measures.
The level of automation applied to the system and ship operation strategy may change the role of seafarer. Largely, a seafarer’s role is divided into operation, decision, supervision and stand-by, with the role of seafarer on for each task decreasing in the same order. The definition of the role at each stage is as follows:
Operation: Conventional role of seafarer regardless of system automation. Seafarer will carry out overall tasks related to onboard system operation with the support or information from the system.
Decision: The seafarer exercises overall control of the system based on support and decision by the system. An autonomous system provides seafarers with strategy and insight into the decision.
Supervision: The seafarer verifies the system’s decision and implementation rather than making decisions on system operation. A system under operation may be shut down or decision-making authority may be withdrawn if necessary.
Stand-by: The role of a seafarer is replaced by the system.
3. Architecture of the digital twin ship
3.1 The architecture for Digital twin ship
A ship is a very complex system composed of various systems. Many technical documents on the concept of digital twin for detailed systems have been published, but the concept of digital twin for ships has not yet been published. ISO 23247 series are standard documents for the digital twin framework of manufacturing published by the International Organization for Standardization. This paper classified the digital twin architecture into 4 areas, and added an operator area in consideration of the ship operation characteristics. In this chapter, each area of the digital twin was identified through the analysis of technical documents and the interaction between the digital twin and the operator was reviewed.
3.2 Information collection area on physical objects
In the information collection area, the ship components synchronize data to applications, cloud systems, stakeholders, and etc, while storing information through various sensors of physical objects. In this area, it is necessary to identify which components in the ship are the physical objects of the digital twin ship.
The technical level applied to each system may differ depending on ship composition and the nature of seafarer task. Thus, to define the level of autonomous technology by system, ship compositions are divided as specified in rows 1 and 2 of Table 2, considering ship structure and function, and seafarer task. At the category level, onboard systems are divided by physical location and function into navigation, berthing (including anchoring and mooring), steering, propulsion, electric power system, hull, cargo storage, loading/discharging, emergency response, security and habitability. Key functions and seafarer tasks a reconsidered for each item to derive the low-level category.
3.3 Analysis area for digital tiwn
The analysis area is a crucial area that analyzes the data collected by the ship and provides a model or information required for the application area. This area is divided into analytics technology and infrastructures for data analysis. The first is data analysis technology. The term data analytics focuses on collecting, analyzing and storing data to gain greater insight.
Machine learning and deep learning are representative data analysis algorithms, and deep learning is a subclass of machine learning. Machine learning is characterized by generating algorithms that allow computers to act through data learning without developers directly programming. In conjunction with the system that automatically collects data, the system takes over the repetitive tasks previously performed by seafarers. It is generally used for systems related to maintenance.(Coraddu et al., 2014; Uyanık et al., 2020) Deep learning learns more data and provides higher accuracy to machine learning that can scale up the size of neural networks. It has shown innovative performance improvement in fields that have previously shown limitations, such as image recognition and natural language processing.
The second is the analytics infrastructure. Analytics technology requires continuous learning and analysis on new data. Most systems in maintenance do not require real-time response, and the method of learning and reasoning on board is not efficient in terms of maritime communication environment or cost. Therefore, strategic use of a onshore infrastructure is required depending on the application.
3.4 Application area
The application area is a service provided to users through data analysis. Applications are generally developed and operated based on data collection and analysis systems.
The main applications of the first column of Table1 were classified into navigation, maintenance, energy management system, automation, and seafarer health in consideration of the functions of the system and the task of the seafarer. Navigation, maintenance, and energy management systems are fields where data-based systems, like big data and artificial intelligence systems, are applied. The systems of other ships were classified as an automated system, and systems related to the safety of seafarers were classified as crew health. In the second column, the functions proposed in the technical documents was classified. It was necessary to classify the functions into the existing automation system and the data-based system. General automation functions are classified into monitoring, optimal control and control, and the data based functions were explicitly written in the system name.
3.5 Area of interaction with stakeholders
Although digital twin technology has reached a considerable technical level and has reached the level of unmanned technology in some fields, since ships are operated at sea, an operation strategy through interaction between seafarer and system is required rather than relying on the system. Recently, the International Maritime Organization has been discussing how to consider remote operators as seafarers through the MASS code. As a new entity is introduced into the maritime system, it is necessary to clarify the role of the seafarers and the role of the digital twin.
3.6 Review of the level of technology applied to individual onboard system
Table 2 is a summary of our analysis on common applications by the onboard system and their technical level based on a review of technical materials. For the classification of onboard systems, column 1 and 2 referred to the Autonomous ship Guideline published by Korean Register. In column 3, which is the main application, according to the onboard systems, Table 2 was used to classify them, and in columns 4 and 5, these were classified according to the roles of seafarers and onboard systems presented in Chapter 2.2. In the case the level of autonomous technology was unclear in the technical material, expert opinion was sought.
In the case of navigation and berthing (including anchoring and mooring) systems, the system is capable of deciding on the navigation route, steering and controlling propulsion under controlled conditions. The system for recognizing the external environment collects and analyzes data on the external environment to support the system in planning movement within port, collision avoidance and navigation. As for technology applied to navigation, the role of system includes informing, support, decision making and implementation, and that of seafarer include stand-by, and decision making or supervision depending on the ship operation environment.
As for propulsion and hull, systems are mainly used to support seafarer in terms of maintenance and repair. In terms of maintenance and repair, system provides information and support, while seafarer engages in operation based on system support. The operation has traditionally been and still remains the role of seafarer. Although the system is capable of analyzing the state of a facility, it would not be able to predict a failure in an accurate manner and execute the repair itself in the near future. Therefore, the mentioned role of seafarer will remain.
As for the power system, technology for energy management and maintenance and repair are mainly applied and used. For energy management, system carries out all four roles while seafarer serves as a supervisor that participates in decision-making process when necessary. For maintenance and repair, system provides information need in making decisions, and seafarer engages in operation. Given the various advantages in maintenance and repair, battery-based electric propulsion systems have been applied to MASS for short-distance coastal operations in many cases.
Currently, cargo storage, loading/unloading and housing system are categorized as automation system, and engage in repetitive tasks based on embedded system. For automation, system plays the role of information collection and implementation of tasks. When making decisions, system relies on pre-determined values by seafarer rather than data analysis. The role of seafarer in such case is to supervise the decision-making process if necessary. For maintenance and repair, system monitors the condition and provides analysis results to the seafarer as support. Seafarer carries out tasks based on support from the system.
In the case of emergency response and security, monitoring technology utilizing sensors such as cameras and data analytics are being used for fire detection, search of distress and detection of infiltration by system. To ensure safe operation of MASS, there are currently efforts to develop technology that enables the system to respond actively after detection. However, it is mainly the role of seafarer to engage in operation for incident follow-up. In terms of escape, the system identifies risk areas on board, and supports seafarer and passengers to escape through the safest route.
4. Development of prototype application
In Chapters 2 and 3, we performed a technical analysis on which vessel system can be configured as a digital twin and what information can be provided to the seafarer. To apply the vessel’s application function investigated in Chapter 3 to the digital twin system, it is necessary to set up a development environment that connects the physical vessel and the digital twin system. Therefore in Chapter 4, a four-part digital twin architecture was constructed and tested. In Part 1 as a physical object, an 8-meter-sized battery-based electric propulsion ship (Figure. 3) was prepared for this study. In Part 2, the Microsoft Azure cloud system (Figure 2) was implemented, and in Part 3, the vessel’s monitoring system was applied. In Part4, HoloLens was used to configure and test MR-based real-time monitoring systems shown in Figure 5 and 6.
4.1 Architecture
Based on the digital twin architecture analyzed in Chapter 3, a digital twin architecture suitable for electric powered ships was constructed. The digital twin architecture of electric propulsion ship is divided into 4 parts as shown in Figure 1. As a physical model of digital twin, Part 1 is equipped with ship IoT system that can collect data inside and outside the ship and marine communication facilities for the linkage with land. Part 2 is a digital model on the cloud that implements digital twin model by receiving data on ship. Part 3 is a MR system that can visually check information on ship and information on cloud. Part 4 is an application that can provide the optimal solution through digital model. ML model, real-time monitoring, battery monitoring, recognition of the surrounding environment and maintenance are included.
4.2 Digital model (Part 1)
It is necessary to secure 3D CAD files on the ship and equipment to visualize 3D model of the physical vessel. For equipment, CAD file that is used in the design stage was used, and direct modeling was made for the hull model. After that, it is necessary to convert model files and do optimization work of model to upload the files to application and operate it in the user device. In this study, Unity program was used to develop an application and Pixyz program was used for optimization. Table3 shows the results of implementing Tessellation and Decimate after performing Remove hole, Hidden Removal and Merge are by using Pixyz are as follows.
Model completed with optimization and uploaded to the application is shown in Figure 4. HoloLens limits the number of polygons in one scene by less than 100,000 for the smooth operation of the application.(J. Lee et al., 2017) However, the number of polygon in the entire model that performed optimization was 182,312 which was not satisfied with the limits. In particular, the number of polygon of motor was 138,637 which accounted for the largest portion. Therefore, additional modeling work is necessary for the weight reduction of the motor model in the future.
4.3 Network (PART 1 - PART 2)
The electric propulsion ship and onshore center communication system consists of one network using Virtual Private Network. In addition, it is composed in parallel with Wifi and LTE in consideration of the unstable communication conditions of the state of sea. It is necessary to connect to Azure Cloud through network on the ship to upload real-time data. Therefore test was performed by connecting prototype server with the network on the ship.
To test the network connection, a packet internet groper was sent to each server to check the response and response speed. Test results confirmed that the respond speed of the data was about 24ms on average. Then, WebSocket was used to test the connection between the server and the application and the server and the cloud, and the results confirmed that it successfully received real-time data through the local IP and port number.
The minimum data transmission cycle of the data transmission system is 1 ms, but it was confirmed that the bottleneck phenomenon occurs if the reception cycle gets shorter. This issue will be analyzed in the future by using the performance profile of the program.
4.4 Cloud system (PART 2)
Azure cloud system was used to visualize the data of digital twin system and implement the function of analysis and prediction. The internal cloud system was divided into five sectors which received data from the ship to be transmitted to users, as shown in Figure 2. The major functions of each sector are as follows.
4.4.1 Ingest and Device Management
IoT Hub collects and manages real-time data of the ship received from the server and the number of connected devices and the number of messages used each day can be checked. Collected data is transmitted to Digital twin sector through message routing.
4.4.2 Digital twins
As a digital replica of an actual object, it draws improved results through control and simulation analysis. It collects, processes and analyzes the data and integrates them with AI service to analyze the patterns.
4.4.3 Serving storage
As a data storage system, it can save the message and file data transmitted to the ship and result values transmitted to the digital twin system. It was designed in SQL in consideration of data structure extracted from the ship and Cosmos DB was used for a function that requires quick response of the database.
4.4.4 Operational rules and insights (PART3)
It belongs to AI service of the digital twin system which can manage model learning and the distribution by using Azure Machine Learning. It is scheduled to be applied to the battery failure diagnosis system and power quality system. It is planned to conduct additional linkage tests in the future.
4.4.5 Solution network
It is responsible for the communication function to transmit data from Azure Digital Twin and Azure Machine Learning system to user application. SignalR communication is a real-time web communication technology using WebScoket method which automatically processes connection between clients and services. It has the advantage of fast data transmission speed.
4.5 Implementation (PART4)
HoloLens is a Microsoft-developed mixed reality-based device that provides a blend of physical and digital worlds. In this study, we developed a prototype digital twin application of cloud-based electric propulsion vessel using HoloLens as the HMI device. On the left of Figure 4, a physical object of an electric propulsion vessel is shown, and a digital twin model is placed on the right. Figures 4 and 5 show the operation of the application using HoloLens, in which a dashboard operating real-time data of the vessel is implemented. As a result, it was confirmed that users can see real-time information of the ship through the application by receiving real-time data to the cloud using the communication network on the ship. Users can operate the application by using Mixed Reality based HoloLens and mobile devices as well as 360-degree view and current state of the ship by moving the position. In addition, they can interface with the application through screen touch and hand tracking.
5. Conclusion and future plans
In this paper we summarized the concept of HMI-based digital twin that can consider interaction between digital twin and crews or operators before developing digital twin system of the ship. And then we investigated the roles and application technologies of each system after classifying the system that is made up of the ship to analyze the ship system. The ship system was subdivided into Navigation & Communication, Berthing, Steering Propulsion, Power System, Hull, Cargo storage, Loading/Discharging, Emergency response, Security and Habitability depending on physical locations and functions, and the roles of operators were divided into operation, decision, supervision and stand-by. The roles of digital twin were divided into information, support, decision and implementation. Application corresponding to each system was classified into navigation, maintenance, energy management, automation and seafarer health to specifically present the digital twin of the ship.
For this research, 8-meter-sized battery-based electric propulsion vessel was prepared, and cloud-based prototype digital twin system was implemented by using the azure cloud system. With the digital twin architecture constructed in this paper, an interworking test was conducted for the maintenance (monitoring) function among the user-based digital twin application and the actual vessel. Digital model is imported to the development program through weight lightening of CAD files and conversion of the file format. And then it was made to transmit and send data to the cloud by using ship communication system to receive real-time data. Users can use HoloLens device as HMI device and see real-time sensor data of the vessel through the developed application.
It is planned to apply and test several function of digital twin algorithms to the configured cloud-based digital twin system.
PDF Links
PubReader
ePub Link
Full text via DOI
Download Citation
CrossRef TDM
Print
