Smart Maritime Surveillance: Leveraging YOLO Detection and Blockchain traceability for Vessel Monitoring

Before the rise of object recognition and automatic detection systems, surveillance systems relied heavily on human supervision. Even today, security systems have not completely eliminated the need for human attention. The goal of Computer Vision is to achieve human-level capabilities in understanding the world. To achieve this goal through AI, a significant amount of research has focused on a connectionist approach. Deep Neural Networks, especially Convolutional Neural Networks [1–2], have shown exceptional performance in object detection, which is a fundamental task in surveillance systems. While certain advanced technologies like Radar and Lidar are exclusive to the military, using Deep Learning for object recognition is relatively straightforward due to its open-source nature.

Numerous implementations are available for various algorithms, along with necessary resources, which expedite the creation of neural network-based object recognition systems. Neural networks, by their nature, are often considered black boxes that are largely task-invariant and perform well on diverse tasks. Surveillance is crucial not only for security but also in other areas such as hospitals for monitoring patients, in sports for tracking players, and in agriculture to detect anomalies in fields, among other examples. The ideal surveillance system minimizes human intervention and triggers an alarm upon detecting an intrusion.

Recent advancements in tracking technologies have furthered the development of more stable and less prone to drift tracking algorithms. Sliti et al. (2019) developed a novel visual tracking technique that utilizes sparse representation combined with a back-projection histogram. This approach effectively mitigates tracking drift by leveraging spatial information. By back-projecting the sparse coefficients of the target template in each frame, the method enhances the robustness of target localization [3]. Additionally, Reference [4] explored the incorporation of texture information using Local Binary Pattern variants in the mean shift framework, resulting in adaptive scale and orientation trackers capable of handling fast-moving objects and target scale and orientation changes, showcasing enhanced stability and accuracy in tracking over purely color or feature-based methods. These advancements underscore the potential of integrating sophisticated object-tracking methodologies into surveillance systems, promising significant improvements in automatic detection and monitoring capabilities.

These systems can employ various techniques, such as signal and image processing, computer vision, and machine learning. Having the right infrastructure, such as hardware computing power (primarily GPUs), can significantly increase system throughput and reduce the risk of failure, making it a worthwhile investment. The ability of deep learning algorithms and neural networks to leverage GPUs for faster computations is a major advantage in reducing system latency. Deep learning models eliminate the need for manual feature extraction because they are designed to learn features directly from large labeled datasets. Through the use of neural network architectures, these models automatically identify and extract relevant features from the data during the training process. The availability of these resources has led to a new method of creating object detectors known as transfer learning, which enables the development of detectors without the need for extensive data, making these techniques applicable to a wide range of applications. [5]

Ship detection (Figure 1) is crucial for ensuring efficient ship navigation, reducing water traffic accidents, improving waterway passage efficacy, and aiding water public security agencies in preventing and detecting criminal activity. Visible imaging offers significant advantages over other imaging techniques for ship detection, as it captures detailed images of the target while minimizing the effects of weather conditions through computer vision techniques.

Figure 1.

Vessel detection

Moreover, the importance of blockchain technology [6–9] cannot be overstated. Blockchain provides a decentralized and secure method for recording transactions and track assets, making it ideal for applications like ship detection and tracking. By using blockchain, data related to ship detection, such as location information and identification details, can be securely stored and accessed by relevant parties in real-time.

This not only enhances the transparency and efficiency of ship tracking systems but also helps in ensuring the integrity and authenticity of the data. Moreover, blockchain can also be used to create immutable records of ship-related activities, which can be valuable for regulatory compliance and dispute resolution. Overall, integrating blockchain technology into ship detection and tracking systems can significantly improve their reliability and effectiveness.

To address the challenges in vessel tracking, the integration of (IoT), artificial intelligence (AI), and decentralized technologies is rising as a highly effective solution. This convergence creates a smart vessel tracking system that facilitates:

• -

  Real-time Monitoring: Continuous oversight of vessel activities enhances operational awareness.

• -

  Efficient Resource Management: Optimizes the use of marine resources, improving overall efficiency.

• -

  Transparent Traceability: Ensures clear tracking of vessel movements, promoting accountability and security.

• -

  By leveraging these advanced technologies, stakeholders can significantly enhance the safety and efficiency of maritime operations.

The implementation of AI techniques alongside IoT devices in the maritime sector presents several challenges, particularly regarding privacy and security. As concerns about data privacy have increased, businesses’ financial models have become more susceptible to potential attacks. While real-time monitoring using IoT data is critical for maritime operations, it also introduces new vulnerabilities.

Manufacturers of IoT devices and end-users encounter significant obstacles when trying to apply AI solutions in real-world settings, especially since data often needs to be collected and processed centrally. To address these issues, Ghadi et al. (2023) explore the integration of Federated Learning with IoT as a decentralized, collaborative AI framework applicable across various IoT scenarios. This method enables AI training on distributed IoT devices without requiring the sharing of sensitive data, thereby mitigating privacy concerns and enhancing security for smart city applications, including maritime surveillance [10].

The convergence of IoT and AI technologies, with the added security and transparency of blockchain, presents a comprehensive framework for developing smart vessel tracking systems. These systems not only enable real-time monitoring but also ensure the data’s integrity and privacy, crucial for the sensitive nature of maritime operations. By leveraging IoT for data collection and Federated Learning for decentralized AI processing, smart maritime surveillance systems can achieve a new level of efficiency and security, ultimately contributing to safer and more sustainable maritime practices.

The utility of blockchain for securing unmanned aerial vehicle (UAV) operations within smart cities, has been demonstrated highlighting the technology’s potential for real-time data gathering and secure data storage in maritime applications [11]. Reference [12] discusses cybersecurity solutions for smart grids using blockchain, suggesting methods applicable for protecting maritime surveillance systems against cyberattacks. Some researchers propose a blockchain and machine learning-enabled architecture for healthcare, underscoring blockchain’s capability for managing and securing critical data, a principle transferable to maritime industry operations [13].

The incorporation of blockchain into maritime surveillance offers unparalleled advantages in terms of data security, transparency, and operational efficiency. By drawing from examples in UAV management, cybersecurity [12], and healthcare data management, the maritime industry can adopt similar blockchain-based solutions to address its unique challenges. This ensures not only the security and integrity of maritime data but also paves the way for automated, smart contracts and efficient, decentralized systems that can revolutionize marine resource management and surveillance.

The article presents a study on AI and advanced detection techniques for ship monitoring. The focus is on preventing ship intrusion and enhancing ship detection in satellite and river images. It highlights the use of YOLOv3 and YOLOv8 neural networks for improved detection accuracy. IoT technology plays a role in real-time tracking. The integration of feature fusion modules aids in efficient information processing. A vital concern addressed is the reduction of pollution from ships. The study contributes to marine ecosystem preservation and maritime safety enhancement. Results show significant improvements in detection accuracy. The integration of blockchain technology ensures transparent traceability and data security. This combination offers a comprehensive solution for efficient vessel monitoring and management. The structure includes a review of tracking technology, blockchain specifics, the proposed approach, results, and future perspectives.

The article is organized into several key sections to provide a thorough exploration of smart maritime surveillance leveraging You Only Look Once (YOLO) detection and blockchain traceability. Following this introduction, Section 2 presents the theoretical foundations underpinning object detection technologies, including a detailed examination of both traditional and deep learning-based algorithms.

Section 3 reviews related work, identifying current research efforts and pinpointing gaps that our study aims to fill. In Section 4, we outline our proposed approach for vessel detection using YOLO and integrating blockchain for traceability, detailing each step from data acquisition to the application of smart contracts. Section 5 presents the results of our study, highlighting the effectiveness of the YOLO models and the blockchain tools employed. The discussion in Section 6 evaluates the implications of our findings, and the article concludes with Section 7, summarizing the study’s contributions to the field of maritime surveillance and suggesting directions for future research.

Object detection [14–16] algorithms fall into two main categories: traditional machine learning algorithms and deep learning-based algorithms. Traditional approaches rely on manually designed features based on prior knowledge or assumptions, leading to lower robustness against uncertainties.

In contrast, deep learning-based algorithms utilize Convolutional Neural Networks (CNNs) to learn image features, resulting in improved detection speed and accuracy. Deep learning target detection algorithms can be further divided into two categories: multistage algorithms and single-stage algorithms. Multistage algorithms, such as the R-CNN series, use pre-generated candidate regions in the image to extract features effectively and filter out background noise, achieving higher detection accuracy. On the other hand, single-stage algorithms like YOLO (Figure 2) and SSD extract features directly from the entire image to predict the object’s location and class, resulting in faster inference times [17–18].

Figure 2.

YOLO algorithm

The use of deep learning for ship detection relies heavily on high-quality datasets, with several SAR ship detection datasets currently available [19]. The SAR Ship Detection Dataset (SSDD) was introduced by Li et al. [20], featuring ships in diverse environments with varying image resolutions, ship sizes, sea conditions, and sensor types. Wang et al. [21] introduced the SAR-Ship-Dataset, derived from multimodal SAR images, allowing a comprehensive deep learning model to be developed for the detection and classification of merchant ships in complex backgrounds. Near-real-time detection and classification are enabled by this dataset without the need for sea-land segmentation. The AIR-SARShip dataset was presented by Sun et al. [22], offering high-resolution, large-scale SAR images for ship detection in both near-shore and open-sea settings. Su et al. [23] introduced the HRSID dataset, which includes high-resolution SAR images suitable for both ship detection and instance segmentation. Most of these datasets provide information on ship location, though ship class details are generally not included.

Nevertheless, impressive results have been achieved in SAR-based ship detection using deep learning techniques with these datasets. An enhanced Faster R-CNN model was proposed by Li et al. [20] to yield promising outcomes on SSDD datasets. The SRSDD dataset, introduced by Lei et al. [24], includes vessel category and angle information, supporting rotating frame target detection, though an imbalance in categories affects detection accuracy. Feature extraction capabilities were further improved by Hu et al. [25], who integrated the SENet channel attention mechanism into Faster R-CNN. Zhang et al. [26] proposed an improved YOLOv3 model, where DarkNet53 was replaced with DarkNet19, achieving faster detection speeds on the SSDD datasets.

Blockchain is a technology for storing and transmitting information transparently, securely, and without a central control body. It is used to secure digital transactions and exchanges by recording data in a decentralized and transparent manner. Each block of data is cryptographically linked to the previous block, forming a chain of blocks, hence the name “blockchain”. This technology is used in cryptocurrencies like Bitcoin, but also in many other areas to ensure the traceability and security of data [27].

In a blockchain, a block is a collection of data that represents a set of transactions. Each block contains a header and a list of transactions. Here are some key components of a block and related terms: (Figure 3)

• Nonce: A nonce is a number added to a block that, when hashed along with the block’s other data, produces a hash value that meets the difficulty target set by the network. Nonce is used in the process of mining to find a valid block hash.

• Merkle Root: The Merkle root is a hash derived from all the individual transaction hashes contained within a block. It serves as a compact summary of the transactions and is incorporated into the block header.

• SHA-256: SHA-256 (Secure Hash Algorithm 256-bit) is a cryptographic hash function used in Bitcoin and many other cryptocurrencies. It is used to hash the block header and produce a unique identifier (hash) for each block.

• Block Header: The block header contains several fields, including the previous block’s hash, the Merkle root of the transactions, a timestamp, a nonce, and the current difficulty target. The header is hashed to produce the block’s unique identifier (hash).

• Block Hash: Each block has a unique identifier known as the block hash, which is generated by applying a cryptographic hash function, like SHA-256, to its header. This hash not only identifies the block but is also incorporated into the header of the next block, thereby linking the blocks together in a sequential chain, forming what is commonly referred to as the blockchain.

Figure 3.

Blockchain components

In Ethereum, a smart contract is defined as a self-executing agreement in which the terms between the buyer and seller are encoded directly into lines of code. The code and the associated agreements are maintained across a distributed and decentralized blockchain network. Execution of the contract occurs automatically when specified conditions are fulfilled, thereby removing the need for intermediaries and enhancing both traceability and security in transactions.

Smart contracts are typically written in programming languages such as Solidity and Vyper, allowing developers to establish the rules and actions that will be carried out upon meeting certain criteria. For example, ownership of a digital asset can be transferred automatically once payment is verified. The decentralized nature of the blockchain ensures that all transactions are securely recorded, providing a transparent log of the contract’s execution. The key points about smart contracts in Ethereum are:

• Code Execution: Smart contracts are executed on the Ethereum Virtual Machine (EVM), which is a decentralized runtime environment.

• Decentralization: Smart contracts run on a blockchain, which means they are decentralized and distributed across many nodes, making them censorship-resistant and tamper-proof.

• Autonomy: Once deployed, smart contracts run automatically without any human intervention, ensuring that the code is executed exactly as written.

• Trust: Smart contracts eliminate the need for trust between parties, as the contract terms are enforced by code and cannot be altered after deployment.

• Immutable: Once deployed on the Ethereum blockchain, a smart contract cannot be modified, ensuring the integrity of the contract.

• Ethereum as a Platform: Ethereum provides a platform for developers to create and deploy smart contracts using its programming language, Solidity.

• Gas: Transactions involving smart contracts require gas, which is a unit that measures the amount of computational effort required to execute operations on the Ethereum network.

• Use Cases: Smart contracts can be used for a variety of applications, including decentralized finance (DeFi), supply chain management, voting systems [7], and more.

Blockchain is revolutionizing vessel monitoring and control by offering transparent and immutable traceability of maritime activities. The decentralized and secure nature of blockchain allows data related to vessels, such as their position, movements, and activities, to be recorded in a transparent and tamper-proof manner. This enables maritime authorities, surveillance companies, and stakeholders to easily verify a vessel’s history and ensure its compliance with regulations and agreements.

Furthermore, blockchain facilitates the establishment of consensus among maritime stakeholders, ensuring that the recorded information is accurate and legitimate. By combining traceability and consensus, blockchain plays a crucial role in promoting sustainable maritime practices, enabling increased monitoring of activities and more efficient management of marine resources.

Currently, apart from route navigation, various information systems onboard vessels provide extensive insights into the conditions of nearby ships and harbors (Figure 4).

Figure 4.

An example of secure maritime communication based on Blockchain.

The identification of vessel locations through these systems allows operators to anticipate conditions, predict arrival times, and manage routes more effectively. Intelligent Transportation Systems (ITS) hold considerable potential to boost both the safety and efficiency of maritime transport. One such technology, the Automatic Identification System (AIS), performs essential functions like vessel identification, object tracking, and information exchange. AIS also facilitates data sharing to prevent collisions. Continuously and proactively, AIS transmits automatic vessel information to nearby ships or harbors, responds to specific vessel inquiries, and manages safety information. However, AIS has a transmission range limited to around 20 nautical miles and lacks strong encryption, leaving it vulnerable to data manipulation by malicious parties. For example, attackers could alter a vessel’s route data, potentially leading to collisions. Since AIS is an unregulated device, it is widely accessible, meaning malicious actors could acquire the equipment, falsify AIS data, or send fake distress signals, potentially causing confusion or delaying rescue efforts.

Pirates, for instance, might exploit Distributed Denial of Service (DDoS) attacks [28] to disrupt vessel signals, which could make ship data disappear from navigational maps and thereby facilitate hijacking. Although AIS allows vessels to download weather information from nearby marine sources, securing the network connection between AIS and marine units is vital. Insecure connections could enable attackers to manipulate weather reports, resulting in unnecessary detours or a rush of vessels entering ports to avoid perceived hazards. Therefore, implementing robust network security measures for AIS is critical to ensure data integrity, authentication, and prevent repudiation [29].

The management of vessel AIS, as discussed in [30], involves route prediction and sharing navigational information with captains to optimize marine traffic flow. Liu et al. [31] proposed a system that combines data from the Gaofen-4 satellite and AIS to track vessels. Using aerial images, the satellite helps accurately pinpoint vessel positions, supporting precise location confirmation. In addition, [32] emphasizes the importance of sharing accurate and complete information, proposing a system named RobustTrust to evaluate the authenticity of shared information. This system filters out false data, thus enhancing the reliability of information.

For machine learning tasks, supervised learning is highly effective but requires well-labeled datasets. When high-quality datasets are limited, semi-supervised learning can serve as a useful alternative by working with a mix of labeled and mostly unlabeled data. Xue et al. [33] presented a rapid, weak semi-supervised method for ISAR, integrating a regional proposal network with deep sparse learning. Another approach, self-supervised learning, introduces a supervised aspect within an unsupervised framework. For instance, Ciocarlan et al. [34] applied self-supervised learning to large, annotated Sentinel-2 image datasets, followed by transfer learning on smaller samples for ship detection. While promising, this approach often requires large batch sizes and extensive training time. Additionally, unsupervised learning models can produce biased predictions unless trained with supplemental datasets.

In [35], blockchain technology, specifically Ethereum, is proposed as a solution to enhance data security in maritime surveillance. Buoys and drones are used to gather data, which is then transmitted via a mesh network to a centralized data fusion center onshore. Performance evaluations conducted with MATLAB simulations focused on assessing latency and throughput. The Food and Agriculture Organization of the United Nations publication [36] examines various ways to integrate blockchain into the seafood supply chain, specifically addressing traceability challenges within the seafood lifecycle. The study suggests using NFC, RFID, and QR codes to improve tracking and verification. Further exploration of blockchain’s potential in global ocean conservation and fisheries management is presented in [37]. Blockchain’s applications across various sectors include fostering resource transparency, reducing plastic pollution, combating forced labor at sea, and supporting sustainable fishing practices. The study points to increasing public interest in conservation authenticity and the traceability of seafood sources.

This paper highlights the role of blockchain in addressing key challenges in maritime operations, especially through transparent traceability across the supply chain from catch to market. IoT sensors and tracking technology enable secure, real-time data access regarding product origin and quality, preserving data immutability. Additionally, blockchain facilitates smart contracts, which can simplify agreements among stakeholders, such as operators, processors, distributors, and regulators. These contracts can automate transactions like payments and licensing, reducing bureaucratic delays and enhancing operational efficiency.

Our work utilizes Ethereum blockchain technology [38], with development and deployment supported by tools like Truffle and Ganache. Smart contracts are coded in Solidity, Ethereum’s specialized programming language, offering diverse use cases within Ethereum’s open-source ecosystem. Digital signatures are employed to guarantee the immutability of smart contract source code, thereby enhancing the security and reliability of transactions. Every message processed by the platform is regarded as a transaction, necessitating gas fees to be paid in Ether. (Figure 6)

Figure 6.

Blockchain tools.

Solidity is a specialized programming language developed to write smart contracts for the Ethereum. Smart contracts are automated contracts in which the terms and conditions are encoded directly, allowing them to self-execute without the need for intermediaries. Solidity is optimized for the Ethereum Virtual Machine (EVM), which is responsible for running these smart contracts. (Figure 7)

Figure 7.

Solidity example

Metamask is a popular browser extension and cryptocurrency wallet that allows users to securely store and manage Ethereum and ERC-20 tokens. It also serves as a bridge to access decentralized applications (dApps) on the Ethereum network. (Figure 8)

Figure 8.

Metamask tool

Truffle provides an integrated environment for Ethereum development, encompassing tools for testing and managing assets. It is designed to streamline the development process for Ethereum developers, making their workflow more efficient and user-friendly. It provides tools for compiling, deploying, and managing smart contracts, as well as for testing smart contracts against various scenarios. Truffle also includes a development console for interacting with your contracts and the Ethereum network.

The key performance metrics used to evaluate YOLOv5, YOLOv6, YOLOv7, and YOLOv8, based on a comprehensive metric [1], include accuracy, precision, recall, mAP@0.5, FPS, and F1-score. Precision is calculated by taking the ratio of true positive predictions to the total number of positive predictions made by the model. This metric indicates the correctness of positive predictions, measuring the share of predicted positive instances that are actually relevant (Equation 1). Recall, which is also known as sensitivity or the true positive rate, is derived by dividing the number of true positive predictions by the total number of actual positive instances in the dataset. This metric gauges the model’s capability to identify and capture all relevant instances (see Equation 2). Both precision and recall can take values between 0 and 1, inclusive. Moving on to the mAP@0.5 metric, it calculates a score by comparing the identified box with the ground-truth bounding box at an IoU threshold of 0.5, as described in Equation 4. The model’s detections achieve higher precision as this score increases. The F1-score combines the model’s precision and recall, calculated as the harmonic mean of both metrics, as illustrated in Equation 5. 1presision=TPTP+FPpresision{\rm{ }} = {{TP} \over {TP + FP}} 2recall=TPTP+FNrecall{\rm{ }} = {{TP} \over {TP + FN}} 3FPS=1TFPS = {1 \over T} 4mAP=1classes∑1cTP(c)TP(c)+FP(c)mAP = {1 \over {classes}}\sum\limits_1^c {{{TP(c)} \over {TP(c) + FP(c)}}} 5F1score=2*precision*recallprecision+recallF1score{\rm{ }} = {{{2^*}precision*recall} \over {precision + recall}}

In video surveillance detection, a true positive (TP) corresponds to correctly detected objects (considered safe and clean) by the algorithm. Conversely, a false positive (FP) represents non-valid objects incorrectly identified by the algorithm, often due to inaccurate bounding box predictions. The false negative (FN) indicates existing objects that the algorithm fails to detect.

In our scenario, we utilize the ABOships Dataset [38], which is designed for maritime vessel detection and classification in spaceborne optical images. The dataset comprises 9,880 images featuring 12,770 ships of various sizes.

Table 1 and Figure 9 illustrate the performance results of YOLOv3 (61 million parameters, 20 FPS) and YOLOv8 (3.2 million parameters, 217 FPS).

Figure 9.

Results

The “parameters” here refer to the number of model parameters, which are the weights learned during the training process. In this context, YOLOv8 has a significantly lower number of parameters (3.2) compared to YOLOv3 (61). A lower number of parameters can be advantageous as it may lead to a more computationally efficient and faster model.

Thus, YOLOv8 appears to have a more efficient model with fewer parameters and superior performance in terms of mAP compared to YOLOv3. (Figure 10)

Figure 10.

Modeling performance of mAP landscape across FPS Space

The FPS measures how many frames (images) the model can process per second. In this case, YOLOv8 demonstrates a significantly higher FPS (217) compared to YOLOv3 (20). A higher FPS is generally desirable, indicating that YOLOv8 is capable of processing images more quickly, making it more suitable for real-time applications. (Figure 11)

Figure 11.

Modeling performance of mAP landscape across Parameter space

Ganache is a well-known tool utilized for developing and simulating a local Ethereum blockchain. It features a user-friendly graphical interface that enables real-time visibility of transactions, events, account addresses, and balances, thus aiding in the tracking and debugging of smart contract interactions. (Figure 12)

Figure 12.

Deployment of standard smart contracts.

Some contracts have been deployed on Ganache to form the smart platform for real-time vessels detection called Vessel-detect.

VesselContract:

• Contract Address:

  0xC19525AA0288046878EDff6fFD818549C169EE34

• Description: This contract allows controller to capture and submit real-time vessel data such as GPS location, Vessel-Id and emergency alerts.

This contract will be stored in a block. (Figure 13)

Figure 13.

Block added in the blockchain