A Framework for the Collection, Storage and Visualisation of Big Maritime Data

Summary

The maritime industry generates vast amounts of daily data and many times publicly available are incomplete; effectively managing and deriving insights from this data remains a significant challenge. The aim of this thesis is to address this challenge by developing a comprehensive data management system designed to aggregate, manage, and visualize ship trajectories and related maritime statistics from various data sources by utilising state-of-the-art tools and techniques to unify incomplete data and create a single source of reference. To achieve this, the developed system employs advanced web scraping techniques for data collection as well as machine learning techniques to enrich the acquired data with missing elements (e.g., vessel’s position). The retrieved data are stored in a NoSQL data store, building upon its scalability and flexibility in handling/querying large datasets. After the data are collected, cleaned and homogenised, the developed system performs various analytics and visualization tasks, providing valuable insights into maritime shipping routes and other relevant statistics. In summary, this thesis presents a framework for managing large maritime data that allows users to simplify data collection, analysis, and visualization, and enables users without an IT background to leverage the collected data into insights.