Mid Sweden University

In recent years, there has been a significant increase in multi-sensor data across various fields, spanning from environmental monitoring and industrial automation to smart agriculture, surveillance systems, healthcare analytics, robotics, remote sensing, smart cities, and beyond. The fundamental drive behind leveraging multimodal data is the amalgamation of complementary information extracted from various sensors, facilitating more comprehensive insights and informed decision-making compared to reliance on a single modality.

The analysis of multi-sensor data presents substantial challenges due to its vastness and the presence of structured, semi-structured, and unstructured data, spanning different modalities with distinct sources, types, and distributions. Data fusion, the integration of information from diverse modalities, becomes crucial in addressing inference problems arising from multi-sensor data. Both analytics-based and learning-based data fusion approaches are widely used, with learning-based approaches, leveraging machine learning and deep learning methods, showing notable effectiveness.

However, the question of "where" and "how" to fuse different modalities remains an open challenge. To explore this, the study focused on three applications as case studies to employ data fusion approaches for estimating and predicting physical quantities. These applications include analysing the correlation between the change in the geometrical dimension of a free-falling molten glass gob and its viscosity using Pearson correlation coefficient as analytical method, predicting fuel consumption of city buses through machine learning methods, and classifying and measuring hazardous gases i.e. hydrogen sulfide (H₂S) and methyl mercaptan (CH₃SH) using deep learning methods.

Results from these case studies indicate that the choice between traditional, machine learning, or deep learning-based data fusion depends on the specific application, as well as the size and quality of the data. Despite this, advancements in computing power and deep learning technology havemade data more accessible and have enhanced its complementarity. Therefore, a comprehensive review to compare a range of deep learning-based data fusion strategies is conducted. The review provides an examination of various feature extraction methods, as well as an outline and identification of the research fields that stand to derive the greatest benefits from these evolving approaches.