Independent vector analysis with flexible statistical modeling for multimodal data fusion: from blind source separation to misinformation detection

Abstract

The widespread dissemination of digital information through various communication channels has significantly altered the way societies generate, consume, and interpret knowledge. However, this shift has also accentuated the spread of misinformation. The intricate nature of this phenomenon lies in its inherently multimodal aspect, in which altered images, misleading texts, and synthetic media intertwine, forming false narratives that defy both human and machine comprehension. This thesis investigates the urgent issue of automated identification of multimodal misinformation based on a theoretical framework grounded in Independent Vector Analysis (IVA) and statistical signal analysis. First, we review the theoretical foundation of IVA, understood as an extension of Independent Component Analysis (ICA), which expands its ability to model multiple data sets by investigating the interdependencies between modalities. Based on this foundation, we propose a multivariate density estimation methodology based on the Maximum Entropy Principle (MEP), which combines global and local constraints through the Multivariate Entropy Maximization with Kernels (M-EMK) estimator. This estimator offers adaptive and expressive probability density functions, ensuring computational efficiency through Quasi-Monte Carlo integration, resulting in the development of the IVA-M-EMK algorithm. Besides source separation, this thesis applies the proposed approach to identifying multimodal misinformation, in which various data types, including text, images, and semantic embeddings, are analyzed together to highlight underlying consistencies and contradictions. Recognizing that misleading signals often exhibit sparse and organized characteristics, IVA-SPICE is employed, which integrates structured sparsity through inverse covariance estimation, enabling the efficient processing of high-dimensional multimodal spaces. Experimental investigations reveal that the proposed framework outperforms traditional unimodal and simplified fusion methodologies across multiple datasets, achieving greater accuracy, generalizability, and explainability. This thesis establishes Independent Vector Analysis, combined with adaptive density modeling and structured sparsity, as a solid, scalable, and interpretable foundation for multimodal data integration.