Privacy-Preserving Multimodal Sentiment Analysis

[IoTJ] H. Xu, W. Li, D. Takebi, D. Seo, and Z. Cai, Privacy-Preserving Multimodal Sentiment Analysis [J]. IEEE Internal of Things Journal (IoTJ), 2025. Download paper here

Multimodal sentiment analysis plays a critical role in numerous IoT-driven applications, such as personalized smart assistants, healthcare monitoring systems, and intelligent transportation networks, where accurate interpretation of user emotions is vital for enhancing service quality. However, a severe threat of privacy leakage in the multimodal sentiment analysis has been overlooked by previous works. To fill this gap, we propose a Differentially Private Correlated Representation Learning (DPCRL) model to achieve privacy-preserving multimodal sentiment analysis by combining a correlated representation learning scheme with a differential privacy protection scheme. Our correlated representation learning scheme aims to achieve heterogeneous multimodal data transformation to meet the requirements of privacy-preserving multimodal sentiment analysis by learning the correlated and uncorrelated representations, where especially, a pre-determined correlation factor is employed to flexibly adjust the expected correlation among the correlated representations. The differential privacy protection scheme is used to obtain the disturbed correlated and uncorrelated representations by adding Laplace noise for ϵ-differential privacy. In particular, the correlation factor can help alleviate the side-effect of the added Laplace noise on the sentiment prediction performance. Finally, via conducting a series of real-data experiments, we validate that our proposed DPCRL model is superior to the state of the art for privacy-preserving multimodal sentiment analysis.