NeurIPS 2023: Congrats to Hao Dong
doctoral candidate at the Chair of Structural Mechanics and Monitoring at ETH Zurich and member of the IMOS Lab - EPFL, who participates in NeurIPS 2023 with his work on multimodal domain generalization.
This work, led by Hao Dong, in collaboration with Prof. Olga Fink (IMOS Lab - EPFL), introduces SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization, which will be presented at Thirty-seventh Conference on Neural Information Processing Systems (#NeurIPS) 2023.
We propose SimMMDG, a simple yet effective multi-modal DG framework. We argue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specific and modality-shared components. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints on modality-specific features to promote diversity. In addition, we introduce a cross-modal translation module to regularize the learned features, which can also be used for missing-modality generalization. We demonstrate that our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon (HAC) dataset introduced in this paper.
Paper: external page https://arxiv.org/abs/2310.19795
Code: external page https://lnkd.in/eV7gSnp2
This work forms a collaboration between Hao Dong, Ismail Nejjar, Han Sun, Prof. Eleni Chatzi (ETHZ), and Prof. Olga Fink (EPFL) and is conducted under the auspices of project SENTINEL, as part of the ETH Mobility initiative.