Research on Cross-Modal Semantic Alignment Methods for Low-Resource Languages

Abstract

This study addresses the challenge of cross-modal semantic alignment in low-resource languages, a critical problem for enabling inclusive and equitable AI-driven multimodal applications. We propose a novel framework that synergistically integrates multi-level textual embeddings, visual Transformer modeling, and the construction of a unified cross-modal projection space. To enhance alignment quality, the approach incorporates advanced mechanisms including contrastive learning, distributed semantic constraints, and fine-grained local alignment strategies. Furthermore, to mitigate data scarcity inherent in low-resource settings, we leverage transfer enhancement techniques such as cross-lingual knowledge distillation, pseudo-pair augmentation, and multi-task training. Comprehensive experiments on the FLORES-200 dataset demonstrate that our method consistently surpasses state-of-the-art models such as CLIP and ALIGN across multiple metrics. Specifically, significant gains are observed in Recall@1 and Mean Rank for languages including Swahili and Sinhala, underscoring the method's effectiveness, robustness, and generalizability in low-resource scenarios. These findings highlight the potential of the proposed approach for advancing cross-lingual multimodal understanding and bridging the performance gap for underrepresented languages.