A Hybrid Speech–Text Foundation Model for Noise-Robust Multilingual Automatic Speech Recognition

Abstract

Automatic speech recognition (ASR) in multilingual settings remains vulnerable to nonstationary noise, code-switching, accented pronunciation, limited labeled data, and ambiguity among acoustically similar words. This paper proposes a hybrid speech–text foundation model integrating a pretrained multilingual acoustic encoder, a noise-aware feature adaptation module, a multilingual text encoder, and confidence-gated cross-modal decoding. The system combines acoustic likelihood with contextual linguistic evidence while controlling semantic overcorrection through uncertainty-aware fusion. A reproducible evaluation protocol is specified for clean and noise-corrupted multilingual corpora, including low-resource languages and mixed-language utterances. Word error rate (WER), character error rate (CER), language identification accuracy, real-time factor, and calibration are proposed as evaluation measures. Illustrative numerical scenarios are provided solely to demonstrate the planned comparative and ablation analyses; they are not measured experimental outcomes. The proposed architecture offers a testable research direction for resilient multilingual transcription under practical deployment constraints.