Cross-modal transfer is the process by which a model applies knowledge acquired from training on one data modality, such as images, text, or audio, to tasks in a different modality, leveraging shared or aligned representations rather than learning each modality's features from scratch. The alignment is typically achieved through joint training on paired data using contrastive learning objectives, or through a shared embedding space that maps semantically equivalent content from different modalities to nearby coordinates. Once aligned, a model trained primarily on text can generalize to image-based tasks using its linguistic knowledge, or a model trained on images can bootstrap audio understanding, with little or no modality-specific supervision.

Think of it like a musician who learned music theory through classical piano and can then apply that same knowledge to play jazz guitar. The underlying harmonic principles transfer even though the instrument and genre are different.

CLIP demonstrates cross-modal transfer in zero-shot image classification: trained on image-text pairs, the model at inference time compares a text description of an unseen category directly against image embeddings to classify images that were never part of the training set. Another example is using a text-pretrained encoder to bootstrap a speech recognition model, where linguistic structure learned from text reduces the labeled speech data needed for the new task.

Cross-modal transfer connects closely to Contrastive Learning, the most common training mechanism enabling it, Multimodal models as the primary beneficiaries, Embedding spaces where modalities are aligned, Zero-Shot capabilities which cross-modal transfer directly enables, and Foundation Model pretraining strategies that deliberately span multiple modalities.

You will encounter cross-modal transfer in research on vision-language models, audio-visual understanding, medical imaging combined with clinical notes, and any application that aims to reduce the labeled-data burden in a target modality by leveraging knowledge from a data-rich modality.