Cross-modal retrieval lets you search across modalities — find images with text queries, or find text documents using image queries. This is powered by unified embedding spaces.

CLIP and Beyond

CLIP (Contrastive Language-Image Pre-training) was the breakthrough: it trains image and text encoders to produce embeddings in the same vector space. Similar concepts get similar vectors regardless of modality.

Modern Multimodal Embedding Models

Building Cross-Modal Search

from transformers import CLIPProcessor, CLIPModel
import torch

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

# Embed images
image_inputs = processor(images=product_images, return_tensors="pt", padding=True)
image_embeddings = model.get_image_features(**image_inputs)

# Embed text query
text_inputs = processor(text=["red sneakers"], return_tensors="pt", padding=True)
text_embedding = model.get_text_features(**text_inputs)

# Find most similar images
similarities = torch.cosine_similarity(text_embedding, image_embeddings)
top_results = similarities.argsort(descending=True)[:10]

Use Cases

• E-commerce search: "Show me something like this" with an uploaded photo
• Content moderation: Find images similar to known policy violations
• Media libraries: Search video archives with natural language descriptions
• Accessibility: Generate descriptions for images based on similar captioned images
• Recommendation: "More like this" across product categories

Hybrid Text-Image Search

For product search and content discovery, combine:

1. Text-to-text matching (product descriptions, metadata)
2. Text-to-image matching (CLIP embeddings)
3. Image-to-image matching (visual similarity)

Weight and fuse results based on your use case. Typically text matching handles exact attributes while image matching handles aesthetic and visual similarity.