Cross-Modal Search: Aligning Images and Text with Dual Encoders

Typing "a red bicycle leaning against a brick wall" and getting matching photos back, or uploading a photo and getting the captions that describe it, is cross-modal retrieval: the query is in one modality and the results are in another. Keyword search fails when images have no captions, and a classifier fails when the query isn't one of its fixed labels. This post explains how dual encoders solve the problem, from architecture and contrastive loss to Recall@K evaluation and serving with FAISS and FastAPI.

Why a shared embedding space solves cross-modal retrieval

The idea is to train two functions, one for images and one for text, that map their inputs into the same d-dimensional vector space. Training pulls a photo and its caption together and pushes unrelated pairs apart.

Search in either direction is then the same operation: embed the query and find its nearest neighbors among the other modality's embeddings, which are computed once, offline.

This approach was popularized by CLIP (Radford et al., 2021, "Learning Transferable Visual Models From Natural Language Supervision"), which trained an image encoder and a text encoder on 400 million image-text pairs to predict which caption goes with which image.

The dual-encoder architecture

A dual encoder has two independent towers that only meet in the shared space.

Image encoder: a CNN backbone

A pretrained ResNet is a solid starting point. Replace its ImageNet classification layer with an identity and you get one pooled feature vector per image: 2048 dimensions for ResNet-50.

Text encoder

The text side is typically a pretrained transformer, with its token outputs pooled into one vector by averaging or by taking a special token's output.

Projection heads and L2 normalization

Each backbone gets a projection head that maps its output into the shared d-dimensional space. CLIP used a single linear projection per encoder and reports no difference in training efficiency compared with a non-linear head.

Then L2-normalize both outputs. On unit vectors the dot product equals cosine similarity, so the loss and the search index agree on what "close" means.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.models import resnet50, ResNet50_Weights

class DualEncoder(nn.Module):
    def __init__(self, text_backbone: nn.Module, text_dim: int, embed_dim: int = 512):
        super().__init__()
        cnn = resnet50(weights=ResNet50_Weights.DEFAULT)
        cnn.fc = nn.Identity()                  # pooled features: [B, 2048]
        self.image_backbone = cnn
        self.text_backbone = text_backbone      # must return pooled [B, text_dim]
        self.image_proj = nn.Linear(2048, embed_dim)
        self.text_proj = nn.Linear(text_dim, embed_dim)
        # learnable temperature in log space, initialized to log(1/0.07) like CLIP
        self.logit_scale = nn.Parameter(torch.tensor(math.log(1 / 0.07)))

    def encode_image(self, pixels):             # pixels: [B, 3, 224, 224]
        return F.normalize(self.image_proj(self.image_backbone(pixels)), dim=-1)

    def encode_text(self, tokens):
        return F.normalize(self.text_proj(self.text_backbone(tokens)), dim=-1)

Dual encoders vs cross-encoders

A cross-encoder feeds the image and text into one model together, lets them attend to each other, and outputs a relevance score. Because it can compare specific words with specific image regions, it is typically more accurate on fine-grained distinctions.

The catch is cost: it needs a forward pass for every (query, candidate) pair, so scoring one query against a million images means a million forward passes.

A common pattern is to retrieve the top 50 to 100 candidates with the dual encoder and re-rank only those with a cross-encoder. I'd start with the dual encoder alone and add re-ranking once you can measure the gain.

Training with a symmetric contrastive loss

Take a batch of N matching (image, caption) pairs and compute the N x N matrix of cosine similarities. The diagonal holds the N true pairs; the other N^2 - N entries are mismatches.

The CLIP-style loss, a symmetric form of InfoNCE, treats each row as a classification problem: for image i, which of the N captions is correct? The answer is always i. It repeats this down the columns for captions and averages the two losses.

The temperature controls how sharp the softmax is. CLIP learns it as a log-parameterized scalar, initialized to the equivalent of 0.07 and clipped so logits are never scaled by more than 100, which the paper says was necessary to prevent training instability.

def contrastive_loss(image_emb, text_emb, logit_scale):
    # image_emb, text_emb: [N, d], L2-normalized; row i of each is a matching pair
    logits = logit_scale.exp() * image_emb @ text_emb.t()      # [N, N]
    targets = torch.arange(logits.size(0), device=logits.device)
    loss_i2t = F.cross_entropy(logits, targets)                 # image -> caption
    loss_t2i = F.cross_entropy(logits.t(), targets)             # caption -> image
    return (loss_i2t + loss_t2i) / 2

for pixels, tokens in loader:
    loss = contrastive_loss(model.encode_image(pixels),
                            model.encode_text(tokens), model.logit_scale)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        model.logit_scale.clamp_(0, math.log(100))              # scale stays <= 100

F.cross_entropy expects raw logits of shape [N, C] and integer targets of shape [N], which is exactly what the similarity matrix and arange provide. Clamping the parameter after each optimizer step is how OpenCLIP implements the paper's clipping.

Batch size, in-batch negatives and hard negatives

Every other caption in the batch is a free negative: with a batch of 256, each image is contrasted against 255 wrong captions. More negatives make the task harder and the gradient more informative, which is why CLIP trained with a batch size of 32,768. Two consequences:

• Gradient accumulation does not add negatives. Each micro-batch's softmax only sees its own captions, so four micro-batches of 64 are not one batch of 256.
• Multi-GPU training needs a gather step. Gather embeddings from all devices before building the similarity matrix, or each GPU only sees local negatives.

Hard negatives

Random negatives are mostly easy, like a dog photo against a caption about a spreadsheet, and teach little once the model has learned the basics. Hard negatives are wrong but plausible: "a red bicycle" against "a red motorbike".

To find them, use the current model to retrieve each example's nearest wrong matches and include them in later batches. Watch for false negatives: a "wrong" caption that actually fits the image, or a near-duplicate image, gets pushed away when it shouldn't. Deduplicate first.

Measuring quality with Recall@K

Recall@K is the fraction of queries whose correct match appears in the top K results. Report it in both directions at K = 1, 5 and 10.

@torch.no_grad()
def recall_at_k(query_emb, gallery_emb, ks=(1, 5, 10)):
    # query i's correct match is gallery item i; both [N, d], L2-normalized
    sims = query_emb @ gallery_emb.t()                          # [N, N]
    top = sims.topk(max(ks), dim=1).indices                     # [N, max_k]
    targets = torch.arange(sims.size(0), device=sims.device).unsqueeze(1)
    hits = top == targets                                       # [N, max_k]
    return {k: hits[:, :k].any(dim=1).float().mean().item() for k in ks}

# text -> image: recall_at_k(text_emb, image_emb)
# image -> text: recall_at_k(image_emb, text_emb)

This assumes one caption per image; with several, count a hit if any valid target lands in the top K. Compute recall over the full held-out gallery, not per batch: finding the right item among 256 is far easier than finding it in your real catalog.

Indexing embeddings with FAISS

FAISS handles the nearest-neighbor search. Its primary metrics are L2 and inner product, and its documentation recommends getting cosine similarity by normalizing vectors and using an inner-product index.

Exact search with IndexFlatIP

IndexFlatIP compares the query against every stored vector, so on normalized vectors it returns exact cosine-similarity results.

import faiss
import numpy as np

d = 512
vecs = np.ascontiguousarray(image_embeddings, dtype="float32")  # [num_images, d]
faiss.normalize_L2(vecs)                                        # in place

index = faiss.IndexIDMap(faiss.IndexFlatIP(d))
index.add_with_ids(vecs, image_ids.astype("int64"))

query = np.ascontiguousarray(text_embedding.reshape(1, -1), dtype="float32")
faiss.normalize_L2(query)
scores, ids = index.search(query, 10)                           # each [1, 10]

FAISS expects float32, and normalize_L2 works in place on a contiguous array. IndexIDMap adds add_with_ids support, which a flat index lacks, so results come back as your database IDs.

Approximate search at scale: IVF, HNSW and PQ

Flat search cost grows linearly with the catalog. Approximate indexes give up some recall for lower latency or memory. Figures are from the FAISS index reference:

• IVF clusters vectors into nlist cells and searches only the nprobe closest. nprobe defaults to 1; raising it improves recall but costs speed, and nprobe = nlist matches brute force. The default metric is L2, so pass faiss.METRIC_INNER_PRODUCT.
• HNSW builds a navigable graph that FAISS's guidelines call very fast and accurate when RAM isn't a constraint. Tune efSearch at query time; it can't remove vectors.
• PQ compresses vectors into short codes: 64 bytes with M = 64 and nbits = 8, versus 2,048 bytes for a raw 512-dimensional float32 vector, at some cost in accuracy.

nlist = 1024
ivf = faiss.IndexIVFFlat(faiss.IndexFlatIP(d), d, nlist, faiss.METRIC_INNER_PRODUCT)
ivf.train(training_sample)       # representative, normalized float32 vectors
ivf.add_with_ids(vecs, image_ids.astype("int64"))
ivf.nprobe = 16

hnsw = faiss.IndexHNSWFlat(d, 32, faiss.METRIC_INNER_PRODUCT)   # M = 32
hnsw.hnsw.efSearch = 64

For collections under a million vectors, FAISS's guidelines suggest nlist between 4*sqrt(N) and 16*sqrt(N), trained on 30 to 256 times nlist vectors. My recommendation: start with IndexFlatIP, and before shipping an approximate index, measure its recall against the flat index on real queries.

Serving search with FastAPI

Load everything once at startup; each request then encodes, normalizes and searches.

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI()
# loaded once at startup: model (in eval mode), tokenize, index, metadata

class SearchRequest(BaseModel):
    query: str
    k: int = Field(10, ge=1, le=100)

@app.post("/search")
def search(req: SearchRequest):
    with torch.inference_mode():
        vec = model.encode_text(tokenize([req.query])).cpu().numpy()
    vec = np.ascontiguousarray(vec, dtype="float32")
    faiss.normalize_L2(vec)
    scores, ids = index.search(vec, req.k)
    return {"results": [
        {"id": int(i), "score": float(s), **metadata[int(i)]}
        for s, i in zip(scores[0], ids[0]) if i != -1
    ]}

The endpoint is a plain def on purpose: FastAPI runs those in a threadpool, so the CPU-bound encoder call doesn't block the event loop. The i != -1 filter matters for IVF and HNSW, which return -1 when they find fewer than k results. Keep metadata in a regular database keyed by ID; my PostgreSQL indexing guide covers fast lookups.

Practical pitfalls

• Forgetting to normalize. If either side skips L2 normalization, inner product rewards vector length, not just direction. Normalizing twice is harmless, so do it in the model and before indexing.
• Train/serve preprocessing mismatch. Resize, crop, mean/std normalization, tokenizer version and maximum sequence length must match training. For torchvision's ResNet-50, ResNet50_Weights.DEFAULT.transforms() (resize to 232, center crop to 224, ImageNet mean and std) is one source of truth.
• Forgetting model.eval(). In training mode, batch norm uses the current batch's statistics, so embeddings depend on what else is in the batch.
• Mixing embedding versions. Vectors from two checkpoints live in different spaces. Tag every vector with a model version, build a new index per version, and switch the query encoder and index together.
• Duplicate images. Near-duplicates crowd the top results and create false negatives in training. Deduplicate with perceptual hashes or an embedding-similarity threshold.

Key takeaways

• Cross-modal retrieval maps images and text into one normalized embedding space and searches it with nearest-neighbor lookup.
• Dual encoders are fast because the catalog is embedded offline; add a cross-encoder re-ranker only when measurements justify it.
• The symmetric contrastive loss is a few lines of PyTorch, and batch size decides how many negatives it sees.
• Evaluate with Recall@K in both directions on a full held-out gallery.
• Start with IndexFlatIP, move to IVF, HNSW or PQ when scale demands it, and version embeddings with the model.