The common description of what Meshora does — "index video content for search" — is accurate at the product level but skips over the technical problem that makes it non-trivial. The challenge isn't building three separate indices (transcript, OCR, visual). The challenge is making those three indices queryable as a single unified retrieval system, where a query can match on any or all modalities simultaneously without requiring the user to know which signal they're searching against.
This post is a technical walkthrough of how we approach that fusion problem. The architecture involves some choices that aren't obvious and some trade-offs we can explain clearly, but it's genuinely the engineering layer that makes the system work — or fail in specific ways that matter.
Why Three Separate Indices Don't Compose Easily
The naive approach to multi-signal video search is: build a text index for ASR transcripts, build a text index for OCR output, build a vector index for visual frame embeddings, and at query time, run the query against all three and merge the results. This approach is buildable and it partially works, but it has a structural problem: the three indices represent incomparable scores.
A BM25 score from a transcript search and a cosine similarity score from a visual embedding search are not on the same scale and don't have the same distribution. A BM25 score of 12.4 and a cosine similarity of 0.87 don't have a principled ratio that lets you say "this result is 30% from transcript relevance and 70% from visual relevance." You can apply ad-hoc normalization and weighting, and this produces something that works in demos, but it produces inconsistent relevance behavior across different query types and content types. Transcript-heavy queries end up over-weighted in some configurations; visual-heavy queries in others.
The deeper problem is that the three indices don't understand each other. A keyword match on the transcript doesn't tell the visual index that the matched segment is probably also visually relevant. A high-similarity visual match doesn't inform the transcript index. The signals are retrieved independently and stitched together, rather than capturing the multi-signal nature of the moment being described.
Unified Embedding Space: The Core Idea
The approach we use is to embed all three signal types — transcript text, OCR text, visual frame content — into a shared vector space where semantic proximity means the same thing regardless of source modality. A transcript segment about "industrial manufacturing process" and a visual frame of a factory floor and an OCR text reading "PRODUCTION LINE A" should all cluster together in this space, even though they come from different signal types and different processing pipelines.
This is sometimes called a joint embedding space or a cross-modal embedding space. The canonical example in computer vision is CLIP (Contrastive Language-Image Pre-training), which aligns image and text representations in a shared space so that an image of a dog and the text "a golden retriever playing fetch" are geometrically close. We work with architectures in that family, with domain-specific modifications for broadcast and archive content.
The critical training signal for aligning modalities is co-occurrence. In broadcast video, a spoken phrase and the visual scene it describes co-occur at the same timecode. A chyron identifying an interview subject co-occurs with the visual appearance of that person. These temporal co-occurrences are supervision signals: the model learns that representations are "similar" if the corresponding content tends to appear at the same moment. Given enough examples across a diverse corpus, the learned geometry generalizes — new content that wasn't in training but that follows the same co-occurrence patterns gets placed in similar relative positions.
The Three Encoding Paths
Each signal type goes through a distinct encoding path before entering the unified space.
Audio/transcript path: ASR output is segmented into semantically coherent chunks — typically sentence or clause boundaries rather than fixed-duration windows. This matters because a fixed 30-second window may split a complete thought across two chunks, reducing relevance for queries that match a complete statement. Each segment is encoded by a text encoder into a 768-dimensional vector. We use a domain-adapted encoder fine-tuned on broadcast transcript content, which handles media-domain vocabulary (terminology from news, legal, and corporate contexts) better than a general-purpose text encoder.
OCR path: OCR output goes through a deduplication step (adjacent frames with the same text are collapsed into a single time-windowed entry), then through the same text encoder as transcript content. This is an intentional design choice: by using the same encoder for both transcript and OCR text, we ensure that semantically similar content from both sources ends up in the same region of the embedding space. A spoken phrase and a displayed graphic conveying the same information will have similar embeddings and will both surface in response to the same query.
Visual path: For each detected shot (we run a shot boundary detector to segment footage into coherent visual units), we sample frames at multiple intervals and encode each through a vision encoder. The shot-level embedding is an aggregation of frame-level encodings — typically a weighted mean that down-weights frames with significant motion blur or low information content. Shot-level embeddings are then projected into the unified space via a learned projection layer that aligns visual representations with the text encoder's geometry.
The projection layer for visual representations is where the cross-modal alignment happens. It's a learned linear (or low-rank non-linear) transformation that maps from the vision encoder's native embedding space into the shared text-visual space. This layer is trained on paired video segments where we have both visual content and associated text (transcripts, OCR), using a contrastive objective that pulls matching pairs together and pushes non-matching pairs apart.
Query-Time Retrieval
At query time, the user's query text is encoded using the same text encoder as the transcript and OCR paths. The resulting query vector is then compared against all entries in the unified embedding index — regardless of whether those entries originated from transcript, OCR, or visual sources. Approximate nearest neighbor search (we use HNSW indexing for sub-millisecond retrieval across hundreds of millions of vectors) returns the top-K entries by cosine similarity to the query vector.
The result set from this unified retrieval mixes entries from all three modalities. A result entry carries its source modality as metadata, so the UI can display whether a result matched on speech, on-screen text, or visual content — useful for the end user to understand why a clip was retrieved. But the ranking itself is agnostic to source: a visual result with cosine similarity 0.92 ranks above a transcript result with cosine similarity 0.86, regardless of which modality produced it.
We layer a hybrid retrieval step on top of this for queries that contain specific names, dates, or exact phrases. For these, BM25 over the indexed text content runs in parallel with the vector retrieval, and results are merged via Reciprocal Rank Fusion. The hybrid step recovers precision for exact-match queries that sometimes have lower vector similarity than approximate semantic matches — a side effect of embedding models that may not perfectly preserve exact-phrase salience.
What This Doesn't Handle Well
The unified embedding approach performs best when the query semantics align with the content distribution the model was trained on. For highly domain-specific terminology — medical Latin in clinical video, specialized legal terminology in deposition review, technical acronyms in engineering training content — the text encoder may not have strong representations, and retrieval quality degrades. We're not saying the approach fails completely for specialized domains, but recall in terminology-heavy queries may be lower than in general news content, and domain adaptation through fine-tuning on relevant vocabulary becomes more important.
Cross-lingual content is handled by the multilingual variant of our text encoder, but the visual-text alignment in the shared space is weaker for non-English content because the pretraining data is less balanced across languages. For archives that are primarily non-English, this is a meaningful limitation — query accuracy in the non-English modalities will be lower than in English, sometimes significantly.
The projection layer that aligns visual and text representations needs to be calibrated for each content domain. The projection we trained on broadcast news generalizes reasonably to corporate video, but visual-text alignment degrades somewhat on documentary and archival footage with different visual characteristics. For archives with significant stylistic departure from news-format content, re-calibration of the projection improves retrieval quality.
There's also a fundamental limitation on visual concepts that don't have natural text correspondences. Abstract visual events — a specific type of camera movement, a particular lighting condition, a visual rhythm in an editing sequence — don't have clean text anchors in the training data and tend to produce weak embeddings in the shared space. For archives where this type of visual query matters (film archives, documentary production companies), pure visual similarity search operating in the vision encoder's native space may perform better than cross-modal retrieval for those specific query types.
We've built the three-stream fusion because it covers the majority of what archive search users actually need. For the edge cases where it doesn't, knowing why it fails is at least as useful as knowing that it works.
