When archive teams first evaluate video search technology, the conversation usually starts with one question: "Can you transcribe our content?" It's a reasonable starting point — audio transcription via ASR is the most visible part of what we do, and it's genuinely useful. But the question conflates two distinct capabilities that have very different strengths and very different failure modes.
Transcription and semantic search aren't competing approaches. They operate at different levels of the search stack, and understanding how they differ — and when you need each — helps explain why some video search deployments feel incomplete even after ASR has been running for months.
What ASR Transcription Actually Gives You
Automatic speech recognition takes an audio stream and produces a text transcript. Modern ASR models — Whisper-class architectures, large language model-backed systems — achieve word error rates below 5% on clean studio audio. That's good enough for broadcast content recorded with professional microphones in controlled conditions.
With a full-text transcript, you can do keyword search. If someone said "infrastructure bill" on air, you can find every segment where that phrase was spoken. You can also do basic temporal navigation: the transcript carries timestamps at the word or sentence level, so a search hit maps directly to a timecode offset in the source file. From a standing start — an archive with no searchable text at all — this is a major improvement.
What you get is essentially a searchable document for the audio track. Think of it as turning a video file into something behaviorally similar to a PDF of the spoken content. Full-text search, fuzzy matching, BM25-style relevance ranking — all the standard text search operations apply.
The constraints are straightforward. WER degrades on noisy audio, field recording, non-studio conditions. In a dataset we processed last year covering approximately 3,800 hours of local news content from a mix of field and studio sources, average WER across the corpus was around 11% — with clear studio segments under 4% and field-recorded segments climbing past 18%. Errors cluster heavily on proper nouns: names of places, people, organizations. Those are the exact terms archive users search for most frequently.
Diarization — speaker attribution — adds another dimension. A diarized transcript tells you not just what was said but who said it. For content where speaker identity matters (news interviews, panel discussions, depositions), this is essential. But diarization accuracy is sensitive to overlapping speech and audio bleed-through between microphones, conditions that are common in multi-person setups.
What Semantic Search Does Differently
Semantic search operates on embeddings rather than text strings. An embedding is a vector representation of a piece of content — a segment of audio, a block of text, a visual frame — in a high-dimensional space where semantic similarity corresponds to geometric proximity. When you query "protests at city hall," a semantic search doesn't look for those exact words. It finds content whose embedding vector is close to the embedding of your query, which captures meaning rather than surface form.
The practical difference is query vocabulary independence. A keyword search for "protests at city hall" returns nothing if the speaker said "demonstrators gathered outside the municipal building." A semantic search retrieves both, because the embeddings encode the conceptual relationship between the two phrasings. For archive users who often don't know the exact language used in a clip they're looking for, this is significant.
Semantic search also handles concept queries that have no direct textual expression. If you want clips of "tense exchanges between public officials," there's no single phrase that reliably signals that. A keyword search produces nothing useful. A semantic index built over both transcript text and visual content can surface moments where the audio and visual signals together indicate elevated tension — raised voices in the audio embedding, certain visual compositions in the frame embedding — even if no one explicitly narrated "this is a tense exchange."
The Retrieval Architecture Question
Here's where it gets architectural. A keyword index (BM25 or similar) and a vector index (approximate nearest neighbor search over embeddings) are different data structures that answer different question types. BM25 excels at exact term retrieval and works well when users know specific names, dates, and phrases. Vector search excels at concept retrieval and handles query vocabulary variation naturally. Neither is strictly better — they complement each other.
For most archive use cases, the right answer is a hybrid retrieval system: run both indices, score results from each, and merge via a re-ranking step that balances lexical precision with semantic relevance. This is sometimes called Reciprocal Rank Fusion (RRF) or, in more sophisticated implementations, a learned re-ranker that uses contextual signals to weight the two retrieval modes based on the query type.
We're not saying pure keyword search is bad — if your archive users primarily search for specific people and specific dates, BM25 against a clean transcript will serve them well at minimal computational cost. We're saying that for archives where search queries are exploratory, conceptual, or expressed in natural language by users who weren't present when the content was created, keyword search alone leaves most of the potential value untapped.
Where Both Approaches Fall Short Without Visual Context
Both ASR-plus-keyword and semantic-over-text approaches share a common limitation: they're operating only on what was said. A significant fraction of video content meaning is visual, and neither approach indexes it without additional pipelines.
Consider news content where a chyron reads "LIVE: Senate Floor Vote" but the audio is ambient crowd noise without meaningful speech. A pure audio-based index produces a near-empty transcript for that segment. A semantic search on that transcript produces noise. But the content is findable — if you're indexing the visual signal, the OCR output from that chyron is "LIVE: Senate Floor Vote," and that's directly searchable.
The same applies to visual scene content. A segment of footage from a factory floor might have minimal spoken audio, but the visual scene embedding encodes that it's an industrial setting, and can surface it in response to queries about manufacturing or production facilities. Without visual embeddings, that segment is invisible to both ASR-keyword and semantic-text approaches.
In the Meshora index, we run ASR for transcript text, OCR for on-screen text, and visual classification for scene and object content. All three streams feed into a unified embedding space so that a single query can retrieve across all three simultaneously. The query "flooding in downtown area" can match on spoken news coverage (ASR), on-screen location labels (OCR), and visual scenes of flooded streets (visual embeddings) — all in one retrieval pass.
Choosing a Starting Point for Your Archive
If you're evaluating where to start and your archive has limited budget or is running a proof of concept, the ordering matters:
Start with ASR if your primary need is exact-term retrieval and your users know what they're looking for — specific names, dates, regulatory references, quoted language. A clean transcript index with good diarization solves the core problem for archives where queries are specific and vocabulary is shared between the original content creators and the searchers.
Add semantic indexing when users search conceptually, when query vocabulary is unpredictable, or when you're building a discovery experience where users don't know exactly what they want — they want the system to surface relevant content from a conceptual description. This is common in creative archive workflows and in legal discovery where the legal team wasn't present when the relevant events were recorded.
Add visual indexing last only if your archive contains significant content where visual information carries meaning that audio doesn't — documentary footage, surveillance material, B-roll libraries, content with heavy on-screen graphics. For predominantly talking-head news content, visual indexing provides incremental value. For a film archive, it's foundational.
The choice isn't permanent. Archives that start with ASR and add semantic and visual layers over time see retrieval relevance improve meaningfully at each step — not because the earlier approach was wrong, but because each layer captures signal the others miss.
