Shot boundary detection is a solved problem. You compute histogram differences between consecutive frames, threshold the delta, and you have a list of cut points accurate to within a frame or two. Open-source tools do this well. We don't spend engineering time on shot detection because it's not the interesting part of scene understanding — it's just the raw material.
What we actually build is a semantic scene model: a representation of continuous meaning across cuts, not just a list of where the edits happened. This post explains what that means concretely, why it matters for search and retrieval, and where our current approach breaks down.
The Difference Between a Shot and a Scene
A shot is a continuous camera take — no cuts. A scene is a semantic unit: a coherent stretch of content with consistent topic, participants, and context, potentially spanning many shots. In a 45-minute documentary, there might be 600–800 shots but only 30–40 scenes in any meaningful sense.
For retrieval, scenes are almost always the right unit. If someone searches for "the segment where the CEO discusses the supply chain disruption," they want a coherent 3–5 minute stretch of content — not a list of 47 shots, some of which happen to contain spoken words about supply chains. Shot-level retrieval forces the caller to reassemble meaning from fragments. Scene-level retrieval returns a navigable chunk.
The challenge is that scene boundaries are not deterministic from visual data alone. Two consecutive shots might share a location, speaker, and topic continuity — same scene — or they might represent a completely different narrative thread edited into the same visual environment. You need audio, language, and temporal context to resolve that ambiguity.
What Meshora's Scene Model Tracks
We track five signals to build scene boundaries, and we fuse them rather than using any one as the primary signal:
1. Shot boundaries — the baseline. Detected by frame histogram analysis; gives us candidate split points.
2. Speaker diarization continuity — whether the speaker set changes across a shot boundary. A cut that maintains the same two speakers in a conversation is unlikely to be a scene boundary. A cut that introduces a new speaker signature (especially one not heard in the prior 60 seconds) is a strong scene boundary candidate. We run speaker diarization via our ASR pipeline and maintain a rolling speaker identity window.
3. Transcript semantic embeddings — we compute sentence embeddings over rolling 15-second transcript windows and measure cosine distance between consecutive windows. A sharp drop in embedding similarity across a shot boundary is strong evidence of a topic change and therefore a scene boundary. We threshold this empirically; the exact threshold varies by content type (news vs. training video vs. deposition recording).
4. Visual scene context embeddings — keyframe embeddings (one per shot, sampled at the temporal midpoint) compared across the shot boundary. Setting changes, context changes (outdoor → indoor, meeting room → presentation stage) register here. This signal is weaker in talking-head content but useful in documentary and event footage.
5. Audio context shift — changes in ambient audio signature: background music in/out, crowd noise, room acoustics. This catches scene changes that are semantically continuous (same topic, same speaker) but physically discontinuous (interview in studio vs. same interview B-roll cutaway). Less reliable than the other four signals but useful as a tiebreaker.
We combine these signals with a learned boundary scorer that was trained on a labeled corpus of approximately 8,000 hours of broadcast and enterprise video content, with human-annotated scene boundaries. The scorer outputs a boundary probability at each shot cut; we threshold at 0.65 for the default segmentation and expose the raw probability scores in the API for teams that want to apply their own threshold.
A Concrete Example: Multi-Camera Interview Content
Consider a scenario that comes up frequently in enterprise learning archives: a 90-minute recorded panel discussion with four speakers and a camera operator switching between a wide shot and individual close-ups every 15–30 seconds. Shot boundaries fire every 15–30 seconds — roughly 200 shots across the session. The semantic scene structure is something closer to 8–12 discussion segments separated by topic transitions.
Shot boundary detection alone gives you 200 retrieval units that are mostly unintelligible in isolation. Our scene model collapses these into the 10 or so meaningful segments, with timestamps marking topic transitions. A search for "the segment about performance evaluation frameworks" returns one segment, timestamped, rather than 20 fragmented shots that collectively cover the topic.
Precision on this content type is around 88% at the scene boundary level, meaning roughly 12% of our detected boundaries are either false positives (splitting a continuous scene) or false negatives (missing a real transition). The false positives are more common — we over-segment in content with frequent speaker-swap edits — and we're working on reducing that through tighter diarization-embedding fusion.
What We Don't Do (and Why)
We don't attempt named entity resolution at the scene level — no "this scene features Person X, Location Y." We track speaker IDs within a video (Speaker 1, Speaker 2) based on diarization, but we don't attempt to resolve those to named identities without external reference data. Some customers integrate our scene output with their own identity databases; we provide the timecoded speaker segments, they handle the resolution.
We're not claiming this is a fundamental limitation — speaker recognition against a known-identity reference corpus is tractable. But it adds a data management layer (maintaining the reference embeddings, versioning them) that we don't want to own on behalf of customers with very different identity management practices. Keeping the boundary at "speaker ID" rather than "speaker name" also avoids a category of false-positive risk: misidentifying someone is worse than not identifying them.
Scene Metadata in the API
Each scene in the API response includes: scene_id, start_ms, end_ms, shot_count (how many shots it spans), speaker_ids (list of diarized speaker identifiers active in the scene), topic_summary (a short generated description of the scene's semantic content), and boundary_confidence (the boundary scorer's probability for the start boundary).
The topic_summary is the field that gets the most attention in search interfaces — it's what surfaces in the card previews. We generate it from the transcript segment covering the scene, not from visual description alone, because transcript text is typically more precise about topic than visual content description in the content types we process most often. For content with significant visual information and minimal dialogue — B-roll, silent training demonstrations — we fall back to a visual description generated from the keyframe embedding.
One caveat worth stating plainly: for very short scenes (under 45 seconds), topic summaries are less reliable. The transcript segment may be too short to establish clear semantic content, and the boundary confidence scores for these short scenes are also typically lower. If you're building a search interface and want to filter out low-confidence segments, use boundary_confidence < 0.5 as a first-pass filter — that removes most of the edge-case fragmentations.
Scene detection is one of the areas we iterate on most actively. The shot-level pipeline has been stable for over a year; the semantic scene model gets a calibration update roughly every six weeks as we accumulate more labeled examples across content types.
