Keyword search and vector search are both retrieval operations: you have a query, you find matching segments. A knowledge graph is a different data structure — it represents the entities in your archive and the relationships between them. The question it answers isn't "find me segments about X" but rather "show me everything connected to entity X, across every video where X appears, and map how X relates to Y and Z."
We've been building the entity linking layer of Meshora's indexing pipeline over the past year, and we recently crossed 10,000 hours of archived content with full entity graph indexing. This post describes how we extract entities, how we link them across documents, where the approach breaks, and what we're still working on.
Entity Extraction: Three Signal Streams
For any given video, entities can surface from three distinct signal streams: transcript text (spoken language), on-screen text (OCR output), and visual content (frame analysis). Each stream has different recall and precision characteristics, and combining them is not as simple as a union of results.
Transcript NER. Standard named entity recognition (NER) applied to the ASR transcript output. We run a domain-adapted NER model that handles the entity types most relevant to our primary content verticals: PERSON, ORG, LOCATION, DATE, EVENT, PRODUCT. Transcript NER has good recall on content where entities are spoken clearly but degrades on content with high WER — noisy audio, heavy accents, domain-specific terminology that ASR mispronounces.
OCR NER. Named entity recognition applied to OCR text from on-screen graphics, titles, lower thirds, and documents captured in frame. Lower-third text in broadcast content is extremely high precision for person/org extraction — it's explicitly labeled — but is absent in most enterprise video content (internal training recordings, depositions, panel discussions). The recall on OCR NER is therefore content-type dependent.
Visual entity recognition. Frame analysis for visual entity types: logos, products, locations. This is narrower than the text-based NER — visual entity recognition is more reliable for fixed visual patterns (logos, branded products) than for general entities like locations or people. We don't attempt person identification from visual features alone for reasons described below.
After extracting from each stream, we run a cross-stream deduplication and alignment step: if the transcript says "the CEO mentioned Q4 earnings" and the lower third reads "Q4 Earnings Report" at the same timecode range, these surface as a single entity instance with multiple source signals, not two independent extractions. Cross-stream alignment improves precision by disambiguating homonyms and near-matches.
Entity Linking: Building the Cross-Video Graph
Entity extraction gives you a list of entity mentions per video, each timestamped. Entity linking takes you from a list of mentions to a graph: "these 47 mentions across 22 videos all refer to the same entity." This is the technically hard part.
The naive approach — match entity string to entity string — fails immediately. "Jon Smith" and "Jonathan Smith" are the same person; "Apple" can refer to a company, a product line, or a piece of fruit depending on context. The linking step requires entity disambiguation.
Our approach uses embedding-based entity clustering: we embed each entity mention using a context window (the surrounding transcript or OCR text, not just the entity string), cluster mentions with high embedding similarity, and then apply a set of disambiguation rules to merge or split candidate clusters.
For a concrete example: we worked through an archive of approximately 10,000 hours of corporate training and panel discussion content. Initial NER extraction produced about 380,000 entity mentions across the archive. After embedding-based clustering with a similarity threshold of 0.88, these resolved to approximately 24,000 distinct entity clusters — the "unique entities" in the archive. About 15% of those clusters required manual disambiguation review because the clustering algorithm had low confidence (crossing the 0.75–0.88 similarity range where we don't automatically merge or split). That manual review takes time and is currently a human-in-the-loop step for new customer archives.
The Person Identification Problem
We handle person entities differently from other entity types, and we're intentional about the limits we set. We extract and link person entities that are named in transcript or OCR text — "according to [Name]," lower-third credits, speaker attributions in deposition transcripts. We do not attempt to identify persons from visual features (face recognition) in the current product.
This is a deliberate product boundary, not a technical one. Face recognition in an enterprise video archive would require maintaining a reference database of enrolled face embeddings, versioning those embeddings as people age or change appearance, and managing the privacy and consent surface that comes with biometric processing of individuals who appear in archived footage but may not have consented to facial recognition in any downstream system. That's a different product than what we're building right now.
We're not saying face recognition in archived video is inappropriate — it's a legal and contextually valid tool in some settings, particularly broadcast archives where public figures in public contexts are the primary subject. We're saying it's a separate product decision with its own regulatory surface, and we haven't made that decision yet.
Graph Structure and Queryability
The entity graph we build is a property graph: nodes are entities (typed: PERSON, ORG, LOCATION, etc.), edges are relationships, and both nodes and edges carry properties including timestamped evidence (which videos, which timecodes).
Relationship types we extract automatically: CO-OCCURRENCE (entities appear in the same scene segment), MENTIONED-BY (person entity is the speaker of a segment that mentions another entity), VISUAL-CO-OCCURRENCE (entities appear in the same frame or frame sequence). Higher-semantic relationships — "is affiliated with," "spoke at," "reported on" — require relationship extraction from transcript text, which is a harder NLP problem and is currently available only for a subset of relationship types.
In the API, the entity graph is exposed through two endpoints. GET /entities returns the entity list for an indexed collection with pagination. GET /entities/:id/graph returns the neighborhood of a specific entity: its directly connected entities, relationship types, and the timecoded evidence linking them. This is the starting point for "show me everything in the archive connected to entity X" queries.
Where the Graph Goes Wrong
Three categories of graph quality problems we see consistently:
Entity proliferation in content-heavy organizations. Archives with high speaker diversity and many internal-use entities (project names, internal org names, product codenames) generate entity clusters that are technically correct but analytically noisy. A 10,000-hour archive from an organization with 500+ people who are routinely named in training content will have entity clusters for every person mentioned — including people who appear once and never again. Filtering by entity frequency (appearance count across the archive) is a useful client-side operation for managing this.
Cross-archive entity collisions. For organizations that manage multiple separate archives under one account (e.g., a media company with separate archives for different production divisions), entity identifiers are scoped to the archive, not the organization. "John Smith" in Archive A and "John Smith" in Archive B are not automatically linked. Cross-archive entity resolution is on our roadmap but not currently supported.
Temporal entity evolution. An entity that changes context over time — a person who changes roles, a product name that changes, an organization that rebrands — will generate what looks like a clean graph but is semantically inconsistent. The entity node for "ProductX" may link to content from three years ago when the product was a different product than the one it is now. We surface creation dates and evidence timestamps on every entity node; using those to filter graph queries is the current mitigation.
The knowledge graph output is available in both JSON (via API) and as a GraphML export for organizations that want to load it into a dedicated graph database or visualization tool. We don't provide a built-in graph visualization UI — that use case is too workflow-specific for a generic interface to serve well.
