Most integrators start building against our API with polling. It's familiar, it's synchronous in the developer's mental model, and it works well enough for initial testing when you're submitting 2–3 videos and watching the job IDs come back. Then they try to process 500 videos overnight and hit the realization that polling 500 job IDs every 30 seconds for 8–14 minutes each is not a production architecture.
This post is about building the production version: a webhook-driven pipeline where our API calls your endpoint, not the other way around. It covers delivery guarantees, idempotency requirements, handling the cases where webhook delivery fails, and some patterns that work well specifically for video indexing workloads (as opposed to general webhook guidance, which often assumes much faster operations).
What Our Webhook Contract Guarantees
First, the delivery contract, because everything else in this post depends on understanding what we guarantee and what we don't:
At-least-once delivery. We do not guarantee exactly-once delivery. A job completion event can be delivered more than once. This is intentional — it's operationally safer to risk duplicate processing than to risk dropped events. Your endpoint must handle duplicate deliveries idempotently.
Retry schedule: Three attempts with exponential backoff on any response other than 2xx — 30 seconds, 2 minutes, 10 minutes. After three failures, the delivery is marked failed. We do not retry further; the event is logged to the job's event history, and you can reconcile via polling if needed.
Ordering: We do not guarantee event ordering across concurrent jobs. If you submit 50 videos simultaneously, the completions may arrive in any order. Design your receiving endpoint to handle any-order arrival.
Payload stability: Webhook payloads include a schema_version field. We will not change the shape of an existing schema version; we add new schema versions for breaking changes and include both versions during a transition period.
Building an Idempotent Receiver
The idempotency requirement is the piece that trips up most initial implementations. Here's a minimal implementation pattern:
Your webhook receiver should do three things on every inbound request, in order: (1) validate the request signature using the shared secret, (2) check if the event_id has been processed before (against a de-duplication store), (3) if it hasn't, process the event and mark it as processed.
def handle_webhook(request):
# Validate signature
sig = request.headers.get('X-Meshora-Signature')
if not verify_signature(request.body, sig, WEBHOOK_SECRET):
return 401
payload = json.loads(request.body)
event_id = payload['event_id']
# Idempotency check
if dedup_store.exists(event_id):
return 200 # Already processed, acknowledge
# Process
process_job_completion(payload)
# Mark as processed with TTL (e.g., 48 hours)
dedup_store.set(event_id, 1, ttl=172800)
return 200The de-duplication store can be Redis, a database table with a unique constraint on event_id, or any key-value store with TTL support. Use a TTL of 24–48 hours — longer than our retry window — so you don't accumulate stale entries indefinitely.
One common mistake: processing the event and then marking it as processed in two separate steps without atomicity guarantees. If your process crashes between those two steps, you'll process the event a second time when the retry arrives. Either use a transaction, or mark as "in-flight" before processing and "complete" after, with a timeout to handle crashes.
Handling Backpressure in High-Volume Ingestion
For batch ingestion workloads — submitting several hundred videos in a window — your webhook endpoint will receive completions in bursts as batches of videos finish processing around the same time. If your downstream processing (updating your MAM, writing to a database, triggering downstream steps) is slower than the arrival rate, you'll accumulate delivery backlogs and potentially hit our retry window for deliveries that your endpoint accepted but hasn't fully processed yet.
The pattern we recommend: your webhook receiver does minimal work synchronously (validate signature, de-duplicate, enqueue) and returns 200 immediately. A separate worker pool processes the queue. This decouples your HTTP response time (which affects our retry logic) from your downstream processing throughput.
def handle_webhook(request):
# ... validate + dedup as above ...
# Enqueue for async processing
job_queue.enqueue(payload)
dedup_store.set(event_id, 1, ttl=172800)
return 200 # Return fast; processing happens in workerFor reference: in a batch submission of 300 videos submitted simultaneously, typical completion timing clusters into two waves — roughly 40% complete in the 8–10 minute window, the rest in 12–18 minutes depending on video length and complexity. Your receiver should expect burst arrival, not a smooth linear rate.
Reconciliation: When Webhooks Don't Fire
Webhooks will miss deliveries. Your endpoint will be down for maintenance. A network partition will eat a delivery. Our retry window will close on a delivery while your server is returning 500s. This is not a hypothetical; in any production system running for weeks, missed deliveries happen.
The correct approach is not to build a perfectly reliable webhook receiver (you can't fully control the network), but to build a reconciliation process that runs independently and catches anything that slipped through.
A simple reconciliation pattern: every N minutes (we suggest 15–30 minutes for near-realtime workloads, hourly for batch workloads), query GET /jobs?status=completed&created_after=<last_reconcile_timestamp> for any jobs completed since your last reconciliation run. Compare against your de-duplication store. Process any completed jobs you haven't seen. This acts as a safety net, not a primary path.
We expose GET /jobs/:id/events specifically to support reconciliation debugging. If you're seeing discrepancies between what you processed and what's in our system, the event history for each job shows every delivery attempt, timestamp, response code, and response body. This is the first place to check when diagnosing missed deliveries.
Signature Verification: Don't Skip It
Every webhook request we send includes an X-Meshora-Signature header: an HMAC-SHA256 of the raw request body, signed with the shared secret associated with your webhook endpoint. Verifying this signature is not optional — without it, your webhook endpoint will accept any POST from any source, which creates an obvious injection risk.
We're not saying signature verification is a panacea for webhook security — it doesn't prevent replay attacks on its own, for example. For environments with stricter requirements, pair signature verification with a timestamp check: the payload includes a delivered_at field in ISO 8601 format; reject requests where the delivery timestamp is more than 5 minutes old. That eliminates the replay attack surface for most practical scenarios.
Testing Your Pipeline Before Batch Submission
Before submitting a large batch, validate your pipeline with a small controlled set. Submit 5 videos with known content (ideally content you can inspect and verify against), let them complete, and verify that your de-duplication logic, downstream processing, and reconciliation path all work as expected. A pipeline that processes 5 test videos cleanly will handle 500 with the same logic.
We also expose a POST /webhooks/:id/test endpoint that sends a synthetic delivery event to your endpoint without requiring an actual job to complete. Useful for validating signature verification and de-duplication logic in a test environment without burning processing quota.
Video indexing jobs are long enough that debugging a broken pipeline mid-batch is frustrating. A 30-minute validation pass before a large batch submission saves hours of debugging after the fact.
