Netflix builds an observability system with an ontology and RDF knowledge graph to support AutoRCA troubleshooting

Netflix uses an ontology to turn observability into a knowledge graph, eating up the time spent "stitching relationships together" during four-hour troubleshooting

Slow troubleshooting is often not due to a lack of MELT data, but because each tool describes "application / service / cluster" inconsistently and relationships exist only in people's heads; Netflix built an Observability Ontology + RDF knowledge graph (QuipuDB), using type constraints to collapse non-deterministic outputs such as LLM and fuzzy matching into deterministic facts that can be queried via SPARQL, then feeds them to AutoRCA, so engineers start out with the context that previously took half an afternoon to stitch together. https://t.co/vRwmsHX07R

Problem scenario · Scale: Jake Paul vs Tyson at roughly 65 million concurrent streams; day-to-day on the order of 2 billion+ requests/day and about 38 million real-time log events/second (figures from the article). · Typical regression: homepage list rendering slows by about 1.5s → alert; suspects can include client version, A/B, infrastructure rate limiting, bad data from a downstream GraphQL 6 hops away, etc. · Investigation timeline of about 4 hours: almost none of it is spent "solving hard problems"; it is spent reconstructing mappings that each system actually knows but never wrote down as discoverable relationships.

Why not "more dashboards" or "just throw an Agent at it" Atlas / Radar / https://t.co/jia3pZ05T4 / PagerDuty / Spinnaker / Slack are each self-consistent but isolated externally; entities with the same name have inconsistent suffixes, casing, and renames, and cross-domain joins rely on people. Giving an Agent the wrong context leads to hallucinations, which is even more dangerous for on-call. The essence is a data modeling problem: there is no machine-readable entity convention.

Ontology and graph · Ontology = types, properties, allowed relationships; the unit is a triple (subject–predicate–object). Example: api-gateway ops:ownedBy "Team Bedrock", and a single query can answer "incidents related to Bedrock applications in us-east-1". · Tech stack: RDF / OWL / SHACL / SPARQL; stored in QuipuDB (an in-house RDF quad store based on Rust redb); each harvest produces a Turtle archive for easy diffing. · About 12 namespaces cover the operations domain; key design decisions: 1. Reject a generic Service: unify as ops:Application, validated against a software catalog of about 49,000 entries; unvalidated strings do not become applications; bidirectional dependency edges are capped to prevent popular services from blowing up the graph. 2. Model incident discussions: structure hypotheses, rejected paths, and mitigations from Slack into triples, so knowledge does not evaporate after channels go cold.

Non-deterministic → deterministic: the entropy ladder LLM summaries, fuzzy matching, and probabilistic causal chains are all unstable; the ontology tightens the output space (e.g., category/sentiment enums) → deterministic triples. A five-layer pipeline progressively reduces the proportion of information "still stuck in unstructured form".

Enricher Many enrichers: read noise → (possibly randomized) analysis → write only typed triples. Invariants: must have rdf:type, a unified URI scheme, and catalog-gated Application. The pipeline runs sequentially (switching to parallel reduces races); external MELT calls are best-effort, and slow APIs degrade graph quality rather than blocking harvest. Metrics / Alerts etc. are also being ontologized, aiming for one-stop querying.

Correlation Engine Reads 13 signal types (log errors, metric anomalies, slow/failed traces, alerts, incidents, releases, configs, experiments, scaling, commits, end-to-end bottlenecks, error patterns, etc.), and within a roughly 10-minute window finds paths by temporal proximity and severity; known causal pairs have a high prior, and correlation chains with confidence <0.6 are discarded. Its role is evidence-backed hypotheses, not verdicts; chains are saved together with the supporting graph records (e.g., 0.87).

Knowledge Flywheel Observe → Enrich → Infer → Adapt: causal chains discovered in one incident become detection rules for the next; the graph grows from schema into "memory" (e.g., narrative insights such as the share of SEV-2s related to api-gateway releases, the share of timeout-type issues, etc.).

Implementation: AutoRCA Multiple specialist Agents collaborate around the same problem; the ontology serves as the shared vocabulary: the person writing trace findings and the person reading logs do not know each other, yet both read and write the same ops:Application triples. Alerts still fire, humans still make the call, code is still written by humans; what is saved is finding owners, stitching together releases, correlating across systems, and recalling similar incidents from last quarter. Hypotheses can still be wrong, so each comes with evidence.

If building your own: eight principles Model identity and the authoritative catalog first; treat the ontology as code (PR/review); separate concepts from instances; use open standards; use the type system to constrain stochastic tools; make enrichment ordered and best-effort; model human reasoning together with telemetry; require human approval for Agent/ontology changes.

Next steps AutoRCA → high-confidence automated remediation (rollback/traffic shifting/code changes) → predictive self-healing. Plans to integrate with Netflix UDA's domain modeling language Upper, starting with incident modeling as a PoC.