Cvent Knowledge Graph — ETL Pipeline

A 5-stage deterministic TypeScript pipeline that transforms platform metadata into an embedded graph database. Zero LLM tokens. Zero git cloning. ~2.5 minutes wall time. Fully reproducible.

927

Services

10,348

API Endpoints

2,214

Dependency Edges

Teams

243

People

~2.5 min

Total Runtime

AI Cost

External Data Sources

Source

Backstage Catalog API

Service catalog — ownership, deps, annotations, user profiles

No Auth286 calls

Source

Sourcegraph Code Search API

Cross-repo regex — code patterns, configs, framework detection

Token Auth~63 calls

Source

14 OpenAPI / AsyncAPI Spec Repos

Machine-readable API specs — endpoints, channels, schemas

9,565 endpoints501 channels

Stage 1 — Service Discovery

1DISCOVER

Crawl 3 Backstage domains → teams → components → user profiles
Domain→Team→Component 3-hop crawl pattern

3 domains37 teams189 components243 people1.75s

Output

backstage-catalog.json

Full entity payloads + external refs + user profiles

01-discover/

Stage 2 — Framework Classification

2CLASSIFY

Assign runtime + build framework per service
Routes Stage 3 extraction queries by runtime class

Dropwizard 53%Next.js 14.NET 1558 need probes

Output

classify-manifest.json

Service → runtime + build + confidence + probe flag

02-classify/

Stage 3 — Multi-Phase Extraction

3EXTRACT — 4 Sub-Phases

Spec parsing + framework probes + config mining + structural code extraction

~148s total~63 Sourcegraph calls

Spec Extraction

Parse OpenAPI/AsyncAPI from 14 spec repos

618 services9,565 endpoints

Framework Probes

Resolve 58 unknown runtimes via code signals

12 probes91% resolved

Config Extraction

Mine config files for queues, URLs, metadata

135 repos87 service URLs

Structural Extraction

Framework-routed code pattern extraction

39 queries5,513 endpoints3,242 events

Output 3a

spec-extractions.json

Endpoints + channels per service

Output 3b

probe-overlay.json

Resolved runtimes overlay

Output 3c

config-extractions.json

Queue names, URLs, metadata

Output 3d

extractions.json

All structural patterns per service

Stage 4 — Graph Assembly

4TRANSFORM

Deterministic graph assembly — merge, correlate, score, validate
Multi-signal confidence scoring + referential integrity checks

927 services10,348 APIs2,214 dep edges27 CSVs

Output

04-transform/*.csv

10 node tables + 16 edge tables
Flat CSV, COPY FROM-ready

27 files

Stage 5 — Database Load

5LOAD

Create schema + COPY FROM CSVs → LadybugDB
Smoke queries verify row counts + referential integrity

26 tablesCOPY FROM bulk~50MB output

Artifact

graph.db

LadybugDB embedded graph
~12,640 nodes • ~15,800 edges

Embedded DB

Serve Layer

Service

MCP Server (NestJS)

Cypher queries + curated MCP tools
service_info, blast_radius, centrality, critical_path...

10 MCP tools

Application

Next.js UI

3D graph visualization + chat + explorer

Three.jsAI Chat

Consumers

Consumer

AI Agents

Copilot · Claude · Glean

Consumer

Engineers & Architects

Browser · Visual exploration

External Source

Pipeline Stage

Stage Output

Database

Serve Layer

Consumer

Cvent Knowledge Graph ETL Pipeline v1 — Source of Truth Diagram
Last pipeline run: 2026-05-04 • Total: ~2.5min wall time, 0 AI tokens, ~349 API calls
Hover any component for detailed documentation

Cvent Knowledge Graph

Ultra-fast, zero-cost context for every agent.