Gauntlet's
Data Platform Migration

Paradox of Choice

• Aera V3Most technical · on-chain / off-chain hybrid vault protocol. Near-realtime oracle, powered by realtime data arch. Strict security. dApp. SDKs. $100M+ TVL in 6mo.
• Wells Fargo Compliance SystemMost operational · Bespoke on-prem legal & compliance system, powered by a TB-scale versioned international legal-doc datastore. 100s+ data entry/qa and eng involved
• COVID Act NowMost globally impactful · public data platform scaled 0 → 4M users in 48h; cited in White House briefings.
• GigsterMost platform-shaped · Gig CLI + self-managed k8s (OIDC), multi-cloud AWS/GCP, ~1,000 freelance engineers.

Risk management for the biggest protocols in DeFi

i.e. what risk parameters keep each protocol solvent under stress?

• ·
  1 successful run in a nine-month period.
• ·
  Each nightly run took ~18h and fired 1,000+ operations - the full suite determined dynamically by the experiments.
• ·
  Unrenderable UI - the batch topology had no observability or debuggability
• ·
  Idempotency not guaranteed - reruns and backfills had to be managed manually.
• ·
  Quants added more ops each day; backfills launched from local laptops piled on compounding scale.
• ·
  etcd request-size limit hit periodically - the Argo/KFP workflow manifest outgrew etcd's ~1.5 MB per-object ceiling.
• ·
  Backfills spawned ~100k pods in a single zonal cluster → the k8s master API went unresponsive.
• ·
  Urgent forcing function: GKE v1.21 hit EOL in January - by mid-April a forced upgrade would break the cluster.

Share a 1 pager, with a phased roadmap

Orchestrator Comparison Matrix

Mobilize and Act

Drive to provable parity

The platform was live; the team had to keep moving the same day.

• ·
  Company demo - set the stage; the company adopted the new paradigm.
• ·
  Team retro - lessons learned; solidify morale.

Harden the platform, ramp the team

# one declaration → a VaR asset per protocol
SIMPSON_LAR_VAR = {
    protocol: FunctionBasedAsset(
        logical_name=f"{protocol.value}SimpsonLarVar",
        materialize_fn=_fetch_data(protocol, info),
        data_schema=SimpsonLarVarSchema,
        table_name=f"{protocol.value.protocol_str}_lar_var",
        grouping=AssetGrouping.SIMULATION,
    )
    for protocol in PROTOCOLS   # compound · euler · moonwell
}

Incrementally improved over next 6mo

REALISM - Real-time Streaming

Reliable Events And Logic In Six Minutes

• Solana & EVM re-org aware ws rust-based indexers
• Continuous complex window aggregations in 15s
• Cascade of 20+ materialized views
• idempotent processing ½ PB scale
• table-aware metric APIs with caching
• python asyncio streaming subgraph for logical data eng framework

Progress through Iteration.

Lasting Retrospective

• ·
  Prioritizing DevX was the most reliable bet.
• ·
  Follow through after crisis was important.
  Fully materialized wins came after 6+ months of continuous improvement.
• ·
  Could have done better - Closed the loop for teams to reliably address their own dags sooner.

Ask me anything

Transpiler: ~2,000 legacy jobs → Dagster, by graph

class LegacyRegistryCompiler:
    # compile every legacy Kubeflow job into a Dagster asset
    def from_registry(self, registry: PlanRegistry, selections=None):
        return [
            self.transpile_map[type(entry)](entry, refs)
            # walk the dependency graph in topological order
            for generation in nx.topological_generations(
                registry.inverted_graph()
            )
            for entry in generation
            if selections is None or entry.logical_name in selections
        ]

One pass over the legacy registry, topologically sorted - no hand-porting, dependency order preserved.

Hard-won infra gotchas