Back to work
2023 · Systems architect
Lakehouse Ledger
Structured audit-trail infrastructure for regulated data pipelines
Context
A fintech data platform needed immutable lineage across batch and streaming transforms.
Problem
Compliance reviews could not reconstruct metric definition changes. Audit prep consumed three engineer-weeks per quarter.
Architecture
Designed an append-only ledger of schema versions, transform fingerprints, and partition manifests wired into Spark and Flink jobs.
Result
Quarterly audit prep dropped from three engineer-weeks to four engineer-days. Lineage lookup latency stayed under 200ms p95.
Stack
Apache IcebergSparkFlinkPostgreSQLgRPC