Production-shaped data engineering portfolio
Ingest to Insight
A full platform case study for how I design trusted data systems: source ownership, CDC, governed batch and streaming, warehouse publishing, dbt Mesh, observability, and clear downstream access boundaries.
Why this is attractive to senior readers
It demonstrates platform judgment, not just tool familiarity.
The repo shows how data reliability, governance, developer ergonomics, and consumer trust fit together in a system that looks like something a team could operate.
Idempotent, auditable data movement
Atomic object layouts, manifest state transitions, source-target reconciliation, commit markers, and repair paths protect the batch lane from duplicates and partial publishes.
Controls where risky operations happen
Dataset pause switches, approvals, maintenance windows, break-glass overrides, lineage, contracts, and audit tables turn governance into runtime behavior.
Metadata-driven platform boundaries
Registry-backed discovery and orchestrator adapters let new pipelines and data products be added through metadata instead of hardcoded application changes.
Trusted analytics and governed access
Postgres-first dbt Mesh marts, public model contracts, certified products, and semantic metadata show the path from engineering controls to useful decisions.
Architecture
From operational systems to governed data products.
Retail, banking, and commerce sources feed CDC, batch ingestion, object storage, Spark processing, warehouse serving schemas, dbt Mesh marts, governance metadata, and downstream consumers.
Sources
MySQL OLTP
Retail and bank source apps
Commerce REST API
Capture
Debezium CDC
Kafka and Schema Registry
API event poller
Platform runtime
MinIO object lake
Airflow orchestration
Spark batch and streaming
Serving
Postgres warehouse
dbt Mesh marts
Consumers
Governed data-product access
Governance metadata
Downstream consumers
Clear ownership boundaries
Source systems own writes, Debezium captures changes, Airflow and Spark materialize data, dbt owns public analytical contracts, and the Git registry exposes metadata without duplicating orchestration logic.
Portable deployment thinking
Compose, Helm, Kustomize, and Terraform assets show how the same platform concerns move from local development into cloud or Kubernetes environments.
Component map
Every layer has a job, contract, and audience.
Source applications
Retail, bank, and commerce services model application-owned writes, protected business endpoints, metrics, and outbox events.
sources/api/, sources/oltp/CDC and streaming
Debezium, Kafka, Schema Registry, and Spark streaming separate raw row capture from business events and downstream readiness.
platform/streaming/schema-registry/, pipelines/spark/Batch runtime
Airflow DAGs and runtime packages handle watermarks, late data, backfills, contracts, manifests, and atomic publish repair.
pipelines/airflow/Object lake and warehouse
MinIO layouts and Postgres serving schemas demonstrate the storage and serving split.
warehouse/, docker-compose.yamlContracts and registry
JSON contracts and Git-versioned registry files define schema, SLA, owners, sensitivity, certification, and discovery.
contracts/v1/, registry/Airflow operations and governance
Airflow owns workflow execution while the runtime packages enforce contracts, approvals, lineage, and audit evidence.
pipelines/airflow/, warehouse/postgres/dbt Mesh
Domain-owned marts, public model access, source freshness, semantic assets, exposures, and certification evidence sit after the governed serving layer.
analytics/dbt/Companion projects
Metric Guard demonstrates governed AI metric access. Model to Market demonstrates the full ML lifecycle, serving, and operations.
Dedicated AI and ML portfoliosProduction signals
What the implementation makes visible.
The most valuable signal is not the list of tools. It is the way failure modes, review paths, deployment boundaries, and user trust are designed into the platform.
Reliability loop
- Contracts validate rows and split accepted, warning, and rejected records.
- Atomic object layouts publish with manifests, checksums, and commit markers.
- Readiness sensors gate downstream warehouse and dbt work.
- Reconciliation compares source and target rows, hashes, and sums.
Operations loop
- Freshness SLAs and SLO burn-rate rules expose data product health.
- Prometheus, Alertmanager, Grafana, Jaeger, and ELK cover metrics, alerts, traces, and logs.
- Backfill previews estimate rows, runtime, cost, and overwrite risk before execution.
- Restore drills rehearse Postgres, MinIO, Kafka replay, and dbt rebuild validation.
Governance loop
- Dataset controls handle pause, approval, maintenance, concurrency, and emergency override.
- Lineage payloads publish source, object, target, contract, quality, and reconciliation context.
- Certified dbt products expose owners, SLAs, sensitivity, and reviewer metadata.
- Registry-backed contracts keep approved data-product interfaces explicit.
Engineering signal
How this work translates across data and platform teams.
This project is built to make engineering judgment visible: deciding where contracts live, how runtime controls should be enforced, and how observability should map to ownership.
Lead production-minded delivery
Translate ambiguous data platform goals into maintainable services, runtime contracts, validation gates, and deployable assets.
Bridge engineering and analytics
Connect ingestion, CDC, warehouse design, dbt Mesh ownership, and governed downstream access into one operating model.
Design for trust
Make freshness, lineage, approvals, reconciliation, audit, and recovery visible before stakeholders have to ask for them.