System design
Architecture diagrams, ADRs, and explicit boundaries between ingestion, transformation, quality, and platform layers.
Senior Data Engineer & Data Architect
Data platforms where pipelines fail loudly — not silently
I build production-grade systems that survive partial failures, schema drift, and operational scale: incremental ingestion with checkpoint recovery, contract-driven data quality, medallion lakehouse patterns, and CI-validated platform automation on AWS, Azure, Databricks, and Snowflake.
- End-to-end ownership — ingestion, transformation, orchestration, and observability
- Operational maturity — idempotent loads, run history, alerts, and documented recovery
- Platform thinking — IaC, governance, lineage, and tests that catch regressions before production
Design for failure. Automate repeatable work. Measure data quality. Document decisions. Keep systems understandable.
About
I'm Bobby Ray (br413) — a Senior Data Engineer and Data Architect who turns fragmented sources into reliable, observable, analytics-ready platforms. Senior data engineering, to me, is less about tool count and more about judgment under constraint: choosing trade-offs intentionally, designing for failure before the first incident, and keeping systems understandable for the team that inherits them.
My open-source work spans connected layers — a **runnable lakehouse platform** (Iceberg, Trino, Cosmos, OpenLineage), a production ETL pipeline with Airflow and dbt, contract-driven quality observability, and platform IaC blueprints. Together they show how I think about the full data lifecycle, not isolated scripts.
Failure handling
Checkpoint recovery, idempotent loads, quality gates before bronze, and runbooks for 2 AM incidents.
Operational proof
pytest coverage, GitHub Actions CI, smoke tests, alert routing, and documented trade-offs in every flagship repo.
Data engineering
Python, SQL, incremental ingestion, ETL/ELT, API pipelines, checkpointing, idempotent loads
Data architecture
Medallion lakehouse, bronze/silver/gold layers, lineage, governance, cost modeling
Orchestration
Apache Airflow, dbt, Prefect, Spark
Cloud & quality
AWS, Azure, Databricks, Snowflake, data contracts, CI/CD, monitoring, alerting
Platform approach
Senior data platform work is a stack, not a single pipeline. My portfolio models how ingestion, quality, and platform governance connect in production.
Flagship
Lakehouse platform
Iceberg + Trino + Cosmos dbt + Airflow + OpenLineage + GE — runnable Docker stack, CI, hosted dbt docs.
Layer 1
Ingest & transform
Incremental API ingestion, PostgreSQL bronze, quarantine/DLQ for bad records, dbt silver/gold, Airflow orchestration, webhook alerts.
Layer 2
Quality & observability
YAML data contracts, schema/freshness/RI checks, persisted run history, alert routing.
Layer 3
Platform & governance
Medallion manifests, Terraform IaC, IAM governance, lineage graph, CI validation.
Outcome: pipelines that fail loudly, contracts that catch drift early, and platform patterns that teams can operate without tribal knowledge.
Data engineering portfolio
Production-style open-source projects with architecture docs, ADRs, tests, CI, and operations runbooks.
Flagship · Runnable lakehouse platform
lakehouse-platform-starter
Airflow + Cosmos orchestrates PyIceberg ingest into Iceberg bronze, dbt-trino transforms via Trino, OpenLineage feeds Marquez, and Great Expectations gates publish.
Senior signal: dual DuckDB/Trino targets, 14 dbt tests, incremental marts, Terraform IaC, CI smoke tests, and hosted dbt docs.
v1.0.0 release →End-to-end pipeline
production-data-pipeline
Incremental API ingestion with PostgreSQL bronze landing, per-record quarantine/DLQ, dbt silver/gold models, Airflow orchestration, and webhook alerts.
Senior signal: checkpoint recovery, idempotent loads, quarantine table for poison-pill triage, and dbt test gates that stop bad data before gold.
v0.2.1 release →Quality layer · Contracts & observability
data-quality-observability
YAML data contracts with schema, freshness, and referential-integrity checks, persisted history, and alert routing.
Senior signal: contract-driven checks with run history — quality regressions are auditable, not one-off surprises.
View repository →Platform layer · Lakehouse blueprint
cloud-lakehouse-blueprint
Medallion architecture manifests, Terraform IaC, IAM governance, lineage graph, and CI validation with validate --json.
Senior signal: platform decisions encoded in IaC, manifests, and CI — not slide-deck architecture.
View repository →Open-source contributions
Production operations knowledge contributed upstream to orchestration and analytics tooling teams rely on daily.
dbt docs #9781 open
Fusion telemetry: use duration_ms for slowest-nodes ranking — fixes misleading span wall-clock examples.
Meltano #10253 open
elt vs run decision guide for replication workloads — closes maintainer-requested docs gap.
Airflow #71158 merged
Clarify metrics vs traces otel_* config options — merged upstream.
Airflow #70171 open
Surface dbt Cloud failure details in Airflow task logs for faster on-call triage.
Prefect #22500 merged
Kubernetes readiness vs liveness probes for workflow orchestrators — the kind of ops detail that prevents false-positive deploy rollbacks.
Prefect #22533 open
Clarify global concurrency limit setup for production Prefect deployments.
dbt docs #9606 merged
Troubleshooting note for prefixed custom schema names — a deployment pitfall that breaks analytics environments silently.
Writing
Technical articles on production pipeline patterns, failure handling, and platform design.
Building a Production Data Pipeline with Incremental Loading and dbt
How to design idempotent ingestion, checkpoint stores, medallion-style transformations, and Airflow orchestration — with failure modes and scale trade-offs.
Related repos: production-data-pipeline · cloud-lakehouse-blueprint · lakehouse-platform-starter
Data Quality Contracts in Production Pipelines (Without a Separate Platform Team)
Row-level quarantine at ingestion, YAML dataset contracts at the quality boundary, alert routing, and CI enforcement — the follow-up to the incremental pipeline article.
Related repos: production-data-pipeline · data-quality-observability
What I Learned Contributing to Prefect, dbt, and Airflow (An Honest OSS Retrospective)
Two merges, four open PRs, and ninety days of building in public — what actually worked for upstream contributions.
Related repos: br413 · production-data-pipeline · data-quality-observability · cloud-lakehouse-blueprint
Contract Versioning in Production Pipelines: Registry, CLI, and Run History
Git-native contract registry, semver pins, versioned run history, and CI guards — the follow-up to the data quality contracts article.
Related repos: data-quality-observability · production-data-pipeline
Collaboration
Open to senior data engineering roles, data platform architecture discussions, and technical collaboration with engineering teams building analytics infrastructure, quality tooling, or lakehouse platforms.