Data Platform Engineer

Data Platform Engineers build the shared data infrastructure that everyone else depends on. They own the pipelines, storage, transformation, orchestration, and governance that turn scattered raw data into a trustworthy product the company can use. Every analytics dashboard, every ML model, every AI application ultimately sits on top of what this role builds.

Before you start

• One cloud provider to a working level (AWS, Azure, or GCP)
• Docker and Kubernetes fundamentals
• Terraform for infrastructure as code
• CI/CD pipelines (GitHub Actions or GitLab CI)
• Python and SQL to a confident, practical level
• Basic observability (Prometheus, Grafana)

The roadmap

1. Types of Data

   Structured (rows and columns, fixed schema), semi-structured (JSON, Avro, schema embedded in the data), and unstructured (PDFs, images, audio, video). The mix of shapes in your platform decides your tooling and storage choices.

2. Object Storage and File Formats

   Object storage (S3, GCS, ADLS) is the bottom of every modern data platform. Cheap, durable, API-accessed. Parquet is the columnar format you will use most of the time for analytics.

3. Table Formats (Delta Lake and Iceberg)

   Parquet alone is not enough for production. Delta and Iceberg add ACID transactions, time travel, schema evolution, and concurrent writes on top of Parquet files in object storage.

4. Cloud Warehouses (Snowflake or BigQuery)

   SQL warehouses are still where most analytics actually lives. Snowflake for enterprise, BigQuery for Google-heavy stacks, Databricks SQL if you are going pure lakehouse.

5. Batch Ingestion (Fivetran, Airbyte)

   Pulling data from SaaS and operational sources is half of ingestion. Fivetran is the commercial standard, Airbyte the open-source equivalent. Both follow the modern ELT model.

6. Streaming with Kafka

   The dominant streaming platform. Producers, consumers, topics, partitions, consumer groups, replication, Kafka Connect. Real-time pipelines all flow through this.

7. Change Data Capture (Debezium)

   Stream every database change as an event. CDC turns operational databases into a real-time source for downstream analytics without adding query load to the source.

8. Transformation with dbt

   The standard for SQL-based transformation in modern ELT. Models, ref(), materialisations, tests, docs, lineage, snapshots, macros.

9. Distributed Compute (Spark and Databricks)

   When SQL is not enough or the data is too big. PySpark for distributed Python processing, Databricks as the most common managed platform.

10. Orchestration with Airflow

    The control plane for your pipelines. DAGs, operators, scheduling, sensors, dynamic task generation, retries, SLAs, alerts.

11. Data Quality (Great Expectations)

    Catch bad data at the boundary before it spreads. Expectations, checkpoints, data docs, integration with pipelines as quality gates.

12. Governance and Catalogs (Unity Catalog, DataHub)

    Discoverability and access control. Who can find what, who can read what, where data came from, who is allowed to use it.

13. Compliance and Governance Regimes

    Every serious platform operates under compliance regimes. Right to erasure, data residency, audit logs, encryption at rest and in transit, access controls, PII handling.

This is the free version. The full one, with what good looks like for every skill and a production capstone project to prove it, is what I teach inside my community. Send me a DM if you want to know more.