Zhe

Worked as a Data Engineer with a strong focus on data platform engineering, owning core infrastructure for energy market data ingestion, validation, orchestration, and consolidation across the European energy market. Key responsibilities and achievements: - Co-led the design and build of the Market Data Platform (MDP) from scratch — a production-grade, streaming-first platform (Medallion + Kappa-style) managing ~20 data sources and ~200 pipelines, replacing the legacy ingestion system (MDIS) with migration ~90% complete at departure. Built around: a connector-based ingestion framework unifying polling and push-based sources; a YAML-based data contract system as single source of truth for schema, validation, and config; and Redis-backed deduplication enabling idempotent reprocessing and realtime vs backfill separation. - Implemented 7–9 production connectors (EPEX MATS, Energy Quantified, Volue, EEX, APG, GME, IPTO, DAMAS) across varying API constraints and polling patterns. - Delivered under pressure: debugged undocumented EPEX MATS streaming API issues ahead of a conformance deadline; built a PDF parser for DAMAS (Romania), unlocking the company's most profitable market. - Owned production reliability — triaging missing data, upstream outages, schema drift; established post-mortem process and alerting channel from scratch. - Maintained a data consolidation layer using dbt, using MDP outputs as canonical sources to support downstream use cases including DWH, analytical workloads, production model inference, and backtesting. - Operated self-hosted Airflow on GKE; independently completed a major version upgrade; improved observability with Grafana/Prometheus. - Frequently covered ownership and operational gaps during extended periods of team instability (multiple management changes, sustained low headcount), ensuring continuity of critical data services relied upon by Trade Optimisation, Market Forecasting, Flex, and Intraday teams.Worked as a Data Engineer with a strong focus on data platform engineering, owning core infrastructure for energy market data ingestion, validation, orchestration, and consolidation across the European energy market. Key responsibilities and achievements: - Co-led the design and build of the Market Data Platform (MDP) from scratch — a production-grade, streaming-first platform (Medallion + Kappa-style) managing ~20 data sources and ~200 pipelines, replacing the legacy ingestion system (MDIS) with migration ~90% complete at departure. Built around: a connector-based ingestion framework unifying polling and push-based sources; a YAML-based data contract system as single source of truth for schema, validation, and config; and Redis-backed deduplication enabling idempotent reprocessing and realtime vs backfill separation. - Implemented 7–9 production connectors (EPEX MATS, Energy Quantified, Volue, EEX, APG, GME, IPTO, DAMAS) across varying API constraints and polling patterns. - Delivered under pressure: debugged undocumented EPEX MATS streaming API issues ahead of a conformance deadline; built a PDF parser for DAMAS (Romania), unlocking the company's most profitable market. - Owned production reliability — triaging missing data, upstream outages, schema drift; established post-mortem process and alerting channel from scratch. - Maintained a data consolidation layer using dbt, using MDP outputs as canonical sources to support downstream use cases including DWH, analytical workloads, production model inference, and backtesting. - Operated self-hosted Airflow on GKE; independently completed a major version upgrade; improved observability with Grafana/Prometheus. - Frequently covered ownership and operational gaps during extended periods of team instability (multiple management changes, sustained low headcount), ensuring continuity of critical data services relied upon by Trade Optimisation, Market Forecasting, Flex, and Intraday teams.

- Built and maintained an analytical data platform on GCP. - Scheduled ETLs to gather data from various data sources within the organization to the data lake. - Enabled over 50 analysts to build and maintain their scheduled pyspark(-sql) jobs through self-service templates. - Enabled over 50 analysts to explore data from the data lake and their pyspark(-sql) job output via Superset. - Established CI/CD pipelines to ensure quality code delivery. - Powered all these functionalities through the GKE cluster.