Data Engineer

About the role

The Hong Kong Jockey Club is seeking a Data Engineer to design, develop and operationalise reliable, scalable ETL/ELT pipelines on AWS. The role focuses on large‑scale data processing using PySpark and SQL, supporting the Club’s racing and entertainment data ecosystem.

Key responsibilities

• Design, develop, test and deploy ETL/ELT pipelines with PySpark and SQL on the AWS platform.
• Implement and manage a cloud data lake, orchestrating data flows using the medallion architecture.
• Optimise pipelines and storage for performance, cost efficiency and reliability while handling terabyte‑ to petabyte‑scale data.
• Collaborate with business users and product owners to understand data requirements.
• Monitor, troubleshoot and support production data workflows.

Required profile

• 5‑8 years of hands‑on data engineering experience, including production‑level PySpark and SQL workloads.
• 3+ years managing end‑to‑end data pipeline lifecycles and cloud data platforms on AWS.
• Experience with data governance, security compliance, and real‑time streaming (Kafka or Solace).
• Knowledge of medallion architecture, data modelling, and infrastructure‑as‑code (Terraform or CloudFormation).
• Familiarity with DevOps/DataOps practices such as CI/CD, automated testing and pipeline monitoring.
• Relevant AWS and data‑visualisation certifications are a plus.

Required skills

• PySpark, SQL
• AWS services: S3, Glue, EMR, Redshift, Athena, Lake Formation
• Orchestration: Apache Airflow, AWS Glue
• Data lake formats: Delta Lake, Iceberg
• Infrastructure as code: Terraform, CloudFormation
• Streaming: Kafka, Solace
• Data visualisation: Power BI, Tableau
• CI/CD, monitoring, data governance and security