Senior Data Engineer – Azure Databricks & Cloud Analytics

About the role

EPAM is looking for a Senior Data Engineer to design and operate cloud‑based data products that enable insurance analytics and reporting. You will build reliable pipelines on Microsoft Azure, leverage Databricks and Python, and ensure data quality, governance, and security while collaborating with data and business teams.

Key responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines in Azure Databricks.
• Create batch and streaming solutions using Azure Data Factory, Azure Data Lake Storage Gen2, Azure Event Hubs and Kafka.
• Partner with data architects, analysts and stakeholders to define requirements and deliver data assets.
• Implement data quality checks, validation, cleansing and governance, including Informatica lineage, metadata and cataloging.
• Apply security and compliance controls (RBAC, managed identities, encryption) aligned with PDPO and GDPR.
• Automate deployments with GitHub Actions and enforce solid operational practices.
• Troubleshoot pipeline issues, perform root‑cause analysis and optimise performance.
• Maintain clear documentation for transformations, testing and runbooks.

Required profile

• Hands‑on experience building data pipelines with Azure Databricks and Apache Spark (PySpark or Scala).
• Strong background with Azure Data Factory and Azure Data Lake Storage Gen2.
• Proficiency in Python for data transformation and automation.
• Experience with streaming platforms such as Apache Kafka or Azure Event Hubs.
• Knowledge of data governance, metadata and lineage tools (e.g., Informatica).
• Understanding of security and compliance controls (RBAC, encryption, Azure Key Vault).
• Practical experience with CI/CD automation using GitHub Actions.

Required skills

• Azure Databricks
• Apache Spark (PySpark/Scala)
• Python
• Azure Data Factory
• Azure Data Lake Storage Gen2
• Azure Event Hubs
• Kafka
• Informatica (lineage, metadata, cataloging)
• RBAC and encryption controls
• Azure Key Vault
• GitHub Actions

What we offer

• Work in a globally recognised, award‑winning workplace.
• Supportive environment where employee ideas drive business.
• Opportunities to work with cutting‑edge technologies and industry trends.
• Transparent career path with individual development roadmap.
• Access to online courses, mentoring, certification grants and global knowledge exchange.