Azure Data Engineer

Senior Azure Data Engineer – Azure | Databricks | Python | PySpark

Experience: 5–10 Years

Location: Remote - India

Engagement: Contract

Duration: 6+ months, with possibility of extension

About the Role

We are seeking a highly skilled Senior Azure Data Engineer to design, build, and optimise a next-generation cloud data platform. The ideal candidate will have strong hands-on expertise in Microsoft Azure, Databricks, Python, and PySpark, along with experience leading complex legacy-to-cloud data migrations and building scalable batch and streaming data pipelines.

The role requires strong technical leadership, stakeholder management, and the ability to maintain business continuity during data platform modernisation and architecture transitions.

Key Responsibilities

• Design, build, and optimise scalable cloud data platforms using Microsoft Azure and Databricks.
• Lead legacy-to-cloud data migration initiatives, ensuring data integrity, performance, and business continuity.
• Develop and maintain robust batch and streaming data pipelines using Python, PySpark, and Databricks.
• Assess existing data architectures and design migration strategies for modern cloud-based platforms.
• Optimise data processing workflows for scalability, reliability, performance, and cost efficiency.
• Collaborate with data engineering, architecture, application, and business teams to translate requirements into technical solutions.
• Lead technical discussions, resolve complex engineering challenges, and establish data engineering best practices.
• Manage stakeholder and client relationships, communicating project progress, technical decisions, risks, and dependencies.
• Identify migration risks and dependencies, and develop mitigation strategies to minimise disruption during architecture transitions.
• Support data quality, validation, testing, and operational readiness throughout migration and deployment activities.

Required Skills and Experience

• Strong professional experience in Azure Data Engineering, ideally at a senior or lead level.
• Hands-on expertise in Microsoft Azure data services and cloud data platform architecture.
• Strong experience with Azure Databricks, Python, and PySpark.
• Experience designing and implementing scalable batch and streaming data pipelines.
• Proven experience handling legacy data migrations to cloud-based platforms.
• Strong understanding of data architecture modernisation, migration planning, data validation, and performance optimisation.
• Ability to lead technical delivery, troubleshoot complex data engineering problems, and guide engineering teams.
• Excellent communication and stakeholder management skills, with experience working directly with clients and business teams.
• Ability to manage technical risks and maintain business continuity during platform migrations.