Senior Data Engineering

Role Description

Job Description for Data Engineer – Databricks Data PlatformRole Overview

Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.

Key Responsibilities

• Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
• Ingest data from databases, APIs, event streams, and flat files.
• Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
• Write data transformation logic in PySpark, Spark SQL, and Python.
• Implement data quality checks, validation rules, and reconciliation processes.
• Optimise Spark workloads for performance, cost, and scalability.
• Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
• Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
• Monitor pipelines, triage failures, and restore services within agreed SLAs.
• Maintain technical documentation: data models, pipeline flows, runbooks.
• Adhere to data security, privacy, and regulatory compliance standards.

Must-Have Skills

• Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.
• Apache Spark: PySpark and Spark SQL for large-scale data processing.
• Delta Lake: ACID transactions, schema evolution, time travel, merge.
• Medallion Architecture or equivalent layered data platform design.
• Python for pipeline development and transformation logic.
• AWS: S3, Glue, IAM, EC2, Lambda.
• Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).
• Data modelling: star schema, snowflake schema, Data Vault.
• CI/CD, Git, version control, and Agile delivery practices.

Good-to-Have Skills

• Unity Catalog for data governance and lineage.
• AWS Kinesis for real-time streaming.
• Data Warehouse and Informatica PWC Knowledge
• dbt for SQL-based transformation workflows.
• AWS Redshift or Athena for cloud data warehousing.
• Terraform or AWS CloudFormation for infrastructure-as-code.
• AWS Step Functions for workflow orchestration.
• Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).
• Industry experience in insurance or financial services.
• Databricks or AWS Data Analytics certification.

Technical Environment

• Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.
• Languages: Python, PySpark, SQL, Spark SQL.
• Ingestion: Auto Loader, APIs, JDBC, CDC tools.
• Processing: Apache Spark, Structured Streaming, Delta Live Tables.
• Cloud: AWS.
• Orchestration: Databricks Workflows, AWS Step Functions.
• DevOps: Git, CI/CD, Terraform, GitHub Actions.
• Analytics: Power BI, Tableau, Looker.

Experience And Qualifications

• Bachelor's in Computer Science, Engineering, IT, or equivalent.
• 4–8 years in data engineering, ETL/ELT, and cloud data platforms.
• Production-grade pipeline delivery on AWS.
• Agile or DevOps-driven delivery environment.