Databricks Data Engineer

Role Overview

We are looking for a highly skilled Databricks Data Engineer with strong hands-on experience in building modern data platforms, Lakehouse architecture, data governance, and AI-enabled solutions on Databricks. The ideal candidate should be capable of designing, developing, optimizing, and maintaining scalable data pipelines and analytics platforms.

Key Responsibilities

Data Engineering & Lakehouse

 Design and implement Medallion (Bronze, Silver, Gold) architecture.

 Develop scalable data pipelines using Python, PySpark, and SQL.

 Build dimensional models including Facts, Dimensions, Star/Snowflake schemas.

 Design normalized and denormalized data models based on business needs.

Data Ingestion & Transformation

 Implement ingestion solutions using Fivetran, Auto Loader, APIs, Streaming,

Delta Sharing/Open Sharing, and CDC.

 Build Lakehouse solutions using Delta Lake, Delta Live Tables (DLT), Lakeflow, and dbt.

 Implement SCD Type 1 & 2, bitemporal data models, and data versioning.

 Leverage Delta Lake features such as ACID transactions and Time Travel.

 Integrate data quality checks, error handling, debugging, and performance optimization.

 Understand and evaluate Delta Lake vs Apache Iceberg architectures.

Orchestration & DevOps

 Develop and manage w orkflows using Databricks Workflows, Jobs, Airflow, and ADF.

 Implement CI/CD pipelines and Databricks Asset Bundles.

 Optimize and manage SQL Warehouses, Serverless Compute, and Clusters.

Governance & Security

 Implement Unity Catalog governance solutions.

 Manage RBAC, ABAC, data masking, and access controls.

 Work with managed and external tables.

 Create semantic layers using Metric Views.

AI & GenAI – Good to Have

 Experience with Genie Spaces, AI Agents, Agent Bricks, and Agent Skills.

 Exposure to spec-driven development using GenAI tools such as Claude.

 Enable AI-powered data and analytics solutions within Databricks.

Required Skills

 Databricks, Delta Lake, DLT, Lakeflow

 Python, PySpark, SQL

 Data Modeling (Facts, Dimensions, SCD)

 CDC, Streaming, Auto Loader

 Unity Catalog

 Airflow, ADF, Databricks Workflows

 CI/CD, Asset Bundles

 Performance Tuning & Troubleshooting

Preferred Qualifications

 Experience with enterprise-scale Lakehouse implementations.

Soft Skills

 Excellent verbal and written communication.

 Strong stakeholder management and problem-solving abilities.

 Ability to drive technical discussions and mentor team members.