Databricks Data Engineer

Key Responsibilities:

• Data Pipeline Development:
• Design, build, and maintain scalable data pipelines and architectures using Databricks Lakehouse Platform (Delta Lake, Spark, Spark SQL, Auto Loader, and MLflow)
• Implement complex ETL/ELT solutions across bronze, silver, and gold layers.
• Build reusable frameworks for ingestion, transformation, and monitoring.
• Databricks Platform Engineering:
• Develop and optimize Databricks notebooks, workflows, and job clusters.
• Implement best practices for Delta Lake: optimization, compaction, schema enforcement, time travel.
• Troubleshoot performance issues and fine-tune Spark jobs for efficiency.
• Cloud Integration (Azure / AWS):
• Integrate Databricks with cloud storage (ADLS/S3).
• Implement security, secrets, and networking configurations.
• Data Quality, Observability & Governance:
• Build data validation frameworks and error-handling mechanisms.
• Work with Unity Catalog for cataloging, governance, and access control.
• Ensure data compliance, security standards, and audit processes.
• DevOps & CI/CD:
• Maintain Git-based development workflows.
• Develop CI/CD pipelines for Databricks workflows using Git, and orchestration tools like Apache Airflow or Databricks Workflows.
• Collaboration, Leadership & Guidance:
• Collaborate with architects, analysts, and business stakeholders to refine requirements.
• Mentor junior engineers and conduct code reviews.
• Participate in solution design and architecture discussions.

Skills and Qualifications:

• Bachelor’s / Master’s degree in computer science, computer engineering, or relevant field.
• Overall 3+ years of experience, at least 2 years of experience in Databricks Data Engineering.
• Strong, hands-on experience with the Databricks workspace, Delta Lake, Delta Live Tables (DLT), and Unity Catalog.
• Proficiency in Apache Spark, PySpark, and distributed computing principles.
• Advanced proficiency in Python and SQL.
• Hands-on experience with cloud environments such as AWS, Azure.
• Experience with Git, CI/CD pipelines, and infrastructure-as-code tools (Terraform, CloudFormation, or Databricks Asset Bundles).
• Experience with ETL processes: Extracting, transforming, and loading data.
• Strong analytical and problem-solving skills.
• Good communication and collaboration skills.
• Experience with data security and governance principles.
• Understanding of data visualization techniques.