Databricks

ETL, PYSPARK, DATABRICKS, Delta Lake, Spark SQL, Data Modeling, Workflow Orchestration, Performance Tuning

Key Responsibilities:

• Develop and maintain data pipelines and transformations using Databricks and PySpark.
• Implement scalable ETL/ELT workflows to ingest, cleanse, and curate data for downstream analytics and reporting.
• Optimize Spark jobs for performance and cost by tuning partitions, caching, joins, and cluster configurations.
• Build reusable notebooks and modular code to support consistent development and easier maintenance.
• Perform data validation, reconciliation, and quality checks to ensure accuracy and reliability of datasets.
• Collaborate with cross-functional teams to gather requirements, clarify data definitions, and deliver aligned solutions.
• Support deployments and production operations by troubleshooting failures, analyzing logs, and resolving incidents.
• Contribute to documentation, coding standards, and best practices for Databricks-based development. Minimum Qualifications:
• Bachelor’s or Master’s degree in BTECH, MTECH, MCA, or MSC (or equivalent).
• 2–3 years of hands-on experience working with Databricks in data engineering or analytics engineering projects.
• Strong experience in PySpark for building transformations and distributed data processing.
• Solid understanding of data pipeline concepts, data modeling basics, and structured/semi-structured data handling.
• Ability to debug and troubleshoot Spark jobs and collaborate effectively within delivery teams.
• Experience with Spark optimization techniques and practical performance tuning in Databricks environments.
• Familiarity with Delta Lake concepts such as ACID tables, schema evolution, and incremental processing patterns.
• Exposure to orchestrating workflows and managing dependencies for end-to-end pipeline execution.
• Experience working in agile delivery models with strong ownership of tasks, timelines, and quality outcomes.
• Strong communication skills to translate requirements into implementable data solutions and clearly document outcomes.