Freelance Opportunity - Databricks & PySpark Engineer

Work Mode: Remote

Employment Type: Part-Time | Freelance

About the Role

We are looking for a skilled Databricks & PySpark Engineer with strong hands-on experience in Python, SQL, data transformations, and data pipeline debugging. The ideal candidate will be responsible for troubleshooting existing Databricks notebooks, validating data-processing logic, identifying data discrepancies, and ensuring reliable data pipeline execution across multiple environments.

Key Responsibilities

• Debug existing Databricks notebooks and troubleshoot data pipeline issues.
• Develop, review, and validate data transformation logic using PySpark, Python, SQL, and DataFrames.
• Analyze conditional logic, thresholds, null handling, and data validation rules.
• Compare input and output datasets to identify discrepancies and ensure data accuracy.
• Work with Databricks catalogs, schemas, tables, job configurations, and access permissions.
• Support data validation and testing across Dev, QA, UAT, and Production environments.
• Create test datasets and validate data-processing scenarios.
• Follow safe data-writing practices, production access controls, and deployment procedures.
• Collaborate with stakeholders to investigate data quality issues and implement reliable fixes.

Mandatory Technical Skills

• Strong hands-on experience with Databricks and PySpark.
• Proficiency in Python, SQL, DataFrames, and data transformation logic.
• Experience debugging notebooks and resolving data pipeline issues.
• Strong understanding of null handling, conditional logic, thresholds, and data validation.
• Experience with input/output dataset reconciliation and discrepancy analysis.
• Knowledge of Databricks catalogs, schemas, tables, jobs, and permissions.
• Understanding of Dev, QA, UAT, and Production environments.
• Experience creating test data and validating data-processing logic.