PySpark Developer

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Process and transform large datasets from multiple data sources.
• Build and optimize ETL/ELT workflows for data ingestion and processing.
• Develop Spark jobs for batch and real-time data processing.
• Write complex SQL queries for data extraction and reporting.
• Perform data cleansing, validation, and quality checks.
• Optimize Spark performance by tuning jobs, partitions, and cluster configurations.
• Integrate data from APIs, databases, cloud storage, and enterprise systems.
• Work with data architects, analysts, and business stakeholders to understand requirements.
• Troubleshoot production issues and implement performance improvements.
• Ensure data security, governance, and compliance standards are followed.
• Participate in code reviews and follow best development practices.
• Maintain technical documentation for data pipelines and processes.