AWS Pyspark Data Engineer
Tata Consultancy Services
Indore, Madhya Pradesh, India
Overall Experience - 5+
Location - Indore
Job Summary
We are seeking an experienced AWS PySpark Developer with strong expertise in building scalable data engineering solutions on AWS Cloud. The ideal candidate should have hands-on experience in PySpark, Spark, AWS data services, ETL development, and large-scale data processing. The role involves designing, developing, and optimizing cloud-native data pipelines, data lakes, and analytics solutions.
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines using PySpark.
- Build and maintain data ingestion frameworks on AWS.
- Develop batch and real-time data processing solutions.
- Optimize Spark jobs for performance and cost efficiency.
- Work with structured, semi-structured, and unstructured datasets.
- Develop data lake and data warehouse solutions.
- Implement data quality checks, monitoring, and alerting.
- Collaborate with business stakeholders, architects, and analytics teams.
- Support production deployments and troubleshoot critical issues.
- Follow DevOps and CI/CD best practices.
Mandatory Technical Skills
AWS Services
- Amazon S3
- AWS Glue
- AWS EMR
- AWS Lambda
- Amazon Redshift
- AWS IAM
- CloudWatch
- AWS Step FunctionsBig Data & Data Engineering
- PySpark
- Apache Spark
- Spark SQL
- DataFrames & RDDs
- ETL/ELT Development
- Data Modeling
- Data Warehousing ConceptsProgramming
- Python
- SQL
- Shell ScriptingDatabase Technologies
- Redshift
- PostgreSQL
- MySQL
- DynamoDB (Preferred)DevOps & CI/CD
- Git
- Jenkins / AWS CodePipeline
- Docker
- Terraform (Preferred)