Big Data Developer

Role: Mid-level Big Data Engineer

Position Type: Full-Time Contract (40hrs/week)

Contract Duration: 6 months + extendable

Work Schedule: 8 hours/day (Mon-Fri)

Location: Hybrid - Wed/Thursday to be in onsite in Hyderabad, India

We are seeking a Mid-Level Big Data Engineer with 5+ years of experience developing scalable Big Data solutions and distributed data-processing applications in AWS environments.

The ideal candidate will have strong hands-on experience with Scala, Python, Apache Spark/PySpark, AWS, Linux, and shell scripting, along with experience building batch and streaming data pipelines and optimizing large-scale distributed workloads.

Key Responsibilities

• Design, develop, test, and deploy scalable Big Data solutions on AWS.
• Build and maintain batch and streaming data pipelines using Scala, Python, Spark, and PySpark.
• Process and transform large volumes of structured and unstructured data.
• Develop and support distributed data-processing and data-ingestion platforms.
• Integrate new data sources and technologies into existing Big Data ecosystems.
• Optimize Spark, Hadoop, EMR, and distributed computing workloads.
• Build cloud-native APIs and microservices supporting data platforms.
• Automate data workflows, deployments, monitoring, and operational processes.
• Collaborate with architects, product owners, QA teams, and business stakeholders.
• Apply security, governance, reliability, and software engineering best practices.Requirements
• 5+ years of experience in Big Data development and data/software engineering.
• Strong hands-on experience with Scala and Python.
• Strong experience with Apache Spark / PySpark.
• Experience with the Hadoop ecosystem, including HDFS, MapReduce, and Hive.
• Experience with Kafka or similar streaming technologies.
• Strong hands-on AWS experience, particularly with EMR, S3, EC2, ECS/EKS, MWAA/Airflow, Step Functions, API Gateway, Lambda, DynamoDB, and/or RDS/Aurora.
• Experience building large-scale batch and API-based data-processing solutions.
• Strong Linux and shell scripting skills.
• Experience with data ingestion, transformation, data modeling, schema design, and data lifecycle management.
• Strong understanding of performance tuning for Spark, Hadoop, EMR, and distributed systems.
• Experience working in Agile/Scrum environments.
• Strong troubleshooting and problem-solving skills.Preferred
• Generative AI / AI-assisted development
• LLMs, RAG, Vector Databases, and AI/ML integrations
• AI-enabled data platforms
• Git, GitHub, GitLab, Bitbucket, Jenkins, Maven, Gradle, Artifactory
• Docker / Kubernetes
• Automated testing and data quality tools
• Enterprise-scale cloud environments
• AWS or Databricks certifications