GCP Senior Data Engineer
Tata Consultancy Services
Chennai, Tamil Nadu, India
GCP Senior Data Engineer | 6-10 Years of experience |
Chennai/ Hyderabad / Kolkata /Bangalore
Required Technical Skills
- Python
- SQL
- PySpark
- GCP (Preferred) or Azure
- Snowflake
- Airflow
- Jenkins / Jira / Git
- CI/CD
- Kubernetes / Docker
Must-Have
- Strong development experience in Python, PySpark, and SQL.
- Experience with cloud platforms (GCP preferred).
- Experience working in an Agile delivery environment.
- Experience with orchestration tools such as Airflow and ADF.
- Experience with GCP services:
- GCS Bucket
- Cloud Functions
- Dataproc
- Dataflow
- Pub/Sub
- BigQuery
- Hands-on experience in Snowflake.
- Experience building automated data pipelines.
- Experience in data analysis and data exploration.
- Experience working in a multi-developer environment using version control tools such as Git.
- Strong critical thinking, communication, and problem-solving skills.
Good-to-Have
- Understanding of DevOps best practices and CI/CD.
- Understanding of containerization technologies such as Kubernetes and Docker.
- Healthcare domain knowledge.
Key Responsibilities
- Perform data analytics on large-scale datasets to solve complex business problems and recommend solutions.
- Develop and enhance end-to-end data pipelines using big data technologies such as Hadoop, Cloud platforms, and the Spark ecosystem.
- Troubleshoot complex data structures and perform root cause analysis to proactively resolve operational and product issues.
- Develop and document test cases, execute testing, and share UAT results as part of Agile sprints.
- Perform end-to-end data pipeline integration testing in compliance with change management processes and standards.
- Adhere to coding, design, and application standards, including Python and SQL best practices.
- Leverage DevOps practices for production deployments, including CI/CD, test-driven development, and version control using Jenkins, Jira, and Git.
- Collaborate with business stakeholders to define product objectives, business requirements, system designs, and key performance indicators (KPIs).