Data Engineer

THE ROLE: Data Engineer for Insight Globals AI Product & Services Division out of our Hyderabad location

We are looking for a Data Engineer to build the data foundation that IG Labs' agentic systems run on. Every agent we deploy into a client is only as good as the data and structure beneath it, and you are the engineer who builds that foundation: the pipelines, the models and ontologies, and the retrieval layer that feeds the agents the right context. This is backbone engineering, core to everything we deliver, not a service on the side.

You take on the hardest data problems in our client work: the source no one can cleanly extract, the ontology that has to capture how a business really works, the retrieval that has to surface the right context every time. You own the data foundation from a raw, messy starting point to a governed, reliable system in production, and you are accountable for whether the agents built on it can be trusted.

Everything you build, you build to last beyond one client. The pipelines, connectors, and ontology fragments you create become Factory IP that compounds across every engagement, so each project starts ahead of the last. You are a senior engineer in our delivery organization, based in India, working within the POD model alongside the Technical Architects, Forward Deployed Engineers, and the UI/UX and Technical/Field engineers who build the rest of the system.

Key Responsibilities

1- Solving the Hardest Data Problems

• Take on the toughest data problems in our client work: the messy, incomplete, undocumented client data that stands between an idea and a working agent.
• Design the data models and ontologies the agentic system operates on: entities, relationships, states, and the business rules that govern them.
• Build the retrieval substrate that agentic systems depend on: chunking, embeddings, indexing, and retrieval pipelines tuned for accuracy.
• Take an open-ended data problem with no clean source and engineer a reliable, governed path to usable data.
• Shape genuinely architectural decisions with the Technical Architect, contributing the patterns that become standard.2- Data Engineering
• Build production-grade ingestion, ETL and ELT, and transformation pipelines across batch and streaming.
• Design warehousing and lakehouse structures, schemas, and ontologies that hold up as engagements grow.
• Stand up and tune vector stores and retrieval pipelines that feed RAG and agentic workflows.
• Own data quality, governance, and lineage, the difference between an agent you can trust and one you cannot.
• Engineer for performance and cost, so the foundation scales with the client without surprises.3- Delivering in Client Engagements
• Own your work end to end within live client engagements, from a raw, messy data starting point to a governed system running in production.
• Bring the deep data engineering expertise a client engagement depends on, as a peer to the Technical Architects, Forward Deployed Engineers, and other engineers in the POD.
• Maintain the documentation and handoff discipline that distributed, multi-client delivery requires, so your work is easy to pick up, extend, and reuse.
• Be available for the hard, real-time problems that need a working session, not a write-up.4- Building Reusable Capability
• Turn everything you solve into reusable capability: pipelines, connectors, ontology fragments, and data-quality frameworks that make the next engagement faster.
• Partner with the Technical Architect to abstract, validate, and document what you build for the IP library.
• Actively drive Reuse Index. The data foundations you create are how the next project starts ahead of where the last one finished.
• Treat capability-building as core, not overhead. Every hard problem solved should leave the Factory stronger.

What This Role Is Not

• Not a BI or reporting role. You build the data foundations that production agentic systems run on, not dashboards and reports.
• Not a one-off pipeline scripter. You build reusable, governed data capability that outlives a single engagement.
• Not an internal platform team. You build the systems clients pay for, not internal tooling for its own sake.
• Not a Forward Deployed Engineer. The FDE leads the engagement on the ground at the client; you build the data foundation the engagement is built on. Different seats, both core.
• Not the Technical Architect. The TA sets architecture and standards across engagements; you build the systems within them and shape the patterns that become standards.

Qualifications

Required: Strongly Preferred

• 8+ years of hands-on data engineering in production, with a track record of building reliable pipelines over messy, real-world data.
• Strong with ingestion, ETL and ELT, transformation, and warehousing or lakehouse design.
• Fluent in SQL and Python. Experience building the data foundations behind ML or AI systems, including embeddings, vector stores, and retrieval. Rigorous about data quality, governance, and lineage.
• A clear written communicator who collaborates well across a distributed team.
• Depth with streaming, modern transformation and orchestration tooling (dbt, Spark, Airflow), and ontology or knowledge-graph design.
• Experience building retrieval for RAG and agentic systems. Experience in a consulting, professional services, or client-delivery environment.
• Familiarity with the data, access, and compliance constraints of regulated clients in financial services or healthcare.

Tools & Technologies

Languages- Expert Level: SQL, Python

Pipelines & Orchestration- Advanced Level: Airflow, dbt, Spark

Warehousing & Lakehouse - Advanced Level: Snowflake, BigQuery, Databricks

Retrieval & Vector- Advanced Level: Embeddings, vector stores (pgvector, Pinecone)

Streaming- Working Knowledge: Kafka, Kinesis

Governance & Quality - Advanced Level: Data quality, lineage, cataloging

Cloud Data Platforms- Advanced Level: AWS, Azure, GCP data services

AI & Agentic- Advanced Level: RAG and retrieval for agents, embeddings pipelines, agentic data workflows

CORE COMPETINCES:

Data Architecture

Pipeline Engineering

Retrieval & RAG Data

Data Quality

Client Impact

IP Mindset

Delivery Ownership

Problem-Solving