Senior Data Engineer / AI Engineer

Job Summary:

• Client is seeking a Senior Data Engineer / AI Engineer with deep expertise in data engineering, AI-driven development, and agentic workflow automation using Claude and Claude Code.
• This role will focus on transforming ingestion and intake processes from multi-week timelines to less than 24 hours through the design and implementation of sophisticated Data Engineering and AI-powered workflows.
• The ideal candidate is not simply an AI engineer, but a senior-level data engineering professional who understands how to build, orchestrate, and deploy enterprise-scale data pipelines while leveraging Claude-based agentic systems to automate complex business processes.
• Experience building harnesses, skills, specifications, and agent orchestration frameworks is highly desired.

Responsibilities:

• Build, deploy, monitor, and optimize enterprise data pipelines.
• Design, develop, and deploy advanced agentic workflows using Claude and Claude Code.
• Build and optimize solutions that reduce ingestion and intake timelines from 5-10 weeks to under 24 hours.
• Develop AI-powered automation frameworks for large-scale, complex workflow orchestration.
• Create and maintain agent harnesses, specifications, and reusable AI skills.
• Design scalable ingestion frameworks supporting high-volume data processing.
• Collaborate with data engineering, platform, and business teams to identify automation opportunities.
• Integrate Claude-based workflows into existing enterprise platforms and services.
• Ensure reliability, governance, and scalability of AI-driven systems.

Experience:

• 6+ years of professional experience in Data Engineering, or related fields.
• Databricks and Unity Catalog expertise
• Claude/Claude Code experience
• Azure Cloud/Azure technologies
• Strong experience with Spark
• Proven experience building and orchestrating enterprise-scale data pipelines.
• Hands-on experience with agentic workflow design and execution.

AI & Agentic Development

• Demonstrated experience using Claude and Claude Code in production environments.
• Ability to clearly explain real-world projects built using Claude Code.Experience developing:
• Agent frameworks
• Harnesses
• Skills
• Specifications (Spec Development)
• Workflow orchestration systems
• Experience with complex multi-step agent execution patterns.

Data Engineering

Strong expertise in:

• Databricks
• Unity Catalog
• Apache Spark
• PySpark
• Data pipeline development
• Data ingestion and transformation frameworks
• Strong understanding of data architecture, governance, and distributed processing.

Nice to Have:

• Good English communication skills.
• Experience designing AI systems beyond traditional RAG architectures.
• Knowledge of AI workflow orchestration platforms and autonomous agents.
• Background in enterprise-scale automation and operational efficiency initiatives.
• Experience working in Agile product and engineering environments.