Distributed Training Engineer

Location: Noida or Bengaluru

About the Role

Our training infrastructure runs distributed jobs across many GPUs at once, and it has to just work no lost checkpoints, no silent slowdowns, no surprises at 2am. As a Training Runtime Engineer, you'll build and harden the distributed training infrastructure our model development depends on, working on some of the hardest reliability and performance problems in the stack.

What You'll Do

• Build and maintain distributed training infrastructure, including tensor, pipeline, data, and expert parallelism and FSDP
• Implement and harden distributed checkpointing and fault-tolerant recovery for long-running training jobs
• Optimize NCCL communication patterns and diagnose distributed-training performance issues
• Support multi-tenant job scheduling and isolation for concurrent training runs
• Partner with our research team to translate new training recipes into runtime requirements

What You'll Bring

• 5+ years of experience in distributed systems or ML infrastructure engineering
• Strong experience with distributed training frameworks (PyTorch, Megatron-LM, DeepSpeed, or similar)
• Hands-on experience with CUDA, NCCL, and RDMA-based communication
• Solid grasp of distributed checkpointing and fault-tolerant systems design
• Proficiency in C++ and Python

Education

B.Tech in Computer Science or a related field; M.Tech a plus for senior scope.

Nice to Have

• Experience at a distributed-database, HPC, or cloud infrastructure company — the reliability and consistency challenges transfer directly
• Prior work on a multi-tenant training or scheduling system