Data Engineer (Databricks)

Every day, millions of shipments move around the world on data that nobody is looking at in real time. Orders, ASNs, invoices, and tracking events flow between 3PLs, forwarders, brands, and retailers, then get flattened into a status that tells you something went wrong only after it already has.

SwiftB2B exists to change that. We are an AI operating system for global supply chains. One platform where 3PLs, freight forwarders, brands, retailers, and manufacturers can see every item, shipment, and exception as it happens. The platform does not stop at flagging a delay. It works out what the delay costs, weighs the options, and starts the fix.

We built it on 15 years of SwiftAnt’s work in supply chain and EDI technology, and on a team that has spent careers in logistics, retail, and enterprise software. We are a Databricks partner and our data platform runs on the lakehouse.

Most of our pipelines pull from systems that were never built to share data seamlessly: a 3PL’s WMS, a forwarder’s TMS, EDI files from hundreds of trading partners. We are looking for someone who can get that data into Databricks and keep it flowing. When a feed stops at 3am, or a partner changes a field format without telling anyone, the job is to have a check that catches it before a customer sees a wrong number. We are a Databricks partner, so we want depth there rather than a bit of everything. And we care about the boring parts, like agreeing the grain of a table before building it and writing the runbook for whoever is on call, because that is what keeps the platform running seamlessly.

What you will do

• Build the pipelines that make our supply chain KPI catalogue real: ingestion from TMS, WMS, ERP, carrier APIs, and EDI feeds (850, 856, 214, 210 and friends) into governed, tested Delta tables that the semantic layer compiles onto.
• Design medallion models on Databricks with the grain agreed up front with our analytics engineers, not discovered after the dashboard is wrong.
• Make pipelines idempotent and re-runnable, with Lakeflow or Workflows orchestration, CDC and incremental patterns, and data-quality checks that fail loudly rather than quietly passing bad rows.
• Own freshness and quality SLAs, and write the runbook another engineer can execute at 2am without calling you.
• Prove row-level security and governance in Unity Catalog by test, not by assertion.
• Ship through a release path with no manual step: CI/CD for notebooks, jobs, and infrastructure, with contract tests on the way out.
• Rehearse cutovers, including the rollback, before any customer migration goes live.
• Use AI to generate transformations quickly, then reconcile every field against live data before it ships. The reconciliation is the evidence; the generation is not.
• Contribute improvements back to our accelerators after every engagement, so the next customer onboarding is faster than the last.

What we are looking for

• Three or more years as a data engineer with production pipelines that other people depended on.
• Deep Databricks: PySpark, Delta Lake, medallion architecture, Unity Catalog, Workflows or Lakeflow, Delta Live Tables. Databricks Certified Data Engineer Associate is a strong plus; we will sponsor it if you do not have it yet.
• Strong Python and advanced SQL, including query optimisation on large tables.
• Hands-on with orchestration and ELT patterns: CDC, incremental loads, late-arriving data, scheduling, retries.
• Data-quality and testing habits: contract tests, expectations, freshness checks, alerting that catches a silent break.
• Azure experience: ADLS, Key Vault, Azure DevOps or GitHub Actions, infrastructure as code. DP-203 or AZ-400 is nice to have.
• dbt experience is a plus. Kafka or Event Hubs experience is a plus.
• A background in supply chain, logistics, or retail data is a strong advantage. Knowing what a policy, a claim, or a TMS milestone actually is lets you tell a wrong extract from an odd one.
• A degree in Computer Science, Data Engineering, or a related field, or equivalent practical experience.

Not sure you tick every box?

Apply anyway. Nobody we have hired matched the list line for line. If you have shipped pipelines on Databricks that are still running without you, we want to talk to you. Tell us about the quietest failure you ever caught and how your check found it. That tells us more than a polished CV.

Equal opportunity

SwiftB2B hires on skill and curiosity. We do not discriminate on the basis of race, religion, nationality, gender, sexual orientation, age, disability, or any other protected characteristic. If you need an adjustment at any stage of the process, tell us and we will make it happen.