LocationGreater London, South East, England
Salary (Per Annum)£600 - £650 per day, Benefits: N/A
Our client is an AI automotive business building the intelligence behind next-generation automated driving. Their vehicles learn to drive from real-world data, not hand-written rules. That makes data ingestion central to how fast their models improve.
This is a hands-on contract role in the Data Ingestion team. You’ll build, debug and optimise the PySpark and Python pipelines that take raw driving data from many sources and turn it into trusted datasets for annotation, data science, model training and evaluation.
Key Responsibilities
This is a hands-on contract role in the Data Ingestion team. You’ll build, debug and optimise the PySpark and Python pipelines that take raw driving data from many sources and turn it into trusted datasets for annotation, data science, model training and evaluation.
Key Responsibilities
- Data ingestion: Build, run and harden high-volume batch ingestion pipelines. Diagnose and fix failing or blocked jobs, and make pipelines resilient so one bad segment can’t block wider workflows.
- PySpark: Optimise Spark jobs for throughput, compute efficiency and cost across petabyte-scale datasets.
- Python engineering: Write clean, production-grade Python for ingestion logic, validation, tooling and automation that reduces manual intervention.
- Complex data domains: Work with varied data formats from partners and suppliers. Investigate corrupt, malformed or unexpected data and build robust handling for it.
- Orchestration: Support multi-step workflows, including dependencies, retries, queue management and prioritisation of critical datasets.
- Collaboration: Work alongside permanent engineers to unblock important data flows while longer-term platform improvements are built.
- Strong production experience with PySpark / Apache Spark
- Strong Python engineering skills
- Proven experience building, debugging or operating large-scale data ingestion or ETL pipelines
- Experience with very large datasets, ideally at PB scale or in similarly high-throughput environments
- Comfort with messy, corrupt, incomplete or inconsistent data across complex data domains
- Experience with distributed data processing and debugging production pipeline failures
- Understanding of orchestration across multi-step pipelines and downstream dependencies
- A practical, delivery-focused mindset and the ability to ramp up quickly with minimal hand-holding