LocationGreater London, South East, England
Salary (Per Annum)£600 - £650 per day, Benefits: N/A
We are looking for an experienced Data Ingestion Engineer to join a cutting-edge AI and robotics organisation, working on the large-scale data infrastructure that underpins the training and continuous improvement of advanced AI models.
This is a highly technical, hands-on engineering role focused on ingesting and processing huge volumes of real-world driving, vehicle and sensor data from automotive manufacturers, fleet operators, dashcam providers and other external data sources.
Scale is critical. We are looking for engineers who have personally built or operated pipelines processing terabyte-scale data volumes per day and can clearly quantify the largest ingestion and data-processing environments they have worked with.
The ability to ingest new datasets quickly and reliably has a direct impact on how rapidly AI models can be retrained, evaluated and improved.
Key Responsibilities
This is a highly technical, hands-on engineering role focused on ingesting and processing huge volumes of real-world driving, vehicle and sensor data from automotive manufacturers, fleet operators, dashcam providers and other external data sources.
Scale is critical. We are looking for engineers who have personally built or operated pipelines processing terabyte-scale data volumes per day and can clearly quantify the largest ingestion and data-processing environments they have worked with.
The ability to ingest new datasets quickly and reliably has a direct impact on how rapidly AI models can be retrained, evaluated and improved.
Key Responsibilities
- Design, build and operate high-throughput data ingestion pipelines processing terabyte-scale datasets
- Ingest real-world driving, vehicle, camera, sensor and robotics data from multiple external providers
- Build scalable ETL/ELT processes for collecting, aggregating, transforming and moving data across complex environments
- Develop production data-processing solutions using Python, Spark/PySpark and SQL
- Debug and resolve failing or blocked ingestion pipelines
- Investigate corrupt, malformed, incomplete and inconsistent datasets without allowing individual failures to stall wider processing workflows
- Improve pipeline resilience through validation, retries, queue management, observability and automated recovery
- Optimise distributed processing workloads for throughput, compute efficiency and reliability
- Orchestrate complex multi-stage ingestion workflows and downstream dependencies
- Integrate new data providers rapidly, enabling datasets to become available for AI/ML model training within demanding timescales
- Explore AI-assisted and self-healing pipeline capabilities to automate the diagnosis and remediation of common ingestion failures
- Build monitoring and observability using technologies such as Prometheus and Grafana
- Deploy data-processing services using modern containerisation, cloud and Infrastructure-as-Code practices
- Work closely with Data, AI, ML and Robotics Engineers to increase the speed at which real-world data can be used to improve AI models
- Strong experience building and operating large-scale production data ingestion, ETL or data-processing platforms
- Demonstrable experience processing terabyte-scale data volumes per day
- Strong hands-on programming experience with Spark/PySpark, Python and SQL
- Experience with additional languages such as Scala, Java and TypeScript/JavaScript would be beneficial
- Strong experience with distributed data processing and technologies such as Spark and Databricks
- Experience with data engineering technologies such as DLT, dbt, Dagster, Airflow, Kafka and AWS Glue
- Experience working with large-scale cloud data platforms and storage technologies including S3 and Apache Iceberg
- Experience with databases and data warehouses such as Snowflake, Redshift, PostgreSQL and Oracle
- Experience working across AWS and/or Azure, ideally alongside on-premises environments
- Strong understanding of workflow orchestration, dependencies, retries, queue management and failure recovery
- Proven experience debugging production pipeline failures and dealing with messy, corrupt or unexpected source data
- Experience deploying production workloads using Docker and Kubernetes, including platforms such as AWS EKS/ECS
- Infrastructure-as-Code experience using Terraform and/or CloudFormation
- Experience developing and operating CI/CD pipelines
- Experience with production monitoring and observability, ideally using Prometheus and Grafana
- Previous experience within robotics, autonomous vehicles, ADAS, automotive AI or another real-world AI/ML environment
- Experience processing vehicle, camera, sensor, telemetry or robotics datasets at significant scale
- Experience working with data supplied by automotive OEMs, fleet operators or other large external data providers
- Understanding of how ingestion infrastructure feeds AI/ML training, retraining and evaluation pipelines
- Experience with Flyte or comparable data/ML workflow platforms
- Experience applying AI or agentic tooling to pipeline monitoring, diagnosis and automated remediation
- Experience with technologies such as Kyubi/Kyuubi and modern lakehouse architectures
- Data analysis experience using libraries such as Pandas, NumPy, Matplotlib or Plotly