LocationGreater London, South East, England
Salary (Per Annum)£600 - £650 per day, Benefits: N/A
We are looking for an experienced Data Ingestion Engineer to join a cutting-edge AI and robotics organisation, working on the large-scale data infrastructure that underpins the training and continuous improvement of advanced AI models.
This is a highly technical, hands-on engineering role focused on ingesting and processing huge volumes of real-world driving, vehicle and sensor data from automotive manufacturers, fleet operators, dashcam providers and other external data sources.
This is not a generalist cloud data engineering role. The successful candidate must have strong hands-on Python, PySpark/Spark and SQL expertise combined with demonstrable experience working with robotics or automotive data, including technologies and datasets associated with LiDAR, SLAM, ADAS, cameras, vehicle sensors and autonomous systems.
We are looking for engineers who have personally built or operated pipelines processing terabyte-scale data volumes per day and can clearly quantify the largest ingestion and data-processing environments they have worked with.
Key Responsibilities
This is a highly technical, hands-on engineering role focused on ingesting and processing huge volumes of real-world driving, vehicle and sensor data from automotive manufacturers, fleet operators, dashcam providers and other external data sources.
This is not a generalist cloud data engineering role. The successful candidate must have strong hands-on Python, PySpark/Spark and SQL expertise combined with demonstrable experience working with robotics or automotive data, including technologies and datasets associated with LiDAR, SLAM, ADAS, cameras, vehicle sensors and autonomous systems.
We are looking for engineers who have personally built or operated pipelines processing terabyte-scale data volumes per day and can clearly quantify the largest ingestion and data-processing environments they have worked with.
Key Responsibilities
- Design, build and operate high-throughput data ingestion pipelines processing terabyte-scale datasets
- Ingest real-world driving, vehicle, camera, sensor and robotics data from multiple external providers
- Process and transform datasets associated with ADAS, SLAM, LiDAR and autonomous vehicle/robotics systems
- Build scalable ETL/ELT processes for collecting, aggregating, transforming and moving data across complex environments
- Develop production data-processing solutions using Python, Spark/PySpark and SQL
- Debug and resolve failing or blocked ingestion pipelines
- Investigate corrupt, malformed, incomplete and inconsistent datasets without allowing individual failures to stall wider processing workflows
- Improve pipeline resilience through validation, retries, queue management, observability and automated recovery
- Strong experience building and operating large-scale production data ingestion, ETL or data-processing platforms
- Demonstrable experience processing terabyte-scale data volumes per day
- Strong hands-on programming experience with Spark/PySpark, Python and SQL
- Strong experience with distributed data processing and technologies such as Spark and Databricks
- Experience with data engineering technologies such as DLT, dbt, Dagster, Airflow, Kafka and AWS Glue
- Experience working with large-scale cloud data platforms and storage technologies including S3 and Apache Iceberg
- Experience with databases and data warehouses such as Snowflake, Redshift, PostgreSQL and Oracle
- Previous experience within robotics, autonomous vehicles, ADAS, automotive AI or another real-world AI/ML environmentExperience processing vehicle, camera, sensor, telemetry or robotics datasets at significant scaleExperience working with data supplied by automotive OEMs, fleet operators or other large external data providers