We are looking for an experienced Senior Data Engineer to join a high-performing data engineering team responsible for large-scale data ingestion and processing.
You'll play a key role in ensuring critical data pipelines remain reliable, scalable and efficient, working with high-volume datasets that support analytics, data science and machine learning initiatives.
This is a hands-on engineering role focused on solving complex production challenges, improving data pipeline resilience and optimising distributed data processing systems.
Key Responsibilities:
This is an Inside IR35 contract role working via Umbrella Company so you must be willing and able to work on this basis to be considered for this role.
You'll play a key role in ensuring critical data pipelines remain reliable, scalable and efficient, working with high-volume datasets that support analytics, data science and machine learning initiatives.
This is a hands-on engineering role focused on solving complex production challenges, improving data pipeline resilience and optimising distributed data processing systems.
Key Responsibilities:
- Build, maintain and optimise large-scale data ingestion and ETL pipelines using Apache Spark and Python.
- Debug and resolve failing or blocked data ingestion pipelines in production.
- Investigate issues caused by corrupt, malformed or unexpected data and implement robust solutions to prevent pipeline failures.
- Improve pipeline resilience so individual data quality issues do not interrupt wider workflows.
- Process and transform large datasets from multiple internal and third-party data sources.
- Design and improve handling of varied data formats from partners, suppliers and external providers.
- Support orchestration across multi-stage ingestion workflows, including dependency management, retries and queue handling.
- Optimise Spark jobs for throughput, reliability and compute efficiency.
- Reduce operational overhead by minimising failed jobs, stalled pipelines and manual intervention.
- Work closely with engineering teams to ensure critical datasets are delivered to downstream analytics, data science and machine learning teams.
- Contribute practical improvements that enhance the scalability, reliability and maintainability of the data platform.
- Strong commercial experience with Apache Spark (PySpark) in production environments.
- Excellent Python development skills.
- Proven experience designing, building and supporting large-scale data ingestion, ETL or data processing pipelines.
- Experience working with distributed data processing technologies.
- Strong understanding of data ingestion, transformation and loading processes.
- Experience diagnosing and resolving production pipeline failures.
- Comfortable working with incomplete, inconsistent or corrupt datasets.
- Experience optimising data processing jobs for performance, scalability and cost efficiency.
- Understanding of workflow orchestration, multi-stage pipelines and downstream dependencies.
- Ability to work independently in a fast-paced, highly technical environment.
- A pragmatic, delivery-focused approach with excellent problem-solving skills.
- Experience working with very large datasets, ideally in petabyte-scale or other high-throughput environments.
- Experience with workflow orchestration tools such as Airflow, Dagster or similar.
- Cloud platform experience (AWS, Azure or GCP).
- Knowledge of modern data lake or data warehouse architectures.
- Experience supporting machine learning or data science platforms.
This is an Inside IR35 contract role working via Umbrella Company so you must be willing and able to work on this basis to be considered for this role.
