Senior Data Engineer

Job summary

The Data and Analysis for Delivery (DA4D) division, part of Delivery and Group Corporate Services in Defra, plays a pivotal role in enabling data-driven decision-making across the department’s major delivery portfolio. Within DA4D, the Delivery Data team is responsible for the collection, management, and exploitation of data relating to the major delivery portfolio, performance and risk. By ensuring accurate, timely, and accessible data, the team supports Defra in understanding delivery progress and making informed strategic decisions. Its work underpins the department’s ability to manage complex programmes effectively and achieve outcomes that benefit the departments core purpose.

You will play a key role in driving improvements in how Defra collects, manages, and harnesses data at enterprise level.

Working closely with delivery teams, you will help identify, manage, and unlock the value of critical data assets, enabling evidence-based decision-making and more efficient operations.

Your day-to-day work will be varied and impactful. One day you may be designing and scoping robust data architectures or modern storage solutions to enhance data quality, integrity, and accessibility across the organisation. On another, you might be developing and optimising data pipelines that deliver trusted, high-quality data to data scientists, analysts, and self-service users, supporting their need for timely and accurate insights.

Collaboration will be at the heart of your role. You will work with a wide range of colleagues from across Defra, its arm’s length bodies, and external partners, optimising data flows, integrating with multiple data collection platforms, and ensuring data solutions are secure, scalable, and fit for purpose. Through your work, you will help shape and deliver Defra’s ambition to become a data-driven organisation.

Job description

The team has recently delivered a critical milestone: an end-to-end Delivery Data Service, including a platform for collecting data, pipelines for ingesting and cleaning data, and a data science hub for analysis, visualisation, and interpretation. This foundation enables key services such as Portfolio Management, Enterprise Performance, and Enterprise Risk.

Building on this, you will lead data engineering workstreams that enhance and scale our data architecture, optimise data flows, and deliver new pipelines to bring critical datasets into the team. Your focus will be on designing and implementing robust, cloud-based solutions that ensure data is accurate, accessible, and performant across the organisation. You’ll work with technologies such as Azure Synapse, Databricks, Python, and SQL to develop ETL processes, improve data quality, and enable downstream analytics and reporting.

Collaboration will be central to your role. You’ll partner with colleagues across Defra group to integrate multiple data sources, streamline ingestion processes, and ensure solutions are secure, scalable, and aligned with best practices. This is your opportunity to shape how Defra manages and exploits data at enterprise scale, supporting one of the most complex delivery portfolios in government.

You’ll thrive in a fast-paced, evolving environment - seeing opportunity in ambiguity, adapting quickly, and learning openly. As part of a small, ambitious, and friendly team, you’ll combine technical expertise with creativity to deliver meaningful change.

Key Responsibilities:

• Design, build, and maintain cloud-based data platforms and architectures that integrate large, complex, and disparate datasets, ensuring scalability, security, and performance.
• Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use.
• Implement robust data models and storage solutions that support efficient querying, data integrity, and enterprise-wide accessibility.
• Collaborate with stakeholders across Defra group to understand data requirements, integrate multiple sources, and deliver solutions that meet business needs.
• Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking to maintain trusted datasets.
• Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement within the team.
• Lead technical workstreams and mentor colleagues, fostering a culture of collaboration, knowledge sharing, and engineering excellence.

Person specification

The ideal candidate will bring deep expertise in cloud-based data solutions, with hands-on experience developing and managing data platforms on technologies such as Azure and Databricks. They will excel at identifying opportunities to reuse and enhance existing workflows, optimising code to drive performance, scalability, and resilience. Equally comfortable engaging with both technical and non-technical stakeholders, they will act as a bridge between engineering teams, data consumers, and business leaders, ensuring data solutions are aligned with organisational goals.

Technical Skills

Essential

• Proven experience designing and building cloud-based data platforms that integrate large, complex, and disparate datasets - ideally using Databricks or similar technologies.
• Advanced programming skills in Python (including PySpark, Pandas) and SQL for data engineering and analytics workloads.
• Strong data modelling expertise, with a deep understanding of industry standards, best practices, and performance optimisation techniques.
• Ability to lead the development of a portfolio of ETL pipelines, across a range of complex and/or large volume data.
• Skilled at defining and implementing robust test conditions and validation frameworks based on business and technical requirements.

Non-technical Skills

Essential

• Outstanding communicator, capable of translating complex technical concepts into clear, actionable insights for technical teams and non-technical stakeholders alike.
• Proactive and forward-looking, with a passion for horizon scanning and keeping up to date with emerging data tools, platforms, and best practices.
• Strong problem-solving abilities — able to investigate complex data challenges, identify root causes, engage the right expertise, and deliver effective, sustainable solutions.
• Strong team player with excellent collaboration skills, able to work effectively with colleagues to achieve shared goals.

Behaviours

We'll assess you against these behaviours during the selection process:

• Changing and Improving
• Communicating and Influencing
• Making Effective Decisions

We only ask for evidence of these behaviours on your application form:

• Changing and Improving

Technical skills

We'll assess you against these technical skills during the selection process:

• The technical assessment will include a range of questions and practical exercises designed to assess the technical skills, knowledge and problem-solving ability required for the role.

Benefits

Alongside your salary of £42,665, Department for Environment, Food and Rural Affairs contributes £12,360 towards you being a member of the Civil Service Defined Benefit Pension scheme.