Lead Data Engineer

Lead Data Engineer   
Hybrid - Attached to our Solihull, London or Hook office  
Full Time, Permanent   
Band 3 / £75,000 - £85,000 (dependent on experience)

Here at Serco, we're looking for a Lead Data Engineer to act as the technical authority for data engineering, driving architecture, standards, and delivery across a modern multi-cloud ecosystem spanning Azure, AWS, and Databricks.

You'll spend around 80% of your time hands-on, designing and building complex, enterprise-scale data solutions. The remaining 20% is just as critical: leading people, influencing stakeholders, and shaping how data engineering operates across the organisation.

Databricks sits at the heart of our platform - this role requires someone who can deeply leverage the Databricks to enable scalable analytics, data science, and AI use cases across the business.

As part of this you'll:

        Define and enforce data engineering standards, patterns, and best practices across the organisation.

        Architect and deliver scalable, resilient data solutions across Azure, AWS, and Databricks, ensuring enterprise-grade performance and reliability.

        Lead complex engineering workstreams - unblocking teams, guiding technical decisions, and ensuring high-quality delivery outcomes.

        Design and govern enterprise data models (conceptual, logical, and physical), including lakehouse zone strategies and schema evolution approaches.

        Champion Databricks as the core data platform, leveraging lakehouse architecture, Unity Catalog, Delta Live Tables, Structured Streaming, and MLflow.

        Drive a consistent multi-cloud Databricks strategy, enabling scalable and interoperable data platforms across environments.

        Optimise performance across pipelines, compute, storage, and orchestration frameworks, embedding cost-efficiency and scalability.

        Define and implement CI/CD for data platforms, supporting automated testing, secure deployments, and multi-environment promotion.

        Embed robust data governance, security, and lifecycle controls, ensuring compliance through classification, encryption, access management, and lineage frameworks (e.g., Unity Catalog, Purview, Lake Formation).

        Act as a senior SME for troubleshooting complex issues, leading root-cause analysis and long-term remediation of data quality and platform challenges.

        Lead, mentor, and grow a community of Data Engineers, driving technical excellence through coaching, reviews, and knowledge sharing.

        Operate effectively across a matrix organisation, influencing architecture, cyber, product, and senior stakeholders.

        Communicate complex technical concepts clearly, articulating trade-offs, risks, and strategic decisions to both technical and non-technical audiences.

What you'll need to do the role:

        Expert-level experience in enterprise-scale data engineering, including defining standards, patterns, and overseeing complex delivery.

        Advanced proficiency in PySpark and SQL, with deep expertise in performance optimisation and scalable data processing.

        Strong experience in data modelling and large-scale data integration design, with the ability to build robust, maintainable, and testable pipelines.

        Proven ability to embed testing frameworks, observability, and data quality controls across engineering workflows.

        Deep hands-on experience with Databricks (essential), including lakehouse architecture, Delta Lake, Unity Catalog, and streaming technologies.

        Broad cloud expertise across Azure (Synapse, ADF, Event Hub, ADLS Gen2, Purview, Key Vault) and AWS (S3, Glue, EMR, Kinesis, Athena, Lake Formation, IAM), with strong multi-cloud experience.

        Strong understanding of modern engineering practices, including CI/CD for data platforms, automated deployment pipelines, and environment promotion strategies.

        Experience designing monitoring, observability, and cost optimisation frameworks, alongside governance, lineage, cataloguing, and compliance at scale.

        Excellent stakeholder management skills, with the ability to influence senior leaders and work effectively across matrixed organisations.

        Proven leadership capability, including mentoring, coaching, and developing engineering teams.

        Strong communication skills, with the ability to translate complex technical concepts into clear business value.

        8+ years' experience in data engineering within enterprise environments.

        Relevant certifications (desirable), such as Azure Data Engineer Associate, AWS Data Analytics Specialty, or Databricks Data Engineer (Associate/Professional).

        SC clearance (or willingness to obtain it)

What we offer:

        Company car

        5% on target bonus scheme

        Private medical insurance

        Flexible working considered

        Pension - 6%

        Chance to contribute to innovation in the public services

        A company passionate about diversity and inclusion

        Serco discounts which include cinema, merlin entertainment and online shopping discounts, and discounts on mobile phone plans and leisure centre memberships.

        A range of benefits to support the health and wellbeing of you and your family such as Employee Assistance Programme, Simply Health Cash Plans, and more.

        A wealth of career development training to suit your future aspirations. These range from role specific training, leadership coaching, formal study and much more to support you to build your career with Serco.

        A safe and supportive culture.