Data Engineer

Data Engineer Salary: Up to £80,000 FTE Location: Home-based, with occasional travel to meetings as required Hours: 40 hours per week Reporting to: Lead Data Engineer The Opportunity We are working with a leading UK-based online mental healthcare provider that is looking to appoint an experienced Data Engineer to join its growing Data, Analytics & Reporting team. This is an exciting opportunity to join an organisation using innovative technology and AI-enabled platforms to improve access to high-quality mental healthcare across the UK. As Data Engineer, you will play a key role in developing, maintaining and optimising the organisation's data infrastructure, ensuring reliable, secure and high-quality data is available to support analytics, reporting and data science. Working closely with the Lead Data Engineer, Data Scientists, Analysts and wider clinical and operational teams, you will help transform data from multiple clinical, operational and third-party systems into trusted, well-governed datasets. This is a hands-on engineering role, offering the opportunity to contribute to the development of a modern Databricks Analytics Platform and help shape the organisation's wider data capabilities. Key Responsibilities Design, build, test and optimise robust ETL/ELT data pipelines within a Databricks Analytics Platform hosted on AWS. Develop and maintain data ingestion processes from clinical, operational, financial and external systems, including relational databases, REST APIs and file-based sources. Build and optimise SQL and PySpark transformations across Bronze, Silver and Gold data layers. Work with technologies including Databricks, Delta Lake, Lakeflow, dbt and Unity Catalog. Implement automated data quality checks, monitoring and alerting to ensure data accuracy, completeness and reliability. Manage and orchestrate workloads, troubleshoot pipeline failures and support production incident resolution. Support deployment processes across development, staging and production environments using Git, CI/CD and Infrastructure as Code principles. Apply data governance and security controls, including access permissions, data masking, pseudonymisation and data lineage. Monitor and optimise platform performance, resource usage and cloud infrastructure costs. Collaborate with Data Scientists to provide high-quality datasets for machine learning, NLP and predictive analytics. Support feature engineering, model training and inference workflows, including the use of MLflow. Work alongside Analysts to develop Gold-layer dimensional models and governed metrics for Power BI, Databricks AI/BI dashboards and Genie. Maintain clear technical documentation covering data models, schemas, transformations and data flows. Promote best practice in data engineering through code reviews, knowledge sharing, collaboration and continuous improvement. Ensure all data processes comply with relevant information governance, GDPR and NHS data security requirements. About You We are looking for a technically strong and motivated Data Engineer who has experience building and supporting production data pipelines and enjoys working collaboratively within a growing data team. You will ideally have: Proven experience working as a Data Engineer or in a similar data engineering role. Strong SQL skills, including performance tuning, CTEs, window functions and data modelling. Good Python programming skills, particularly PySpark, for data transformation, API integrations and automation. Hands-on experience with Databricks, including Delta Lake, Spark and Unity Catalog, or a comparable cloud data platform such as Snowflake or AWS Redshift. Experience working with data ingestion techniques, including change data capture (CDC), REST API integrations and incremental loading. Understanding of data modelling approaches, including star schemas, medallion architecture and slowly changing dimensions. Experience with Git, version control and CI/CD pipelines. An understanding of Infrastructure as Code principles, ideally with exposure to Terraform or Declarative Automation Bundles. Knowledge of data governance and security, including access controls, data masking and pseudonymisation. An awareness of data science and machine learning workflows and how data engineering supports these processes. Personal Attributes Alongside your technical expertise, you will demonstrate: Strong analytical and problem-solving abilities. Excellent attention to detail. The ability to manage your own workload and prioritise competing demands. A collaborative approach and the ability to work effectively with technical and non-technical stakeholders. Strong written and verbal communication skills. A proactive attitude towards learning, development and continuous improvement. Why Consider This Opportunity? Salary of up to £80,000 FTE. Fully home-based working arrangement. Opportunity to work with modern cloud data technologies and a Databricks-based analytics platform. Exposure to AI, machine learning and advanced data science projects. The chance to contribute to meaningful projects within the healthcare sector. Opportunity to work alongside an experienced and collaborative Data, Analytics & Reporting team. If you are an experienced Data Engineer looking for an opportunity to work with modern data technologies while contributing to an organisation making a real difference in healthcare, we would love to hear from you.