Senior Data Engineer (4644)

Job summary

Join us at Social Security Scotland as a Senior Data Engineer and be at the forefront of our ambitious digital transformation. This role offers an exciting opportunity to work with a talented team and turn data into a driving force for positive change, helping us create a system that supports the people of Scotland for decades to come.

The Digital, Delivery & Change division is responsible for delivering the infrastructure, capability and digital solutions that will support Social Security Scotland. As a Senior Data Engineer within the division, you will play a crucial role in developing robust data infrastructure and systems that underpin our diverse digital initiatives. Collaborating with a growing team of digital experts across Engineering & Cloud Services, Architecture, Applications, and more, you will design and implement data flows that facilitate analytics and business intelligence.

Your expertise will ensure the delivery of high-quality data products that meet the needs of our stakeholders and enable effective decision-making. We are committed to fostering a positive and inclusive culture that empowers all team members to thrive, making this an exciting opportunity to contribute to meaningful societal change.

A Senior Data Engineer designs and leads the implementation of data flows to connect operational systems, data for analytics and business intelligence (BI) systems.

At this role level, you will:

• Recognise opportunities to reuse existing data flows.
• Lead the build of data streaming systems.
• Optimise the code to ensure processes perform optimally.

GDD Pay Supplement
This post is part of the Government Digital and Data (GDD) profession and currently attracts a £4,000 annual GDD pay supplement, which is paid monthly. Pay supplements are reviewed regularly.

Job description

• Design, develop and support real-time data streaming solutions using Apache Kafka to deliver scalable, reliable data integration.
  
• Design and implement scalable Python-based solutions for data processing, integration and automation.
  
• Lead technical discussions during sprint planning, story refinement and other agile ceremonies, working closely with stakeholders.
  
• Analyse, profile and assess data from multiple complex source systems to understand structure, quality and business value.
  
• Design, build and maintain complex, large-scale data products and integration services.
  
• Consolidate data from multiple sources into conformed models that support analysis, reporting and decision-making.
  
• Create and maintain data models, including relational, dimensional, graph, metadata and subject-area models.
  
• Select and implement appropriate technologies from the approved portfolio to deliver scalable, secure and future-ready data solutions.
  
• Develop, test, document and maintain complex programs, scripts and data pipelines in line with agreed standards, specifications and user requirements.
  
• Perform data transformation, cleansing and normalisation to ensure data is fit for analytical and reporting purposes.
  
• Lead the development and adoption of reusable design patterns, standards and best practices.
  
• Present data insights and technical findings clearly to colleagues and stakeholders to support informed decision-making.
  
• Monitor databases, data products and services, proactively identifying issues, trends and risks, and implementing corrective and preventative actions.
  
• Review requirements and specifications, define test conditions, analyse test results and produce synthetic test data where required.
  
• Build capability within teams by providing technical leadership, mentoring and support across data integration activities.
  
• Proactively identify opportunities to maximise value from data and drive innovation across services and products.
  
• Maintain and apply up-to-date knowledge of relevant technologies, frameworks and industry best practices.
  
• Contribute to the wider Government Digital & Data community through knowledge sharing, technical peer support, code reviews, pair programming and specialist consultancy.

Person specification

Success Profiles
We use an assessment framework called ‘Success Profiles’ which lists the elements we test and provides detailed descriptions of each. Find out more about the framework here.

For this post, the following Success Profile elements will be assessed:

Experience

1. Experience of designing, developing, testing, documenting and maintaining automated, repeatable data solutions using agreed standards, tools and best practices.
   
2. Experience of monitoring, troubleshooting and resolving issues across data, data processes, data products and services, implementing preventative measures where appropriate.
   

Behaviours

• Leadership (Level 4)
• Working Together (Level 4)
  

You can find out more about Success Profiles Behaviours here.

Technical skills

This role is aligned to Senior Data Engineer within the Government Digital and Data Profession.

These skills will be tested during the Technical Assessment if you are successful at sift stage. They will not be assessed at application stage.

Please review the following to understand the skill expectations: Data engineer - Government Digital and Data Profession Capability Framework

Benefits

Annual Leave - You will receive 25 days annual leave on joining us. This will increase to 30 days after four full years of service. You will also have 11.5 public and privilege days of leave every year. We also offer Flexi-time. Any extra hours you've worked can be taken as leave when suitable.

A Civil Service Pension - This job comes with a Civil Service pension. New joiners to the Civil Service will join a career average pension scheme as standard. Read more here - www.civilservicepensionscheme.org.uk.

Healthy work life balance - We can offer the possibility of full-time, part-time, term-time, and job shares. We also encourage flexible working.

Discounts - You can enjoy a vast range of retail, travel and lifestyle discounts through our benefit scheme.

Personal support for you - Our Employee Assistance Programme gives you confidential, independent information and guidance 24/7.

Volunteering special leave - Up to six days paid special leave a year for volunteering. We support our staff to help causes important to them.

Great locations - Our bright and modern offices in the heart of Dundee and Glasgow have been designed with staff in mind. Both locations are ideal for public transport.

Things you need to know

Artificial intelligence

Artificial intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance (ope