Databricks Lead Engineer

NEW CONTRACT ROLE – DATABRICKS LEAD ENGINEERA new opportunity has just landed for an experienced, hands-on Databricks Lead Engineer to join a major data transformation programme.We’re looking for someone with strong recent experience delivering modern, production-grade data platforms on Databricks. This is a hands-on technical leadership role, so we're looking for someone who can lead engineering delivery and design solutions, rather than simply work from an existing blueprint.Location: London – 3 days per week on-siteRate: Up to £575 per dayContract: Inside IR35Duration: 6 months initial contract with potential extensionStart: ASAPKey requirements:• 2+ years of recent, hands-on Databricks experience delivering production solutions• Active/current Databricks certification – Data Engineer, Machine Learning or Generative AI• Strong experience designing and implementing Databricks Lakehouse architectures• Hands-on implementation of Medallion Architecture – Bronze, Silver and Gold layers• Experience with Unity Catalog, including governance, access control, lineage and secure data sharing• Strong experience building Declarative Pipelines / DLT, including Expectations, data quality, failure handling and observability• Experience with Databricks-native ingestion approaches such as AutoLoader and/or Lakeflow Connect• Strong PySpark and SQL skills• Delta Lake experience, including MERGE, schema evolution, optimisation and performance tuning• Experience with Databricks Workflows and production pipeline orchestration• Strong understanding of CI/CD, testing, monitoring, alerting, incident triage and operational best practice• Ability to mentor engineers and act as the technical escalation point within the delivery teamNice-to-have experience includes:• Mosaic AI, Vector Search, model serving or RAG pipelines• Lakeflow ingestion connectors• Row-level security, column masking and sensitive data handling through Unity Catalog• Databricks performance optimisation, including Photon, cluster sizing, partitioning and shuffle/skew mitigation• Cost optimisation and understanding of Databricks compute and DBU driversThis is a great opportunity for a technically strong Databricks specialist who wants to take ownership of engineering delivery and help shape a modern enterprise data platform.If you're interested, or know someone who could be a good fit, please get in touch.#Databricks #DataEngineering #DataEngineer #Lakehouse #PySpark #DeltaLake #UnityCatalog #DataJobs #ContractJobs #LondonTech #InsideIR35