Senior Data Engineer

London | Hybrid (2–3 Days in Office)
Full-Time

The Opportunity

A growing and well-funded software organisation is seeking a Senior Data Engineer to help design, build and secure its modern data platform.

The business delivers technology solutions that support public sector and regulated organisations in improving operational efficiency and compliance. Following significant investment and expansion, the company is focused on strengthening its data infrastructure and scaling advanced analytics capabilities.

This is an opportunity to play a key role in shaping a scalable, secure and high-performing data environment, working closely with engineering, analytics and product teams.

The Role

You will design, implement and secure a modern cloud-based data platform (Databricks, pipelines and governance frameworks), ensuring that data is:

• Accurate
• Accessible
• Secure and compliant
• Cost-effective at scale

This is a hands-on engineering role with architectural input, suited to someone comfortable building platforms from the ground up in production environments handling large-scale datasets.

Key Responsibilities

Data Platform & Engineering

• Architect and build end-to-end data pipelines, data lakes and warehouses on Databricks (Azure environment)
• Integrate ETL/ELT tools and cloud services to create scalable data workflows
• Establish standardised processes for:
  • Data modelling
  • Quality assurance
  • Versioning
  • Orchestration (e.g. dbt, Airflow, Unity Catalog)

Security & Governance

• Implement security best practices including:
  • Role-based access control (RBAC)
  • Data encryption and masking
  • Compliance with GDPR and relevant standards
• Support governance frameworks ensuring integrity and compliance at scale

Performance & Optimisation

• Optimise platform performance, cost-efficiency and scalability
• Support real-time and large-scale data processing
• Automate monitoring, alerting and recovery processes
• Drive CI/CD adoption for data pipelines and infrastructure-as-code

Collaboration

• Work closely with data scientists, analysts and business stakeholders to define data requirements and SLAs
• Contribute to proof-of-concept reporting where required
• Translate business needs into robust technical solutions

Experience & Technical Skills

Essential

• 5+ years’ experience in data engineering
• At least 2 years operating in a senior or lead capacity
• Proven experience building data platforms from scratch in production environments
• Strong experience with Azure Databricks, Spark and Python
• Experience with MySQL, PostgreSQL and MSSQL
• Strong experience troubleshooting complex data issues and optimising queries
• Experience working in agile, fast-paced tech or data-driven environments

Desirable

• Experience with dbt, Airflow or similar orchestration tools
• Experience implementing CI/CD for data engineering
• Experience building basic reports for proof-of-concept work

Personal Attributes

• Security-first mindset with strong ethical data handling principles
• Comfortable operating in ambiguity and owning end-to-end delivery
• Strong analytical and problem-solving skills
• Collaborative team player with strong communication skills
• Eager to stay current with emerging data technologies
• Committed to continuous improvement

What’s on Offer

• Competitive market salary
• 25 days annual leave
• Additional “Moments in Time” days
• Life assurance (4x base salary)
• Private pension
• Health care cash plan
• Hybrid working model
• Additional leave linked to length of service

Diversity & Inclusion

The organisation is an equal opportunities employer and makes recruitment decisions based on skills, qualifications and experience. Applications are welcomed from all suitably qualified candidates.

Please note that applicants must have the right to work in the UK.