Overview

Senior Data Engineer — On-site — Full time

Methods Business and Digital Technology Limited is a UK-based IT Services Consultancy with a strong focus on transforming public sector operations and data solutions for central government departments and agencies.

Methods was acquired by the Alten Group in early 2022.

Responsibilities

We are seeking a seasoned Senior Data Engineer to design, build, and maintain sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The role involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, contributing to organisational growth and innovation.

• Develop and Manage Data Pipelines: design, construct, and maintain efficient and reliable data pipelines using Python/Go/Azure Data Factory, supporting streaming and batch processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.

• Hybrid Cloud and Data Storage Solutions: implement and manage data storage solutions leveraging on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.

• Containerisation and Orchestration: utilise Docker for containerisation and Kubernetes for orchestration to enable scalable deployment across cloud and on-premises environments.

• Workflow Automation: use tools such as Azure Data Factory to automate data flows and manage complex workflows within hybrid environments.

• Event Streaming Experience: utilise event-driven technologies such as Kafka and NATS to handle real-time data streams.

• Security and Compliance: manage security setups and access controls, including tools like Keycloak to protect data integrity and comply with legal standards.

• Database Development: design and develop PostgreSQL databases with high performance and availability across deployment scenarios.

Essential Skills and Experience

• Strong Python Skills: expertise in Python for scripting and automating data processes across varied environments.

• Experience with ETL/ELT: demonstrable experience in developing and optimizing ETL or ELT workflows in hybrid (on-premises and Azure) environments.

• Expertise in Hybrid Cloud Data Architecture: knowledge of integrating on-premises infrastructure with Azure cloud services.

• Containerisation and Orchestration Expertise: solid experience with Docker, GitHub and Kubernetes in managing applications across on-premises and cloud platforms.

• Proficiency in Workflow Automation Tools: practical experience with Azure Data Factory in relevant environments.

• Experience in Event Streaming: proven ability in managing and deploying event streaming platforms like Kafka and NATS.

• Data Security Knowledge: experience with implementing security practices and tools, including Keycloak, across multiple platforms.

• Search and Database Development Skills: strong background in Elasticsearch and PostgreSQL across on-premises and cloud infrastructures.

Your Impact

You will empower business leaders to make informed decisions by delivering timely, accurate, and actionable data insights from a robust, hybrid infrastructure. You will champion adoption of modern data architectures, mentor team members, and advance engineering practices to cultivate a data-driven culture within the organisation.

Desirable Skills and Experience

• Azure and Other Relevant Certifications: cloud and on-premises certifications are beneficial.

• Data Engineering Experience: minimum of 5 years in data engineering with significant exposure to on-premises and cloud infrastructure.

• DevOps Experience: some DevOps engineering experience would be preferable.