Data Engineer

New Yesterday

The Job As a Data Engineer, you will be working in the Red Engine Business Insights team, helping to build out the existing data and analytics platform. This role is placed within a small team, allowing the successful candidate to work closely with the senior engineers in implementing bespoke features and enhancements to our data platform using the latest technology. In this role you will assist the senior and lead engineers as well assist in driv ing the development of the data platform. This will i ncluding meeting with key business stakeholders to gather technical requirements and converting these requirements into technical data solutions under the guidance of the lead engineer within the Data & Analytics platform. Key responsibilities will include:
Developing and maintaining data pipelines to orchestrate the ingestion of data from disparate source systems into a centralised data analytics platform . Work with the Lead Engineer to d esigning and implement data engineering solutions using T-SQL, Python, PySpark and DBT in the Azure cloud environment .
Working with Data Analysts in promoting business logic into the analytics platform, to support business reports and dashboards .
Maintaining and leveraging CI/CD deployment pipelines to promote application code into higher tier environments.
Maintaining and making additions to the technical documentation suite in Azure DevOps .
To be successful in this role, youll : Essential Skills: D emonstrated experience developing of ETL/ELT ingestion pipelines to handle data movement, transformation and visualisations from structured and unstructured sources.
Experience with the Azure Platform including:
Data Ingestion: Azure Data Factory (ADF ), Databricks , Logic Apps and Function App
Data Storage: ADLS, SQL Server and Unity Catalog (Medallion Architecture)
Data Analysis: Databricks Notebooks, SQL Queries, Data visualisation
Strong understanding of the Databricks Platform including managing, developing and deploying workflows, jobs and notebooks.
Proven experience in modelling data in a Data warehouse using an Inmon or Kimball approaches .
Database development experience in SQL Server including Store Procedures and T-SQL or similar enterprise database toolset.
Experience working in an Agile software development framework.
Experience working with Data Build Tool (DBT) including building data models, contracts, tests, validation, and transformations.
Experience working with modern data distributed file formats (i.e., Parquet, Delta, Iceberg, Hudi) .
Demonstrated experience with building data ingestion pipelines from REST API data sources.
Strong ability to produce technical documentation that can be understood for both technical and non-technical audiences.
Desirable Skills: Experience with IaC solutions using Terraform, Pulumi or similar tools.
Experience with modern CI/CD DevOps frameworks.
Experience in developing data visualisations using PowerBi , Tableau or similar tools
What you'll get Competitive pay A nnual bonus? 33 days annual leave inclusive of Bank Holidays? Fusion working (our team are regularly in our venues, working collaboratively in our bright offices in Angel, or focusing on individual projects with work from home Thursdays)? Staff discount in all venues (50% off Sunday, Monday, 25% off Tuesday Saturday, and free game hire)? Private healthcare? Regular team socials? and w eekly lunch in venue? Monthly learning and development classes, quarterly teambuilding events? Summer and Christmas socials? Help @ hand 24/7 health support? Free access to therapy, nutritionists, and physiotherapists? Weekly lunch in venue?
TPBN1_UKCT
Apply
Location:
North London

We found some similar jobs based on your search