Overview

Job title: Data Engineer

Location: UK

Duration: 6-month contract Inside IR35

Working Mode: Fully Remote

Vetting: Active SC Clearance

Job Description:

Data Engineer Skills and Technologies

• Data Engineering: Strong understanding of data concepts, data types, data structures, schemas (both JSON and Spark), schema management
• Strong understanding of complex JSON manipulation
• Experience working with Data Pipelines using a custom Python/PySpark frameworks
• Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data
• Strong understanding of Data Security principles, data owners, access controls
• Row and column level, GDPR compliance including experience of handling sensitive datasets
• Strong problem solving and analytical skills, particularly able to demonstrate these intuitively

Required Skills:

• Languages / Frameworks
  • JSON
  • YAML
  • Python (as a programming language, not just able to write basic scripts. Pydantic experience would be a bonus.)
  • SQL
  • PySpark
  • Delta Lake
  • Bash (both CLI usage and scripting)
  • Git
  • Markdown
  • Scala (bonus, not compulsory)
  • Azure SQL Server as a HIVE Metastore (bonus)
• Technologies
  • Azure Databricks
  • Apache Spark
  • Delta Tables
  • Data processing with Python
  • PowerBI (Integration / Data Ingestion)
  • JIRA