Data Engineer

New Yesterday

Job Description

Technical Expertise:


  • Solid experience in Python programming, particularly using data manipulation and processing libraries such as Pandas, NumPy, and Apache Spark.
  • Practical experience in designing, developing, and maintaining robust data ingestion pipelines.
  • Demonstrated ability to optimize code efficiency, database queries, and system performance.
  • Hands-on experience with open-source data frameworks like Apache Spark, Apache Kafka, and Apache Airflow.
  • Strong proficiency in SQL, including advanced query development and performance tuning.
  • Good understanding of distributed computing principles and big data ecosystems.
  • Familiar with version control tools (Git) and CI/CD automation pipelines.
  • Experience working with relational databases such as PostgreSQL, MySQL, or equivalent platforms.
  • Skilled in using containerization technologies including Docker and Kubernetes.
  • Experience with workflow orchestration tools like Apache Airflow or Dagster.
  • Strong grasp of data warehousing methodologies, including dimensional modelling and schema design.
  • Understanding of cloud infrastructure management, preferably using Infrastructure-as-Code (IaC) tools and approaches.
  • Familiar with streaming data pipelines and real-time analytics solutions.

Apply
Location:
City Of London
Job Type:
FullTime
Category:
Technology

We found some similar jobs based on your search