Overview

Senior Software Engineer - ETL Pipeline Orchestration Platform

Location: London

Business Area: Engineering and CTO

Ref #: 10046054

Description & Requirements

Bloomberg is a global leader in business and financial information, delivering trusted data, news, and insights that bring transparency, efficiency, and fairness to capital markets. Bloomberg's Data Technology Infrastructure Department designs and builds large-scale platforms to acquire financial reference and pricing data globally from various sources, delivering billions of data points daily to clients for investment analysis.

The ETL Pipeline Orchestration Platform team owns a low code platform that helps engineers and analysts author and manage data transformation, derivation, validation and database loading themselves. As adoption accelerates, we are redesigning key components to support further throughput and resiliency while maintaining the availability of the system.

What’s in it for you?

We are the connecting tissue across the various core capabilities offered by our sibling teams. If you join our team, you will be working on solving the fundamental problems in event-driven orchestration systems using both open source technologies as well as in-house ones. We are a multi-lingual team taking care of python applications, performant queue broker in Go to C-based Python extensions and everything in between. If you have a knack for distributed systems or a love for orchestration platforms, join us!

Future Initiatives

We would like to work on the following in this year and next:

Develop, deploy and integrate our internal priority queue system to mitigate head-of-line blocking issues and optimize the time to market for the most important tasks

Deprecate the on-cloud orchestration system by unifying with on-prem system

Component registration / management faculty to make the onboarding processing easier for the service owners

Introduce multi-tenancy with granular privileging for both pipeline creation and edit time as well as runtime

Better support and capability for creation of DAG in orchestration

Programmatic recipe creation, modification and interaction to be API first

LLM integration to assist in authoring ETL tasks

We’ll trust you to:

Work with us to design the user-friendly, fault tolerant and scalable orchestration platform

Participate in the full SDLC of various components and systems that make up of the orchestration platform

Familiarize yourself with the Data Processing / Analysis tasks taken on by the Data Department and systems developed by our sister teams

Have your antenna up and bring the latest development in OSS and capabilities into our stack

Mentor less experienced members of the team

You’ll need to have:

Demonstrable experience building fault tolerant and horizontally scalable distributed applications / systems dealing with distributed system specific concerns

Demonstrable experience writing production code in either Python, Go, Rust, JavaScript/TypeScript or Java

Strong problem solving skills

We’d love to see:

Knowledge of Database Systems and trade offs in the distributed systems

Familiarity with API Designs

Familiarity with Orchestration Frameworks such as Apache Airflow, Argo Workflows, Conductor etc.

Experience working with and designing systems utilizing AWS

Bloomberg is an equal opportunity employer and we value at our company. We do not discriminate on the basis of age, ancestry, , or expression, genetic predisposition or carrier status, marital status, or , race, religion or belief, , , and other reproductive health decisions, parental or caring status, physical or mental , or parental leave, protected veteran status, status as a victim of domestic violence, or any other classification protected by applicable law.

Bloomberg is a inclusive employer. Please let us know if you require any reasonable adjustments to be made for the recruitment process. If you would prefer to discuss this confidentially, please email amer_recruit@bloomberg.net

#J-18808-Ljbffr