Data Engineer

Data Engineer – Databricks & Real-Time Data London (Onsite/Hybrid)

Cititec are working with a global commodities trading organisation that is expanding its Enterprise Data Platform capability as part of a major shift in its data strategy. They are onboarding Databricks as a new central analytics platform to sit alongside their existing data estate, with a view to it eventually sourcing data across the wider business, including future integration with market risk data feeds. This is a hands-on role for someone who can build real-time data pipelines and help stand up a platform that is still taking shape, working closely with cloud engineering, risk and data architecture teams.

You’ll be responsible for helping build and roll out the Databricks platform, with a strong focus on real-time and streaming data integration across the business.

Typical responsibilities include:

• Supporting the build-out and rollout of a new Databricks platform, working alongside an existing analytics environment during the transition

• Building real-time and streaming data pipelines (e.g. Kafka, Spark Structured Streaming, Delta Live Tables) to support business-critical data flows

• Contributing to the delivery of a substantial pipeline of data products for business consumption, as part of an early wave of platform rollout

• Integrating data from across the business, including eventual connection to market risk data feeds

• Migrating legacy platforms and datasets (e.g. Hadoop, SAP BW, on-prem warehouses) into a modern lakehouse environment

• Helping evolve the platform’s approach to data governance, cataloguing and cross-platform access as the business scales

• Collaborating with a wider Enterprise Data Platform team, including an international delivery function, and stakeholders across risk, cloud engineering and semantic data teams

• Working effectively in a fast-moving, still-evolving team structure where priorities and ownership are still being defined

What We’re Looking For

• Demonstrable, hands-on Databricks experience is essential: production workloads, notebooks, workflows and cluster management, not a single proof-of-concept

• Real-time / streaming data experience is essential: genuine end-to-end streaming pipeline delivery, not solely scheduled batch ETL/ELT

• Background in enterprise data platforms, data engineering or similar, ideally including legacy platform migration into a modern lakehouse

• Comfortable working in a fast-moving, still-evolving team structure with a distributed, international delivery team

• Data governance and cataloguing exposure is a plus, such as Unity Catalog, data lineage or data quality frameworks

• Experience in commodities trading, energy trading or financial services is strongly preferred but not required