Senior Azure Databricks Engineer

Please note, this job is no longer available. You can view our live jobs here.

Senior Azure Databricks Engineer – Energy trading – 850£/d – Inside IR35 – London – Hybrid

Our client, a national energy trading company, is looking for a Senior Azure Cloud Databricks Engineer to work on a contract basis in London.

As an Azure Cloud Databricks Engineer, you will focus on developing energy trading systems. This role, based in the London offices with 2-3 days/week in-office work, involves crafting solutions aligned with the client’s Energy Marketing and Trading (EM&T) business requirements.

The responsibilities include developing high-quality, efficient software within specified timeframes and budgetary constraints. The engineer must address front, middle, and back-office needs by comprehensively understanding the energy trading lifecycle and advocating for Agile and DevOps delivery methodologies.

This position offers the opportunity to join a dynamic development team working on cutting-edge platforms built on Azure Databricks and Microsoft Azure Cloud Stack. The candidate will actively contribute to the rollout of platforms, enhancing the capabilities of EM&T business functions.

Key Technical Skills:


  • Experience in Data Warehousing and Data Engineering using the Microsoft Stack.
  • Over 3 years of experience in Azure Cloud Data Engineering with Azure Databricks.
  • Expertise in Energy Markets, particularly in Trading, Credit, and Market Risk functions.
  • Proficiency in Azure Databricks, including knowledge of PySpark clusters.
  • Experience with Databricks Workflows and Unity Catalog setups for unified governance in a multi-domain environment.
  • Exposure to Data Points, Data Points History, DeltaLake, Databricks SQL, Dataflow, and Datamart solutions.
  • Experience with data model management for reporting using Delta Live Tables and implementing a Star Schema in Databricks with Delta Lake.
  • Strong communication skills and collaborative development experience.
  • Proficient in object-oriented programming.
  • Skillful in writing reusable Python and C# libraries, deploying using Azure Databricks DevOps stack.
  • Demonstrated experience in high-quality design and development with clean and readable Python and C# code.
  • Experience with dynamic data mapping and code-free ETL/ELT data integration.
  • Exposure to Java, Apache Kafka, C#, .NET 6 or later, with knowledge of their ecosystems.
  • Knowledge of Power BI stack, Serverless SQL endpoints, Power BI reports, and essential performance & scale considerations.
  • Familiarity with Azure cloud technologies, including Microsoft Azure, Functions, Azure Data Factory, Synapse Analytics, and Azure SQL stack.
  • Knowledge of relational databases, data warehouse, and star scheme concepts.
  • Pragmatic and practical approach to problem-solving.
  • Ability to identify bottlenecks and bugs, devising effective solutions.
  • Ensure peer reviews and unit tests are conducted across the entire team.
  • Experience with Agile TDD, producing and executing test scripts, component test reports, and ensuring adherence to coding standards.


  • Experience with message queues (e.g., Kafka) in Azure.
  • Experience with containerization (e.g., Docker) in Azure.
  • Experience in integrating data integration and enterprise data warehousing on Microsoft Azure.
  • Experience with Streaming analytics and Power BI integration.
  • Familiarity with Azure Identity and access management, including column and row level security.
  • Operational mindset with knowledge of high availability, monitoring, and diagnostics.
  • Development experience within a Microsoft DevOps environment, following proper project lifecycle and change control procedures.
  • Compliance with statutory and company procedures, including security policies.
  • Responsible for effort estimates, timely development deliverables, and ensuring solutions are fit-for-purpose and efficiently supportable. Contribution to retrospectives to incorporate learnings into future work.

Senior Azure Databricks Engineer – Energy trading – 850£/d – Inside IR35 – London – Hybrid

At a glance

  • Location: London
  • Job type: Contract
  • Job sector: Trading & Risk Systems
  • Salary: Negotiable
  • Published: November 16, 2023