The Data Journey: Mastering Collection, Management, and Output

Data is the backbone of decision-making in today’s organisations, but its true value can only be unlocked through a clear, well-structured journey. This journey starts with the collection of raw data from diverse sources, followed by its careful management and organisation to ensure it’s accessible and usable. Finally, the most crucial step is the output – transforming this data into valuable insights that can guide strategic decisions. Along the way, challenges arise, but by mastering each stage – collection, management, and output – you can build a solid foundation for success.

1. Collection

A significant challenge faced by many organisations is ensuring their data is democratised across the business. Many organisations have extensive proprietary information that is spread across various divisions and technologies. Given the vast and varied nature of this data, the starting point should be to capture it and make it available throughout the organisation.

Companies undergoing data-centric transformations have been trying to achieve this successfully for decades. In the 90s, people talked about concepts like data warehouses, OLAP (Online Analytical Processing), and OLTP (Online Transaction Processing). These represented early efforts to manage and analyse data. Over time, these ideas evolved into today’s concepts like data lakes, data mesh, and data virtualisation. Despite these advancements, managing and integrating data effectively remains one of the biggest challenges for organisations.

2. Management

Managing data is costly, and no easy feat. It’s frustrating to realise that up to 90% of a data scientist’s time can be spent on engineering rather than analysis. One way to gain a competitive edge is by reducing the time spent on engineering. That’s why you shouldn’t underestimate the value of roles like data managers or master data specialists. While these positions may seem less technical, they are critical for creating the infrastructure needed for data engineers and data scientists to succeed. They ensure that data is well-structured and accessible, allowing the entire organisation to benefit from self-service analytics.

Ten years ago, roles like Master Data Manager and Data Manager were common. But suddenly, the “sexier” roles emerged, and everyone believed that hiring a few data scientists would solve all their problems. Why would they need a data manager when they could just hire someone with a degree in mathematics, who could write code but had no understanding of their business? As a result, many data managers were let go, and, realising there wasn’t much demand in their area, they shifted to other fields. So today, this breed of professional has almost died out. There isn’t a large talent pool of candidates with competencies in master data management, data governance, and similar areas.

Umid Akhmedov, CDO at Danske Commodities

3. Output

Driving useful insights from your data requires both strong technology skills and domain expertise. Without domain expertise, tech teams risk falling into potholes, where they might spend weeks or even months on what they believe are valuable insights. Yet, it will only take a domain expert a few seconds to recognise they’ve taken a wrong turn. Equally, if the tech team informs the domain expert that something they believe is crucial for strategy or profitability is actually irrelevant, the trader must be comfortable with being told they’re wrong. They need to be open to acknowledging that their preconceptions might not be accurate.

A massive error that a lot of people make is bringing in very clever technical people and then relegating them to the IT department, so to speak, telling them to analyse the data and report back. The issue is that while they might identify patterns or insights, it will take someone with domain expertise just five minutes to see why those insights might be invaluable. There needs to be a cohesive approach where traders and technical people work together. When they operate separately, it simply doesn’t work.

Jack Nugent, Director at Tradavex

As Nugent explains, there needs to be a ‘symbiotic relationship’ between the tech team and the traders:

You could have the best technical person, best coder, best data scientist, best engineer. But if they don’t have someone to share the domain expertise with them, they’re never going to be able to add value to the business.

Jack Nugent, Director at Tradavex

While it’s becoming a common requirement for traders to have basic Python skills, it’s more important to focus on building a structure that breaks down silos between tech teams and domain experts, rather than searching for talent that possesses both skill sets – because, if we’re honest, finding someone who truly excels in both is rare.

The success of this approach also hinges on the type of people you hire. When speaking with our experts, the most commonly highlighted trait was curiosity. As Nugent explains, it’s about being curious enough that, even as technologists, they want to understand “how a ship gets chartered, how a cargo gets sold, and the entire lifecycle of a trade.”

To unlock the real value of data, organisations need to get three things right: collection, management, and output. It’s not just about having the latest tools or the smartest people – it’s about making sure the right people are working together. Break down silos. Pair tech teams with domain experts. Hire curious individuals who ask the right questions and dig deeper. When everyone’s on the same page, data stops being a resource and becomes a competitive advantage. It’s not easy, but for those who get it right, the rewards are game-changing.

Looking for more insights?

Read the second edition of our eBook series: The Democratisation of Data in Commodities – Today’s Data-Driven Markets.

Featuring insights from current and former employees at: Trafigura – Danske Commodities – Vortexa – Tradavex – ChAI – Tallarium