From Petabytes to Profit: Unlocking the True Value of Data

Today, data is one of the single most important drivers of success. Yet, managing modern data’s sheer volume and complexity is a challenge that few organisations are prepared for. We had the pleasure of speaking with David Garrett, a senior hands-on tech-consultant and leader with over 30 years of experience in trading system integration, design, and development. David explains how companies can unlock the full value of their data, highlighting that without solid systems in place, businesses risk falling behind as data becomes both a crucial asset and a growing cost.

The Evolution of Data

How we quantify data has evolved dramatically. “When I started, it was all about kilobytes. Now we’re talking gigabytes, terabytes, and even petabytes or exabytes—the equivalent of 250 million DVDs,” David reflects. But the story doesn’t end there. The rise of synthetic data—generated by algorithms rather than collected from real-world events—adds yet another layer to this complexity. And this growth comes at a cost: not just in the immense power consumed by data centres, but in the demand for talent capable of understanding and extracting value from the data.

As companies scramble to manage this explosion of data, many struggle to keep up.

“Companies, I think, are just playing catch-up in this space. On the one hand, they may have big end drives or big databases, but on the other, they have no data lineage or data catalogues. They really need to get the data sorted out. And then off the back of that, you’ve got GDPR, you’ve got FADP (Federal Act on Data Protection), or even the European Union’s risk stuff for AI. Data is really at the centre and it’s growing.”

Building the Right Foundations

For all the talk of advanced systems, the fundamentals are essential. As David puts it, “If you don’t know where your data came from, you don’t know what the quality is.” Data lineage – the ability to trace data’s origins, transformations, and destinations – is critical for companies looking to fully leverage their assets. Without this foundational understanding, data becomes difficult to trust and use effectively.

“If you have a large network drive of ten years of your company’s data and you don’t know where it came from, it’s going to take you a bit of time to sort it out. Maybe it’s the first job for AI to go in and do the classification. It’s not a human activity. You can’t download the directory of the drive to your Excel spreadsheet. You need something that goes in, shrinks it, adds metadata, can classify it in different ways and can look inside the data and help you.”

Aligning Teams for Success

A crucial element in mastering the data boom is ensuring organisations have the right teams in place. Often, IT teams operate in isolation from the business, creating a disconnect. As David explains, “They’re still caught up in legacy. They’re writing Java, they’re writing Python, but they’re not really dealing with the business knowledge needed to push the boundaries.” This disconnect can lead to IT professionals spending too much time “sitting in a dark space and programming,” as David puts it.

“That’s no disrespect – I’ve had some of the best programmers on my team who don’t always communicate well. But when a business analyst gets involved in between, the message can sometimes get lost in translation.” And that’s why David emphasises the importance of direct collaboration: “When I started on the trading floor, sitting next to the trader, listening to what they wanted, and building something quickly – maybe we go back to that kind of mentality.”

The Importance of Trust 

As companies explore the world of AI, David stresses that “trust is the big keyword,” and this hinges on AI’s ability to explain how decisions are made.“If you can’t explain to the regulator why a model made a specific decision, particularly if it results in a loss, that’s a big problem.”

However, understanding how these decisions are made is becoming increasingly challenging. “When you train a neural network, like ChatGPT, you’re working with layers of neurons and millions of synapses. While these systems can process vast amounts of data, the path from input to output is often unclear,” David notes. As AI models continue to evolve, this issue only intensifies. David highlights that “models are becoming more fluid, not rigid. The connections between neurons are constantly adapting, making it even harder to trace the path of a decision.” This dynamic nature makes explainability more complex, especially as models handle larger datasets and more sophisticated tasks. Companies need to decide for themselves where they stand in the trade-off between accuracy and explainability, as the most sophisticated AI models are often the most opaque.

Embracing Change 

The evolving landscape of technology calls for roles and responsibilities within organisations to be redefined. For instance, David mentions the role of “Chief Technical Transformation Officer.” Emerging roles like this not only challenge traditional norms but also reflect the growing need for innovation and adaptability in modern organisations.

That being said, introducing new roles, such as site reliability engineers or specialists in AI-driven innovation, can meet resistance due to uncertainties around pay structures, market benchmarks, and organisational fit.

This resistance often stems from a lack of understanding and fear of the unknown. It’s crucial for leadership to be proactive in supporting these changes, as creating space for such roles drives transformation. Collaboration with external advisors can help design frameworks like centres of excellence, ensuring a seamless transition to modernised operations.

But, building effective teams is about more than just technical expertise. Organisations must prioritise inclusivity and create a culture that identifies and nurtures hidden potential within existing talent. The best people for the job might not always fit preconceived notions but can drive impactful transformation when given the opportunity.

By embracing flexibility and innovation, organisations can shape a future-ready workforce that thrives in dynamic environments.

Final Thoughts: A World of Opportunity and Challenge 

From regulatory pressures to the sheer scale of data growth, companies face a daunting challenge—but also a golden opportunity. Those who can master their data, trace its lineage, and build the right teams will be the ones who come out on top. 

Looking for more insights?

Get exclusive insights from industry leaders, stay up-to-date with the latest news, and explore the cutting-edge tech shaping the sector by subscribing to our newsletter, Commodities Tech Insider. Interested in being featured in a future spotlight? Reach out to [email protected].