Australia’s AI Leaders Are Rethinking Data and Analytics

After more than two years of rapid advancement in artificial intelligence, Australian business leaders are refining their data and analytics strategies to better support current and future AI applications. Organisations are prioritising three core objectives: enhancing agility for data-driven innovation, shortening the time required to extract insights and value from data, and achieving these goals in a scalable, cost-efficient manner. A modern, AI-ready data ecosystem is now considered essential for meeting these demands.

Many companies initially turned to hyperscale cloud-based data lakes or lakehouses as a straightforward solution for centralising information. While this consolidation simplifies data storage, it often introduces unexpected trade-offs in both flexibility and expenditure. The process involves copying operational data into the cloud, which then requires significant effort to organise, secure, clean, and transform before it becomes usable. This not only extends project timelines but also drives up costs, especially when businesses end up replicating more data than they actually need. Furthermore, the expense associated with exploring this data for new insights within a single cloud repository can be substantial. It’s also vital to factor in computational costs, which vary widely depending on the specific business task and can heavily influence both the budget and the agility of AI initiatives.

Cloud-based data lakes remain a critical component of any robust data and analytics framework, but they are not the only option. To gain stronger cost control and accelerate time-to-value, a growing number of enterprises are adopting a hybrid architecture for both data and infrastructure. Since 2023, industry analysts have noted a clear trend of organisations using hybrid cloud models to speed up their AI adoption. This strategy offers a balanced approach to managing costs and performance, while also addressing crucial needs like data sovereignty, low latency for edge computing, and reducing dependence on any single vendor.

Getting the most from a hybrid setup requires a careful evaluation of both computational and data storage needs. For instance, a company employing a machine learning algorithm for predictive maintenance should assess where the data originates and in what volume. It might be most effective to develop the initial model in a cloud analytics platform but run the final model directly at the edge, next to the machinery. Alternatively, for very large datasets, building and executing the model within a private cloud environment could be the smarter choice.

Data virtualisation is a key technology enabling this hybrid approach, creating a unified access layer that allows users to find and utilise all enterprise data with real-time connectivity to sources stored in various locations. This method abstracts data from on-premises systems or different clouds into a logical framework, where governance and security policies are applied as information is delivered to consumers. In practical terms, this means the bulk of the data can remain in its original systems; only the specific data needed is brought forward for an external AI model when required. Other datasets might reside in hyperscale, private, or hybrid cloud environments, selected according to the use case.

The tangible benefits are compelling. A Forrester Consulting Total Economic Impact study on the Denodo Platform highlighted that organisations achieved an 83% reduction in time-to-revenue, a 67% drop in data preparation effort, and a 65% decrease in delivery times compared to traditional ETL processes.

Adopting a hybrid methodology across data analytics platforms and infrastructure provides greater agility and lowers risk when executing data and AI strategies. It enables incremental transformation, allowing companies to modernise their data platforms in stages without disruptive “big bang” migrations to the cloud. This approach links new AI capabilities with both legacy systems and modern cloud platforms seamlessly.

A hybrid architecture also supports diverse use cases and stakeholder requirements. Different business units often have conflicting needs regarding data governance and urgency. For example, a Finance department might insist that data remains on-premises for security, while a Marketing team requires immediate access to cloud-based AI tools for campaign analysis. A hybrid model accommodates these varying demands without enforcing a one-size-fits-all solution.

This strategy future-proofs investments against vendor lock-in. By not tethering data and AI capabilities to a single provider’s roadmap, organisations can take advantage of new technologies from a wider range of vendors as the AI landscape continues to evolve. It also enhances operational resilience and business continuity. Distributing data and AI workloads across a hybrid environment reduces the overall attack surface, and data virtualisation helps by minimising the creation of additional data copies. Finally, a hybrid approach ensures alignment with data sovereignty regulations, as it allows sensitive information to be kept on-premises or in specific jurisdictions while still connecting securely with other data assets.

A hybrid architecture that integrates multiple data and infrastructure options, connected by an overarching data virtualisation layer for centralised oversight and management, is rapidly becoming the new standard for organisations pursuing their AI ambitions.

(Source: ITWire Australia)