Our increasingly data-rich world is transforming how we gain insights. It’s important to have the right plumbing to streamline your data collection and analysis. Not only will this help to improve the availability of insights you need to make better business decisions, but it enables you to be more responsive to a changing world.

Being able to turn on a tap to get a glass of water is a luxury that many of us take for granted. Yet, how many of us work in organisations with data pipelines that allow us to simply turn on a tap to get the insights we need? For many, finding those insights can be like carrying water in a bucket where you’re reactively sloshing data through ad-hoc manual steps to inefficiently gain your insights. By thinking strategically about the insights that will inform your business decisions, you can design repeatable and automatic data pipelines to have the right insights on tap.

The first step to establishing data pipelines is to have a systematic process for collecting, storing and analysing your data. It’s usually better to have a watertight process in place with basic tools than a sloppy process with the fanciest tools. Clear processes will also help inform where you may need to invest in technology platforms to support your process. You shouldn’t worry about getting it right the first time; start small and once the insights start flowing you can iteratively see where the process can be tightened. 

To prevent yourself from drowning in a puddle of irrelevant data, it’s important to consider what insights you need first and then work backwards.

4. Insights

Ask yourself, "What insights inform the decisions I make?” This might be meeting specific criteria to find a property for a new site development, or information that helps you determine which assets are most in need of maintenance or replacement. Having a clear idea of what insights you want first, and why you need it, will inform the next steps. 

3. Analysis

It doesn’t matter if you’re determining transportation network performance, operation costs for industrial plants or multi-criteria analysis of geospatial data - there are numerous tools and ways to analyse data to draw out insights. And they are constantly evolving. 

Microsoft Excel is a flexible tool that enables you to produce models, aggregate, and visualise data. However manual steps are expensive, time consuming and can introduce human errors, therefore it’s a good idea to consider ways to automate your analysis. Tools like PowerBI are making it easier for us to conduct advanced analysis and visualisation.

The increasing availability of artificial intelligence is also creating opportunities for us to extract new insights, such as detecting the size of crowds through image processing video data or analysing free text to extract topics or sentiments. Real-time dashboards and automated alerts allow analysis to be automatically and continuously applied to provide ongoing insights.

2. Store

Data pipelines depend on consistently storing your data - it's inefficient to use a new location or format every time you have a project you need insights for. Inconsistently storing your data inhibits its reuse which may prevent you from doing valuable trend analysis in the future.

The technology you use to store your data doesn’t need to be complex. A consistently structured spreadsheet can be adequate provided the structure is consistent and well understood by all who need to use it. However as the volume of your data and the velocity in which it grows increases, you can start looking as specialist ’Big Data’ technology platforms. 

1. Collect

Collecting data more quickly and more often increases the timeliness and value of your insights. Mobile devices are replacing paper forms, allowing our asset inspections data to flow instantly from the field to real-time dashboards.

The Internet of Things is allowing us to use remote sensors to constantly monitor a range of environmental factors, including air and water quality. Where data is extracted from other sources, automatic extraction processes can avoid manual steps of downloading and importing data. Once it becomes cheap enough to collect your data, you no longer have an excuse not to use it to inform your business decisions.

With insights on tap, you can change the way you make decisions. Traditionally an energy study took weeks to decide how to better tune the performance of equipment within a building or plant. A dashboard analysing live data feeds from all of your equipment telling you where and when your energy is consumed, allows you to tune these constantly; responding to changes in production or occupancy.

Having the insights we need on hand allows us to be more responsive to changing circumstances. Getting immediate feedback also enables more experimentation, letting us test theories in practice.

At Beca, data pipelines have allowed us to replace expensive manual vehicle surveys by automatically analysing data collected by vehicle GPS units. Our streamlined ‘business as usual’ processes for inspecting bridges in Fiji for example, allowed us to rapidly respond and assess the damage after Cyclone Winston (you can read a thought piece about it here). 

Running water is much more than the convenience of turning on a tap to fill a glass. The businesses that succeed tomorrow will have data pipelines providing the insights they need on tap, when they need it, not dripping from holes in their data buckets.