When Does Big Data Become Too Much Data?

When Does Big Data Become Too Much Data?

By Greg Farrell, Head of System Strategy, Northern Powergrid

Greg Farrell, Head of System Strategy, Northern Powergrid

The fourth industrial revolution is driving radical change across our society. New technologies are opening up new possibilities, changing the way we manage everything from our homes to our energy infrastructure. The driving force? Big data.

For an energy network operator, working with data sets covering every aspect of operation is invaluable to ensure efficient running of an increasingly complex system and obtain previously unobtainable insights about everything from electricity demand and supply to public attitudes towards decarbonisation and electric vehicles, which can revolutionise business operation and the customer experience. This data also enables us to better plan for the future, devise and refine scenarios, and support frank and open discussions with our stakeholders.

This wealth of information has turbo-charged the energy transition. For instance, 20 years ago, who would have thought technology would let us switch our gas boilers off and on from our mobile phones? Crucially, access to and analysis of data is essential as customer demands for new and different services from the networks drive the transition from distribution network operators (DNOs), managing a centralised network, to distribution system operators (DSOs) operating the network dynamically to deliver reliable, clean power in a system with decentralised resources. Data provides critical insights about our networks, enabling operators to build a more interconnected, cost effective, and clean energy system —a system fit for the customers they serve.

As more data becomes available, however, careful management of collection, storage, and use is becoming increasingly important for energy companies. There is no doubt that data is hugely beneficial, but maintaining increasingly large data sources and knowing what is valuable poses a significant challenge.

Energy companies have been eagerly gathering all the potentially useful data they can find about their systems and customer energy consumption behaviours and storing it across a range of platforms. While much of the data is of value, managers are often left with vast sets of information that isn’t relevant, has considerable gaps, or is valuable but may never be put to use. It is imperative that energy companies manage the data-gathering process carefully to ensure the data is useful, can be efficiently accessed, and does not waste resource.

This is more important now than ever before. The UK has committed to become carbon neutral by 2050, the sale of petrol and diesel cars will be banned by 2040, if not before, and more distributed and intermittent energy generation is being brought online.

Delivering the DSO transition will be crucial to achieving these aims and decarbonising our energy and transport industries. It is therefore essential that time is put to good use by gathering valuable insights that help us achieve these ambitious targets.

How do we ensure that we are gathering valuable data?

1. Engage the Users: Companies must understand at the outset what information they need and what is valuable to the intended user. Providing dedicated forums to bring data users and collectors/providers together to understand the challenges that data can address and what is useful and possible is essential.

2. Target the Right Data: Next, it is time to pinpoint the information that could provide a solution and implement a detailed plan to harvest it. The data needed may be technical, harvested from substations and network reports, require academic research or qualitative insights. Whatever the approach, it must be structured to secure the most valuable data in the shortest timeframe.

3. Accessible Data: With ever-expanding data sets, presentation becomes an increasing challenge. Important data should not be left to languish on tab eight of a complex excel sheet, never to see the light of day. Finding the right format, whether that be extensive data sheets, graphs or online data hubs, tailoring the format to the data is key to ensuring the information is managed, interrogated and utilised effectively.

4. Remember the Customer: As a network operator, the primary audience is always the customer. We must always ensure the data we collect will benefit our customers and ensure we collect information directly from them too. In the digital age, the means of engaging this audience have expanded significantly. Social media provides direct access, and online surveys can quickly provide customer insights and other digital tools, such as apps and online discussion forums, can complement traditional engagement methods to ensure more extensive and robust data collection.

"There is no doubt that data is hugely beneficial, but maintaining increasingly large data sources and knowing what is valuable poses a significant challenge"

5. Artificial Intelligence: More data requires increasingly careful management. The rise of artificial intelligence (AI) is incredibly valuable as it can assess full data sets in seconds, highlighting trends, patterns, and flagging missing or duplicate information. Energy networks can engage new AI-driven technologies to process large, disparate data sets and should collaborate to deploy the best tools across the industry.

Remember, the more valuable your data, the greater the results you will able to achieve.

Box Out: Modelling Data

On its DSO transition, Northern Powergrid has already moved beyond the DNO paradigm of being a ‘fix on fail’ service. The company is working with small-scale generators to provide grid services and optimise network performance.

To effectively expand this function, Northern Powergrid is securing meaningful insights into the network’s changing requirements. The company wants to understand how its stakeholders’ requirements and external influences will change.

This necessitates development of complex data models to predict how the energy industry will change over the coming decades. This presents inherent uncertainty and Northern Powergrid manages this process through the development of a series of scenarios, with varying external inputs, such as policy and consumer behaviour changes.

These scenarios will enable Northern Powergrid to deliver grid services from small-scale generators that cuts costs, increases reliability, and reduces emissions.

Read Also

Supporting the Zero-Carbon Transition as a Service

Supporting the Zero-Carbon Transition as a Service

Loic VILLOCEL, Director, ENGIE Lab Singapore
Baking in Cyber from the Top

Baking in Cyber from the Top

Lena Smart, VP & CSO, New York Power Authority
The Pathway to Revolutionize the Energy Industry

The Pathway to Revolutionize the Energy Industry

Kevin Sullivan, CEO, Advanced Control Systems (ACS)

Weekly Brief