Case Study: The National Energy Analytics Research (NEAR) Programme in Australia

The Australian Energy Market has been changing rapidly over the last few decades, with new technologies, consumer choices and climate-driven policies leading to massive changes in energy use patterns and new pressures on managing the electricity grid. 

However during this time, lack of granular and transparent data on energy use and how it is changing has limited effective forecasting, policy and management of these changes, resulting in unnecessary costs and market disruption. The need for improved energy data and analysis in Australia was emphasised following a significant investment in energy transmission and distribution infrastructure, which could have been minimised if accurate energy demand was more available.

During the late 2000s, Australia’s National Electricity Market experienced a number of peak demand events following extreme climate conditions during summer months. These spikes in demand combined with forecast growth in economic activity, led AEMO to forecast near linear growth in electricity demand through in its Electricity Statement of Opportunities (ESOO). However, as a result of a combination of factors, including the ongoing impacts of energy efficiency policies, changes in economic structure and (more recently) adoption of rooftop solar photovoltaic systems, electricity demand began to fall, contrary to the views of forecasters.

Electricity consumption in the Australian National Electricity Market compared with published forecasts, 2005-06 to 2016-17

A major factor that contributed to the excessive demand forecasts was a lack of detailed energy consumption data. This data would have assisted forecasters to understand important trends, including: the impact of energy efficiency policies; the adoption of new technologies and appliances in buildings; and industrial efforts to improve efficiency, also driven by government policy.

Lack of data has also limited wider policy and regulation across other areas, including: driving adoption of smart meter technology; reforms to electricity tariff structures; monitoring market competition; and measuring the impacts of efficiency policies and new technologies. Without clear evidence of consumer impacts many policies have been delayed or poorly targeted.

In an effort to improve the availability of energy consumption data and supporting research, and therefore improve energy planning and policy, the Australian government has committed over AUD 20 million (USD 14.3 million) to develop the National Energy Analytics Research (NEAR) Program. The NEAR Program is a partnership between the Australian government, the Commonwealth Scientific and Industrial Research Organisation (CSIRO) and the Australian Energy Market Operator (AEMO).

As part of the NEAR Program, CSIRO is working with stakeholders across the Australian energy system to improve access to energy demand data, including data on electricity and gas consumption across the residential, services and industry sectors. The NEAR Program seeks to link energy consumption data with the factors that drive demand, including: appliance and equipment usage; building characteristics; demographics and consumer behaviour; and changes in economic structure. Beyond gathering data and making it available, the NEAR Program also includes a broad research agenda, aiming to create new ways to link, analyse and use data.

High level substation data, for example, collected by the NEAR Program is being mined and analysed to understand the impact that key appliances and equipment, particularly air conditioning and rooftop solar PV, have on energy demand. The more granular data collected from households has led to analysis that enables a better understanding of the building, demographic and appliance use characteristics and how these correlate with electricity use across different seasons and climate

The data collected and analysed through the NEAR Program is improving forecasting capacity and impacting policy. This includes forecasting of potential peak demand, particularly on summer days, where increased space cooling demand drives peaks in electricity demand. Other forecasts, including network reverse flow, from increasing amounts of rooftop solar PV, is also being improved.

In addition to correlating total energy demand with appliances, equipment and demographics, the more granular data obtained at the residential level has improved knowledge of consumer outcomes and influenced competition policy. Specifically, the data provides a greater understanding of current tariffs offered in Australia’s competitive electricity retail market, and their impacts on different types of households, including vulnerable households.

Forecasts of daily electricity consumption in the state of South Australia compared with actual demand profile

Analysis of electricity bills paid by Australian households

The biggest ongoing challenge is availability and access to data which can be fragmented and difficult to access due to restrictions associated with privacy or commercial concerns. As a result, the NEAR programme is also seeking to provide a basis for the introduction of reforms and processes to address these barriers, including providing a protected environment to link and analyse restricted datasets, research on better ways to aggregate and allow access, and new ways to gather data and fill gaps.

Obtaining granular data at the individual household level is a key challenge, due to privacy considerations and the need for statistically meaningful and consistent data over an extended period. In an attempt to address this challenge, the NEAR Program introduced an innovative approach to conduct better surveys in which volunteers are sought to provide data through the NEAR Program’s “Energise app”. With around 4 300 registered users, the app is providing a new source of data collection at the residential level. Another issue is how detailed personal data from smart meters can be collected in a without risking privacy concerns. Clustering techniques are therefore being used to group energy users in to various categories, allowing for more informed analysis of different consumer types and behaviours.

Clustering techniques used to sort residential energy demand data

The NEAR Program was originally developed as a pilot concept, but has demonstrated clear value and is now being transitioned to a permanent market system supported by the Australian Energy Market Operator. This will ensure that data capabilities and research across Australia’s energy market can continue to improve, something that is critical to managing ongoing market change. The datasets aim to eventually cover all fuel types and seek to link energy use data to a wide range of drivers such as demographic, economic and technological facets of energy consumption in Australia.

In addition to the NEAR Program and to support its development, the Australian government is pursuing a range of regulatory reforms to improve the collection, access and sharing of data. One of these reforms is Australia’s Consumer Data Right (CDR), which is being introduced for both the banking and energy sectors over the next 12 months. The CDR is a competition and consumer reform that will allow Australian consumers to require that a company, such as their energy retailer or bank, shares their data with a trusted accredited service provider. This seeks to promote better services and advice for consumers, such as personalised estimates on electricity tariffs or rooftop solar systems, optimised for the their individual energy use patterns. By empowering consumers to navigate increasingly complex energy services, the government hopes to promote better competition between energy service providers, leading to better prices and more innovation of products and services.

• White, S. (2019), The NEAR Program, https://www.energy.gov.au/government-priorities/energy-data/energy-use-data-model