Energy end-use data collection methodologies and the emerging role of digital technologies

Data are the key to track policies effectiveness and to monitor trends over time, and energy data are no exception. In particular, disaggregated energy demand-side data collection has been a challenge in many countries worldwide, although the role of the demand-side of energy systems, notably of energy efficiency, is widely acknowledged for delivering energy savings and avoiding emissions, and hence contributing to curb climate change.

In order to appropriately track energy efficiency progress, disentangle the different drivers of energy demand (such as activity, structure, and efficiency), and develop appropriate and detailed energy efficiency indicators, it is indispensable to have sub-sectoral or end‑use data together with activity data with similar boundaries.

Increasingly, governments and organisations acknowledge the importance of and are committed to developing energy efficiency indicators across sectors, depending on national priorities, and to collecting the relevant data. Traditionally, four main methodologies are widely applied for end‑use data collection: administrative sources, surveys, metering and modelling. These are often used on a complementary basis. Each has its own strengths and weaknesses, which are discussed in more detail later in this paper.

In addition to traditional methodologies, new and digital technologies represent an unprecedented opportunity for energy demand-side data collection to fill some of the most challenging data gaps as of now. Overall, new technologies can be categorised into three main types depending on their main purpose: data collection, data management and data analysis. Increasing volumes of data collected in almost real-time, broad connectivity, and advanced data analytics could support end‑use data collection and availability, if properly streamlined and structured.

The inherent challenges and difficulties in the use of new technologies have been widely pinpointed in the literature. Huge amounts of data require proper management (including standardising the data collected), ensuring data privacy, raising social acceptance, and allocating proper resources.

This paper is developed under the umbrella of the G20 energy end‑use data and energy efficiency metrics initiative, co-led by the International Energy Agency (IEA) and France through the French Environment and Energy Management Agency (ADEME – Agence de l'environnement et de la maîtrise de l'énergie). It aims at reviewing traditional end‑use data collection methodologies, as well as at exploring and discussing the role of digitalisation/new technologies in energy data collection, while also presenting good practices from different countries in both cases.

The bulk of the material for the section on traditional end‑use collection methodologies derives from the IEA’s Energy Efficiency Indicators: Fundamentals on Statistics, which has been expanded in the IEA Country Practices Database. The digitalisation section results of research work, alongside discussions from the 2019 workshop of the G20 end‑use data and efficiency metrics initiative – Uncovering the role of digitalisation for energy efficiency indicators, and a survey conducted by the IEA on the role of digitalisation for end‑use data collection.

Our review indicates that digitalisation has strong potential for supporting end‑use data collection across sectors, and seems to have been already largely implemented in collecting data particularly for buildings, for example through smart meters. Experts also agree that, at least during a transition stage, new technologies should be a complement to traditional ones, in order to ensure proper quality and statistical representativeness of the data collected.

The paper is structured as follows. The first section focuses on well-established data collection methodologies in line with the IEA efficiency indicators manual. It discusses their respective advantages and drawbacks, and pinpoints examples of good practices based on input provided by countries. Section 2 introduces and describes new/digital technologies from the perspective of their potential use for data collection. It highlights advantages and drawbacks (including barriers for broader implementation) for each technology. The third section discusses and consolidates the review presented above while also showcasing the results from the survey launched by the IEA on the role of digitalisation for end‑use data collection. Lastly, the conclusion section summarises the key findings from the paper.