2 Audit approach

Audit methodology

An audit methodology was developed by the EPA as a guide for undertaking the performance audit of electricity retailers' GHG reduction strategies. The audit methodology sets out the objectives, scope, audit criteria, timetable and audit approach. To ensure transparency of the audit process, the draft audit methodology was mailed out in March 2000 to approximately 100 organisations, including those in the electricity and energy sectors and environmental groups. The audit methodology was revised in response to comments received from 28 stakeholders including electricity retailers.

The EPA audit methodology was prepared in accordance with and conforms to existing auditing standards, the requirements of the Electricity Supply Act 1995, relevant subordinate licence conditions and ministerial guidelines. An independent opinion on the audit methodology given by Ernst & Young (June 2001) states: 'In our opinion, the EPA's Methodology for Auditing the Effectiveness of Electricity Retailers' Greenhouse Gas Emission Reduction Strategies dated May 2001, is reasonable and appropriate in accordance with applicable Auditing standards and the requirements of the Electricity Supply Act 1995, and relevant subordinate licence conditions and ministerial guidelines in force at May 2001.'

Although the requirement for retailers to prepare GHG emission reduction strategies entered legislation in December 1995, the guidelines and procedures for this were not finalised until early 1997. Taking 1997 as the effective start date, and given the requirement for the EPA to audit at 'no more than three years', the first audit was scheduled to be done after the 1999–2000 reporting period. The EPA began the audit process by conducting preliminary audits for the 1998–99 reporting period. These preliminary audits were used to test the audit process and to uncover any immediate issues with data provision. Feedback from this process was provided to all NSW electricity retailers in an EPA letter distributed by the MEU, so that reporting could be improved for the 1999–2000 reporting period.

Back to top

Scope of the EPA performance audit

The audit was limited to a review of NSW electricity retailers who:

 

• hold a NSW Retail Supplier's Licence, and

• have negotiated GHG emission reduction strategies with the Minister for Energy, and

• covered the 1999–2000 reporting period.

During 1999–2000, 16 NSW retailers both held Retail Supplier's Licences and had negotiated GHG emission reduction strategies with the Minister for Energy.

The objectives of the audit are:

• to determine the sufficiency and appropriateness of the evidence cited in the retailers' IVRs to support the EPA's conclusions, and
• to determine the effectiveness of the strategy measures implemented by the retailers.

The GHG emission reduction strategy measures that were examined by the EPA audit fall into two specific categories. The first category relates to supply-side measures, where electricity is obtained from low-emission sources, including:

• hydro
• landfill gas
• biomass
• wind
• solar
• coal seam methane
• cogeneration.

The second category relates to actions for reducing electricity sales that give rise to a claim for ESF. The allowable actions are:

• energy efficiency
• cogeneration at customer site
• fuel substitution
• off-grid, remote-area power supply

A third category relates to GHG emission 'credits' for carbon sequestration (for example from forestry), which is beyond the scope of this audit.

Back to top

How is a performance audit conducted?

The EPA audit of the GHG emission reduction strategies is a an environmental performance audit based on the principles espoused in 'AS/NZS ISO 14010: 1996—Guidelines for Environmental Auditing—General Principles' and auditing standards 'AUS 806—Performance Auditing', 'AUS 808—Planning Performance Audits' and 'AUS 606—Using the Work of an Expert'.

The audit is a two-stage process. The first stage consists of collecting and examining the retailers' GHG emission reduction strategies, 1-, 3- and 5-year plans, IVRs and greenhouse reports (a standalone attachment to a retailer's licence condition compliance annual report). The EPA's level of assurance on the reliability and accuracy of retailers' GHG emission data and sales forgone data is determined by a desktop examination of the IVR that should accompany each retailer's greenhouse report. The EPA assesses the sufficiency and appropriateness of the expert audit evidence contained in the IVRs. Any reporting deficiencies or anomalies identified during the EPA's examination of the retailers' greenhouse reports and IVRs are reported to MEU or IPART and through them to the retailers to provide the required data.

During the second stage of the audit, the EPA assesses the effectiveness of the individual strategies (both supply and demand side) of each retailer, and the aggregate of each retailer's strategies in achieving the apportioned 'guideline emission benchmark' for that retailer.

For a full description of the stages of the audit process, see 'How and What Does the EPA Audit?' in Appendix 1.

Back to top

Limitations

In undertaking the performance audit the EPA was faced with a number of significant limitations. These limitations can be broadly classed into two major groups. The first relates to the adequacy of the data available to the EPA to undertake the audit in relation to:

• direct electricity sales
• performance summary table
• independent verification reports
• GHG emission reduction strategies
• 1-, 3- and 5-year plans and revisions.

The second group relates to some grey areas in the existing guidelines on how electricity retailers could make claims in relation to:

• electricity sales forgone, including the use of the NSW Sustainable Energy Development Authority's (SEDA) deeming formula
• Energy Smart Business
• Greenpower assigned generation and other schemes
• the status of 1-, 3- and 5-year plans and revisions that do not lead to compliance.

Direct electricity sales

The 'direct electricity sales' are sales contracted directly between electricity generators and very large consumers who take supply at high voltage. These consumers take supply directly from the high-voltage transmission grid (primarily operated by Transgrid in NSW) as opposed to the medium- and low-voltage distribution grids (operated by distribution groups such as Integral Energy and EnergyAustralia).

For national electricity market purposes, these direct sales are attributed to a 'retailer' deemed to be associated with the generation group concerned. Examples are Pacific Power and Macquarie Generation, both of whom held NSW Retailer Supply Licences throughout 1999–2000 and continue to do so (note that Eraring Energy has now inherited the licence formerly held by Pacific Power).

The NSW EPA has not received greenhouse gas reports for the 1999–2000 compliance period from these direct retailers. As a result, the EPA is unable to audit the effectiveness of the greenhouse performance in respect of these direct sales. The Further Environmental Guidelines prepared by the Department of Energy (now the MEU) state: 'For the purposes of these guidelines, the NSW electricity sector covers both sales to retail suppliers and sales to wholesale customers who are not also retail suppliers.'⁶

In 1999–2000, the total electricity sales in NSW were as shown in Table 2.1.

Back to top

Table 2.1 Breakdown of direct and retail sales in NSW in 1999–2000

Total electricity sales in NSW 1999–2000	58,492 GWh
Retail sales	49,811 GWh
Direct sales	8,681 GWh

Thus, approximately 15% of the electricity sales in NSW in 1999–2000 were direct electricity sales.

Because the EPA did not receive information on the performance of the groups involved in direct sales in terms of improving the GHG emissions of the electricity supplied, the EPA performance audit has been limited to only 85% of the NSW retail electricity sector.

Performance summary tables

As mandated by Section 3.3.1 (c) of the Guidelines and Requirements Policy (footnote 1), each retailer is required to 'report on its performance on the implementation of its strategies for audit by the EPA . . .'. While generally this information is collected for other purposes to facilitate the assessment of the effectiveness of GHG emission reduction strategies, the EPA through the MEU requested retailers to include a performance summary table (PST) in their annual greenhouse report covering the 1998–99 and 1999–2000 reporting periods and forecasts for at least the 2000–01 reporting period.

The PST would include negotiated GHG emission reduction strategies, forecasts of electricity sales and sales forgone for each strategy, GHG emissions and GHG emission reductions associated with each strategy, self-assessment of the effectiveness of each strategy and the extent of implementation, and any changes to the strategies and forecasts.

Thirteen of the 16 retailers' greenhouse reports either did not include PSTs or contained PSTs with insufficient information to enable a performance assessment by the EPA of the individual GHG emission reduction strategies of each retailer. One retailer has not negotiated a GHG reduction strategy with the Minister for Energy.

One of the key pieces of information in the PST used by the EPA in assessing the effectiveness of retailer strategies is the cost of measures implemented.

Section 9 (g) of the Further Environmental Guidelines mandates that retailers supply information on the cost of each GHG emission reduction measure (i.e. strategy) both in total and as an estimate of dollars per tonne of CO₂-equivalent saved. This information is treated as commercial in confidence. The cost data is used to:

• determine whether particular types of measures are proving more cost-effective than others on average
• examine differences in reported costs between retailers implementing similar measures
• highlight unusually high or low costs, which may then merit further EPA investigation.

In the greenhouse reports originally supplied to the EPA by the MEU, most retailers had neglected to supply cost data for each GHG emission reduction measure, both in total and as an estimate of dollars per tonne of CO₂-equivalent saved.

The EPA accordingly asked IPART to forward a letter to all retailers highlighting the need for providing the EPA with the PST data, including the cost data, and ten of the retailers obliged.

However, the PST supplied in some cases was not complete, or some retailers declined to provide cost data for measures implemented, citing the data as being 'commercial in confidence'.

Since the data in the PST, including the cost data, is an intrinsic part of assessing the effectiveness of implementation, the non-conformance by retailers with this licence requirement greatly hampers the EPA in making an objective assessment of retailers' GHG emission reduction strategies.

Back to top

Independent verification reports

To facilitate the transfer of information to the EPA for auditing purposes, each retailer is required to prepare a greenhouse report including independent verification as a standalone attachment to their Licence Condition Compliance Annual Report. Each retailer is required to engage the services of an independent verifier in respect of the greenhouse report, and to include the IVR in the greenhouse report forwarded to the regulator.

The requirements for independent verification are outlined in:

• the Electricity Supply Act 1995
• Section 8 of the Further Environmental Guidelines ⁷
• Guidelines for Independent Appraisal ⁸.

In a letter sent to all NSW retailers on 9 September 2000 by the MEU, EPA has provided feedback on the IVRs reviewed during the preliminary audits of the 1998–99 reports. Key comments in that letter⁹ were as follows:

• The IVRs examined in the preliminary audit (the 1998–99 IVRs) 'generally appeared to be incomplete in that they did not contain all the information required by mandatory reporting requirements.'
• 'The sufficiency of information varied considerably between IVRs and was generally considered unsatisfactory.'

That letter outlined the requirements for an IVR and the information requirements for the EPA audits. The EPA uses the IVRs as expert audit evidence that the GHG emission data reported by retailers in their annual greenhouse reports are reliable and accurate.

The independent verifier must include sufficient detail on each of the reporting requirements in the IVR that have been mandated in the Minister for Energy's Guideline for Electricity Retail Suppliers on Licence Condition Compliance Annual Reporting, dated 30 June 1999. These Ministerial guidelines require each retailer
'. . . to prepare the greenhouse report including independent verification (licence conditions 3.1.1–3.1.5) as a standalone attachment to the Licence Condition Compliance Annual Report. This section will also be provided to the EPA for future audit purposes.'

Mandatory reporting requirements for the IVR include the following:

• the methodology used for verification of emissions, emission reductions and ESF
• what was verified (with examples where practical)
• how and when it was verified and any assumptions made
• references of records, documents or other information used as evidence for verification
• who did the verification and their qualifications and experience.

The IVRs supplied with the retailers' greenhouse reports (see Section 3 for details of IVRs), although an improvement on the 1998–99 IVRs, did not contain all the information required by the mandatory reporting requirements. The sufficiency of information varied considerably between IVRs and was generally considered inadequate to give the EPA confidence in using the underlying data reported by the electricity retailers.

Back to top

GHG emission reduction strategies

Under the Act and in accordance with electricity retailers' licence condition 3.1, retailers were required to negotiate environmental strategies with the Minister for Energy to reduce GHG emissions from electricity they will supply to benchmark emission levels by 2000–01.

To audit the effectiveness of individual GHG emission reduction strategies, the EPA requires that retailers provide sufficient information about individual strategy measures. However, many of the retailers' GHG emission reduction strategy documents did not contain sufficient specific information. The strategies did not provide firm enough quantified targets for the action items proposed to allow the EPA to conduct a formal performance audit of the effectiveness of individual strategy measures against forecasts.

As required by the Further Environmental Guidelines, the inclusion in future strategy documents of outlines of specific measures, budgeted costs for those measures and specified key performance indicators would provide an appropriate base against which to assess effectiveness.

Electricity sales forgone

ESF is a concept developed to overcome a loss of revenue for retailers who consider helping their customers either to become more energy efficient or to generate some of their own electricity on their premises (for example, by installing a solar panel or a small cogeneration plant). The cause of this loss of revenue is illustrated by the example in Box 1.

Box 1: What is ESF? Consider a factory consuming 10 GWh of electricity per annum (this corresponds to an annual electricity bill of approximately $500,000). Given that the greenhouse intensity of the NSW electricity pool was 0.866 kg CO2-e per kWh10 in 1999–2000, this amount of electricity would represent around 8.6 kt of CO2-e emissions. Inspired by the GHG emission reduction requirement in the retailer licence conditions, the electricity retailer for this factory conducts an energy audit of the factory and identifies the potential to reduce electricity use by upgrading a piece of equipment. The factory owner agrees, for example, to replace an electric boiler with a high-efficiency gas boiler and purchase gas from a different retailer. As a result, the electricity consumption of this factory fell to 9 GWh of electricity per annum. Although the factory owner would still have to pay for the gas (which would be around $20,000), the annual electricity bill would have dropped to around $450,000, so overall the owner would be saving $30,000. However, the electricity retailer would receive $50,000 less sales revenue than previously. In terms of an environmental outcome this is a good result, leading to reduced GHG emissions. However, the retailer's revenue has fallen substantially, and clearly the retailer has suffered a financial disadvantage from having worked with the customer to achieve a good environmental outcome. It is doubtful that any commercially focused retailer could afford to act in such a manner. ESF was developed as a policy mechanism to offset this disincentive. The effect is that a retailer who has conducted activities giving rise to ESF obtains a higher benchmark, which allows it to sell more electricity under the benchmark than would otherwise be the case. This enables retailers to compensate for the lost revenue resulting from providing a service to customers that reduce GHG emissions.

The concept of ESF was first described in Further Environmental Guidelines and Requirements, Retailers Suppliers—Greenhouse Gas Reduction Strategies, July 1997, which stated that: 'For the purposes of these guidelines, "market share" means actual electricity sales (GWh) to customers in NSW by a retailer, plus electricity sales forgone through end use energy efficiency programs undertaken by that retailer, expressed as a proportion of total electricity sales and total electricity sales forgone by retailer energy efficiency programs in NSW in the same period.'

Back to top

The greater a retailer's market share is, the higher is its allowable emissions benchmark. The concept of ESF is intended to remove the disincentive for retailers to undertake activities that reduce actual electricity sales, since a demonstrable increase in ESF would lead to an increase in benchmark emissions, all else being equal.¹¹

However, there are some complications in this definition. The definition of market share assumes that all claims for ESF are equally valid, and of an equivalent value (in unit terms) to electricity sales.

Thus, in calculating any individual retailer's market share, and hence its benchmarks, an important variable is the total ESF of all retailers in the market.

As a general comment, the level of reliability of retailer ESF claims observed by the EPA during this performance audit is highly variable. In many cases the lack of robust measurement and verification calls ESF claims into question, even if they are indeed valid.

It is thus worth considering the effect of including quantities of questionable ESF in the market share calculation process. The hypothetical example in Box 2 of a (much simplified) two-retailer, two-consumer electricity market illustrates how the ESF affects market share calculations. 

For the reason explained in Box 2, it is particularly important that all ESF claims be verified to a known standard that is consistent with that applying to supply-side measures. If the auditing standard of ESF claims is lower than that for supply-side claims, the effect is to reward those with less reliable claims at the expense of those with more reliable claims.

Back to top

The EPA notes that the ESF claims are not generally of a standard to allow a high level of reliably to be attributed.

The Framework for Calculation of ESF¹² makes clear that the minimum reporting requirements are forms ESF1 and ESF6. Retailers do not appear in general to be meeting these minimum  reporting requirements for ESF claims.

In cases where significant claims for ESF arise from a single client, the Framework for Calculation of ESF recommends the use of ESF7, which is an ESF equivalent to an Assigned Generation Declaration on the supply side.

Only one of the retailers used form ESF7, although a number of significant ESF claims have been made. In some cases, claims have been based on alternative procedures (such as claims under other government programs such as the SEDA Energy Smart Business program). Although such procedures do not fall within this framework, in some cases the EPA accepts the validity of such claims.

Uncertainty as to the validity of ESF claims undermines the validity of the benchmarks as applied in 1999–2000, and this may in turn affect the compliance (or otherwise) of several retailers.

It is interesting to note comments by some retailers in their strategy documents that energy efficiency activities cannot be sustained in the competitive market.

The SEDA deeming formula and its use in ESF claims

SEDA has provided a deeming formula to allow retailers to claim ESF on the basis of historical expenditure on generic activities related to end-use energy efficiency. The EPA was represented on the working group that developed the formula. The intention was to allow retailers to be given some recognition for significant energy efficiency actions taken in the past, even though they may not have kept records to the standard required under the guidelines.

The SEDA deeming formula has become the dominant mechanism for claiming ESF under the guidelines. In some cases, a claim made under the deeming formula represents the only ESF claim made. Sometimes an ESF claim made with the deeming formula is the only claim for abatement made by the retailer.

Most retailers have not provided adequate information about the nature of the expenditures giving rise to a claim under the deeming formula. The EPA is frequently unable to determine whether the expenditure was for some relevant energy efficiency activity (such as a series of information brochures distributed to customers) or some other activity (such as a series of glossy marketing brochures seeking to sell more electricity, distributed to customers).

In general, the EPA is concerned that the SEDA deeming formula has been misapplied and overused. Originally intended as a supplementary measure to encourage generic activity related to improving end-use energy efficiency, the deeming formula has become one of the most common methods for supporting ESF claims (39%). It is very difficult to audit claims made under the deeming formula, but in a general sense the reliability of these claims appears to be low.

Back to top

Greenpower, assigned generation and other schemes

Greenpower is a national accreditation scheme allowing retailers to 'brand' an electricity product as supporting low-emission generation. SEDA introduced an accreditation scheme in NSW in the late 1990s and now acts as the national accreditation and administrative body in cooperation with other states.

In simplified terms, a low-emission generator (such as a wind turbine) is accredited by SEDA. An electricity retailer who is marketing a Greenpower product may then purchase the output from this generator. The amount of Greenpower purchased by the retailer over a 12-month period must exceed the amount sold by the retailer under the Greenpower label. SEDA organises a periodic audit of the sales and associated purchases for all accredited products. Such an audit was performed by the environmental engineering consultancy of Environmental Resource Management for the period 1999–2000. A public audit report is available from SEDA.

There is some overlap between Greenpower and the concept of assigned generation in the guidelines. Assigned generation refers to generators (whose emissions will be lower than the NSW electricity pool average, but need not be zero or negative) whose output is assigned to a particular retailer by the owner of the generator (presumably at a price).
All accredited Greenpower generators would be eligible to be assigned generators. However, not all assigned generators would be eligible for Greenpower accreditation.
The result has been that the guidelines treat Greenpower as follows:

• A retailer may claim Greenpower sales in NSW towards its licence compliance benchmark at 100% value.
• Any output from accredited Greenpower generators purchased by the retailer and surplus to its Greenpower product sales may be claimed by the retailer as assigned generation.
• If claimed as assigned generation, only power generated inside NSW may be claimed at 100% value. Surplus Greenpower purchases from outside NSW are 'devalued' using a scaling factor related to the proportion of NSW sales by the retailer compared with the retailer's total sales.

This has a couple of important implications for the current audit process:

• Retailers may claim Greenpower as an eligible measure only in regard to their sales of Greenpower product in NSW. Sales of electricity under an accredited Greenpower product in another state are of no value in this audit framework. At least one retailer (TXU Electricity Ltd) has attempted such a claim, and the EPA (and the MEU previously) has disallowed it. This makes the difference between TXU Electricity Ltd being compliant (when the out-of-state Greenpower sales are included) and non-compliant (when the out-of-state Greenpower sales are removed.)
• The national Greenpower audit report does not provide a breakdown of the sales of retailers' Greenpower product by state. As a result, the EPA has no information with which to check retailers' claims for Greenpower sales in NSW and must rely on the independent verification process. However, in practice the EPA has assumed that retailers have not made false claims for Greenpower sales in NSW, even though the independent verifiers have usually not verified these claims.

  Back to top

Energy Smart Business program

A number of electricity retailers participate as program support managers (PSMs) with the SEDA Energy Smart Business program. Under this program, SEDA pays a PSM to provide energy efficiency advice to corporate clients who have signed up with SEDA, and generally help those clients to identify and implement energy-related projects that reduce GHG emissions. The payments by SEDA to the PSM have two components—a base fee for the consultancy services, and a performance-based component on the basis of the amount of CO₂-e savings that they are successful in having their designated clients make.

The Energy Smart Business program was not designed (in terms of audit, verification and recording of claims) in a manner that is directly consistent with the guidelines concerning the retailers' GHG reduction licence conditions. Given that the retailer is paid by SEDA for delivering energy efficiency services to customers, it could be argued that claiming these savings under the retailer's licence condition represents a form of double-dipping. It could also be argued that the retailer has no right to the resulting 'credits' at all.

On balance, for the current period, the EPA feels that a claim for ESF arising from a retailer's PSM role is acceptable and reasonably robust, given the SEDA program verification requirements. However, it is an area that deserves a clear determination under future guidelines.

1-, 3- and 5-year plans that do not lead to compliance

In some cases retailers have provided strategy plans that do not target compliance with the announced targets. That is, the plans do not contain sufficient measures to achieve compliance in the stated timeframe, even if every measure in the plan was implemented with 100% effectiveness.

The EPA has (where sufficient information has been provided) attempted to audit the effectiveness of all measures contained within the strategy document against actual implementation. However, the existence of some plans that target 'non-compliance' means that a retailer could achieve a high EPA assessment of effectiveness, and yet still be non-compliant against the target benchmark. This situation simply indicates that the retailer has been effective in implementing the measures as outlined in the plan, but had insufficient measures in the plan to actually achieve the benchmark emissions.

The EPA has attempted to identify where this situation arises, but in some cases the strategy documents are lacking in the data required to determine whether this is the case.

Back to top

 

⁶ Further Environmental Guidelines and Requirements, NSW Department of Energy, 1997. Available at http://www.doe.nsw.gov.au/.

⁷ Further Environmental Guidelines and Requirements: Retailer Suppliers—Greenhouse gas reduction strategies, NSW Department of Energy, July 1997.

⁸ Guideline for Independent Appraisal for Electricity Distributors and Retail Suppliers, Ministry of Energy and Utilities, 30 June 1999.

⁹ Letter to all NSW retailers from Dr R. Bosler, Assistant Director of MEU, 9 August 2000. Attached to this letter was a discussion by the EPA of the 'information required for EPA audits'.

¹⁰ Electricity Distribution and Retail Licences: Compliance Report for 1999–2000, p. 25, IPART.

¹¹ Greenhouse Gas Emissions from Electricity Supplied in NSW: Framework for Calculation of Electricity Sales Forgone, p. 3. NSW Department of Energy, February 1999.

¹² Greenhouse Gas Emissions from Electricity Supplied in NSW: Framework for Calculation of Electricity Sales Forgone, NSW Department of Energy, February 1999.