We recently released a paper presenting the findings of a first-of-its kind, randomized controlled evaluation of the returns
to some common residential energy efficiency investments. The study’s
context is the nation’s largest residential energy efficiency program,
the Weatherization Assistance Program (WAP). You can read media coverage
of the paper here, here, here, and here.
For
those who haven’t read about the paper, between 2011 and 2014, we
administered a randomized controlled trial (RCT)—considered the gold
standard in evidence—on a sample of more than 30,000 WAP-eligible
households in the state of Michigan in order to shed some light on a
critical question: Do investments in important residential energy
efficiency measures (improved insulation, air sealing,
weather-stripping, window replacement, furnace replacement, etc.)
deliver the energy savings they promise?
The research revealed
five main results: (1) The energy efficiency measures undertaken by
households in the study reduced their energy consumption by between 10
and 20 percent on average; (2) However, these savings were just 39
percent of the average savings predicted by engineering models; (3)
There is no evidence that the shortfall in savings is the result of
rebound—households did not turn up their thermostats after the
investments were made; (4) While the investments cost roughly $4,580 on
average, our best estimate of the energy savings was about half of these
costs[1];
and (5) The costs also greatly exceeded the benefits when the monetary
value of pollution reductions are added to the energy savings to
calculate benefits. While the WAP program has a number of goals, when
measured by the energy savings and emissions benefits, these efficiency
upgrades were not a good investment.
The
urgency of the climate challenge means that it is critical to identify
cost-effective strategies that will deliver real greenhouse gas
emissions reductions. Energy efficiency is a crucial component of most
climate change mitigation plans, underscoring the importance of
developing a body of credible evidence on the real-world—versus
projected—returns on energy efficiency investments in the residential
sector and beyond.
Such a process will undoubtedly uncover some
gems, but in some instances it will also be necessary to update our
beliefs. When seemingly inconvenient evidence comes to light that
challenges our beliefs—as we have uncovered with this analysis—that data
should not be undermined and ignored. Instead, it should be used to
inform our strategy to confront climate change. The magnitude of the
climate challenge requires that we ruthlessly pursue the most cost
effective mitigation options. Our paper has generated some strong
reactions and important questions, some the result of misconceptions
about what exactly we evaluated and how the evaluation was conducted. In
the remainder of this blog, we respond to the most common criticisms of
our study and its findings.
Reaction 1: This is just one study and scores of other studies have opposite findings.
Some
critics have cited prior evaluations showing that residential energy
efficiency programs are good investments and that our study is an
anomaly. Many of these evaluations, however, are based on savings
projections that- as we found- can significantly overestimate the
savings when applied in the real world. Other studies use real- world
data, but analyze these data using methods that can confuse the effects
of energy efficiency improvements with other factors that drive changes
in energy consumption.
Our study is different. It represents a
first-of-its-kind evaluation using a randomized controlled trial, the
gold standard for rigorous evaluation. Society routinely relies on this
methodology to assess the efficacy of new drugs, treatments, and other
interventions. This approach is increasingly used in the social
sciences, including criminology, education, development economics, and
energy economics. In many instances, the application of randomized
control trials has changed the conventional wisdom. Our application of
this approach to residential energy efficiency measures is therefore an
important departure from, and improvement upon, previous analyses.
Reaction 2: The study unfairly paints WAP as an ineffective program.
WAP
has multiple goals and improving the living standards of its recipients
is clearly a central and worthy one. Our study does not claim to
provide a comprehensive evaluation of WAP, nor would it be appropriate
to do so.
Rather, the study’s purpose is to measure the real-world
energy savings resulting from WAP-funded energy efficiency
improvements. We then compare them to both the investment costs and the
projected energy savings generated from detailed energy audits.
In
interpreting the results, it is important to bear in mind that for a
measure to be implemented under WAP, federal regulations require that it
pass a cost-benefit analysis—that is, the projected cumulative
energy savings must be greater than the investment costs. This
cost-benefit analysis is based on an in-home energy audit conducted
using an engineering model, in this case the National Energy Audit Tool
(NEAT).
For the households we studied, NEAT-driven audits
projected that the WAP measures would reduce annual energy consumption
by 43.7 million British thermal units (MMBtu). Yet, when we observed the
energy bills of households that received WAP measures, the actual
energy savings were just 17.2 MMBtu. In other words, the model
systematically over-predicted energy savings by a factor of 2.5.
This is an important finding. The investments in efficiency in our study underperformed relative to projected values
and in a way that the program was expressly designed to avoid.
Homeowners, program managers, and taxpayers only received 39 percent of
the projected savings. According to the Department of Energy,
the NEAT model is used by approximately 700 state and local
Weatherization Assistance Program subgrantees in more than 30 states.
Broader
program objectives notwithstanding, WAP is a compelling setting to
learn about the returns to energy efficiency investments. WAP is the
nation’s largest residential energy efficiency program. According to the
Department of Energy, which administers the program, more than 7
million homes have participated in the program since its inception in
1976. If one is attempting to assess the performance of commonplace
residential energy efficiency investments on a large scale, there may be
no better option.
Reaction 3: The study’s calculations of costs and benefits are inaccurate.
Here
again, it is important to note that we recognize WAP has benefits
beyond saving energy. But, the intent of our study was focused solely on
evaluating the energy-related (and associated emissions) costs and
benefits. We never claim to evaluate the other benefits of these
upgrades, as that is beyond the scope of our study. It is also important
to note that, no matter how one decides to evaluate monetary costs and
benefits, a central finding of our analysis remains unaffected:
efficiency upgrades delivered just 39 percent of the energy savings they
promised. It is therefore challenging to find a set of assumptions
(e.g., about lifespans and discount rates) that would cause these
efficiency investments to have energy savings and emissions benefits
that exceed their costs. To drill down a bit more, here are some of the criticisms of our calculation of the costs and benefits and our responses:
Costs
Some
have argued that it is inappropriate to factor in costs that don’t
directly lead to energy savings. As anyone who has done home repairs
knows, once you start down the path to do something like lay new
insulation, additional costs are necessarily incurred. For example,
weatherization can reduce indoor air quality by tightly sealing a house,
so additional costs may be required to maintain indoor air quality.
Separating what’s required to lay the insulation from what’s completely
separate is not easy. The average household in our sample received
approximately $4,600 in energy efficiency upgrades, which includes
roughly $800 in costs required to make installation of the
weatherization measures safe and functional, such as wiring upgrades.
Our judgment is that the most reasonable assumption is to include all of
these installation and materials costs. It is worth noting, however,
that if we take the polar opposite view and exclude all costs that do
not directly result in energy savings, the average cost per household
still significantly exceeds our central estimate energy savings.
Moreover,
there are other costs associated with these retrofits that are not
reflected in our cost-benefit comparison. For example, we do not include
any program overhead or administrative costs. Nor do we account for the
hassle and effort that households expend to implement a weatherization
retrofit, even one with zero out of pocket costs. An earlier blog
makes the point that these process costs can be large (we found that it
cost $1,050 per weatherized household to encourage take up of these
measures). Accounting for these additional expenses would of course
widen the gap between costs and savings.
Benefits
We
measure benefits by calculating the net present value of annual energy
savings using a range of discount rates (3, 6, and 10 percent) and
investment lifespans (10, 16, and 20 years). Our estimate of the
benefits also includes an estimated upper bound on the benefits
households derive from increased warmth (based on our analysis of
“rebound” in demand for heat in the winter). In no case does the present
value of energy savings reach parity with actual costs, even if we
ignore the indirect efficiency-related improvements.
In
calculating program benefits, we used real 2013 residential energy
prices for electricity and natural gas in Michigan and assumed that
these figures would increase at the rate of inflation over the lifetime
of the investments. While some have criticized this as too conservative,
it is standard to use current energy prices as a predictor of future
energy prices.
Reaction 4: The results cannot be
generalized because they only relate to one part of Michigan, to one
program, and to one subset of the population.
We study a
subset of low income households in Michigan undertaking a particular
set of residential efficiency measures recommended by NEAT. However,
minimizing the significance of our findings on account of this context
ignores the ubiquity of the measures we analyze and of the reliance on
audit tools like NEAT.
As noted above, the households in the
sample we studied were subjected to the same measurement tool that is
used by residential weatherization programs throughout the country to
gauge which upgrades are the most cost effective; and all implemented
measures had to pass the same cost-benefit analysis. The types of
upgrades installed at the WAP households in our sample (e.g., furnace
upgrades, improved insulation, and weather stripping) are commonplace
for home retrofits for all income groups.
Drawing implications
from a study is not an all-or-nothing proposition. For example, the
results of a randomized controlled trial studying the effectiveness of a
given drug or treatment on middle-aged men will in some instances tell
us everything we need to know about its effectiveness on young women.
Of course, in other instances, less decisive conclusions are warranted
until further research is conducted.
Our study tells us that a
common set of efficiency measures installed in the low-income households
we studied in Michigan did not deliver the expected energy savings, and
that investment costs significantly exceeded these savings. Given
similarities between the setting we evaluate and other efficiency
applications, these findings likely generalize to a broader set of
residential efficiency investments. There is logic behind this
implication, while also acknowledging the need for further experiments
on the returns to energy efficiency investments in other contexts
(Indeed, we have already begun to do them and, in at least one case, our
preliminary results are qualitatively similar).
Reaction
5: The study period covered a time when the program experienced a
significant increase in funding that led to poor results (e.g.,
inexperienced contractors).
The time period we studied
included an unprecedented number of weatherizations because the American
Recovery and Reinvestment Act (ARRA) increased the amount of money
allocated to the program dramatically. As a result, some say new,
inexperienced contractors were called in to do weatherizations and their
work may not represent the norm.
To investigate this possibility,
we compared savings at homes where the contractors were experienced to
homes where the contractors weren’t experienced and found no difference
in the average energy savings. Consequently, we find no evidence that
inexperience during the time period played a role in explaining the
lower-than-expected savings.
[1]
This blog focuses on a subset of the numbers and results reported in
the paper. Here we emphasize our preferred estimates from the randomized
controlled trial that estimates average impacts for the subset of
households whose participation in weatherization was the result of
random assignment to our experimental intervention. These households
are associated with somewhat lower average costs ($4,580) as compared to
the larger sample of recipient households from whom we collected data
for our quasi-experimental analysis ($5,150).
http://www.theenergycollective.com/meredithfowlie/2247322/do-residential-energy-efficiency-investments-deliver