Thread by @JesseJenkins

There's a new working paper from econonomists @UChiEnergy circulating today that claims Renewable Portfolio Standards implemented in 29 states raise electricity rates and are a very costly way to cut CO2. I have some serious questions/concerns about the paper. A thread...

The paper is by Michael Greenstone, Richard McDowell, & @Ishan_Nath & can be read here bfi.uchicago.edu/working-paper/…

@AmyAHarder wrote a summary @Axios axios.com/newsletters/ax…

She asked me to comment & sent the paper which I was able to read over the weekend. I shared some concerns.

I am -- and encourage others to be -- very hesitant to treat this study as the definitive word on the cost of RPS policies. In particular, I urge any policy makers or stakeholders in policy debates to treat these findings WITH CAUTION at this stage.

First, this is an econ *working paper,* which should be read as "work in progress." Findings are preliminary & subject to change. It will go through several rounds of peer review &likely revision. Its release is an invitation for critique/feedback, NOT to be taken as definitive.

Carefully assessing the cost impacts of RPS policies is an important effort, and I commend the authors for taking up this challenge. Using statistical techniques to estimate the causal effect of policies like this is important but also extremely challenging work.

As I told @AmyAHarder, the challenge is that you have a very small sample size (only a handful of policies) and lots of potential confounding variables that make it difficult to cleanly estimate the effect. After reviewing the paper's methods, I have some questions/concerns.

Concern #1: The authors are reporting the *average* effect of 29 state RPS policies on retail rates 7 years after passage. The problem: each state RPS is unique! They're not the same event/treatment. Every one differs in substantive ways.

Each state's policy is unique in structure, definition of eligible resources, requirements for specific subsets of technologies (such as solar PV), rate of increase in renewables share required, cost containment provisions, and many other factors.

Additionally, many state RPSs were implemented as part of larger packages of legislation or regulatory reforms. Some were part of the whole restructuring of a state's electricity sector. Some were part of broader energy bills with multiple programs implemented at same time.

Treating all RPS policies as same & estimating their average effect is a bit like treating 29 patients with a dozen similar but distinct drugs, each using different doses, and then trying to say something about the average effect of all of the treatments. What does that tell us?

Was it one particular drug that was most effective, while others had no effect? Did some drugs hurt while others helped? Looking at an average treatment effect for heterogeneous treatments makes it impossible to say.

Same is true here: do a handful of policies drive the average? What does the average effect say then about any specific state? Very little. Any specific state policy maker or stakeholder should be VERY cautious taking the estimate from this study & applying it to their state.

Im maybe 1/10th the economist Greenstone is (I'm not even really an economist, although I've trained in these methods). But based on my training this feels like a serious "no no" to use standard statistical methods to study average treatment effects of heterogenous treatments!

At best, I think you could use this kind of event study with heterogenous events to estimate the *qualitative* or directional effect, e.g. to ask, do RPSs raise rates? (Answer: probably yes, not controversial). But the authors here are making specific quantitative claims.

To use another analogy, this study is a bit like studying the effect of minimum wage policies on employment and averaging every state that implemented a min wage increase in the last 30 years, then taking the average effect on employment 7 years after implementation...

...even though one state went from $5.50 to $7.00 in and raised the rate at $0.25/year, another went from $10.00 to $12.00 over 5 years, another went to $6.50 for tipped employees and $10 for non-tipped employees but exempted certain service sectors, etc. etc.

What would the quantitative result from the average effect of all of those policies tell us? Should any individual state take average result to heart? Can you even really apply standard errors to that kind of sample and expect an unbaised confidence interval for your estimate?

My view is that each state RPS policy should be treated as a unique treatment and methods suitable to studying individual events should be applied to study it's effect (e.g. synthetic controls method, which has been used to study effect of carbon pricing policies for example)

Note that the authors actually start to explore this variation in state RPS policies in part of the paper (Section 6.2 & Table 4). The authors find indication of different effects for different subsets of RPS policies.

In particular, states that required a specific share of solar PV in their policies appear to have a much higher cost than others (doesnt surprise me) & states w/competitive electricity markets appear to have lower impacts on retail costs. So clearly the effects are heterogenous.

Okay that's (more than enough) on concern #1. In sum, it is hard to take the estimated average effect here as indicative of any SPECIFIC STATE policy's costs. Policy makers in a given state should be thus be circumspect in applying the paper's findings to their own state.

Concern #2: State RPS policies are also not the only state-level policies likely to effect retail electricity prices. It is not clear to me how the paper's methods control for additional state policies implemented after passage of the RPS that would also affect electricity prices

The study uses state level fixed effects to control for differences in states that do NOT change over time plus time fixed effects by year, which control for year by year trends that affect ALL STATES equally (e.g. national trends).

The methods do NOT control is time variant trends WITHIN states. That's important. To produce an unbiased estimate of policy effect, the differences-in-differences approach employed requires no difference in time trends between states EXCEPT for the RPS policy enactment.

This is unlikely. e.g. California first enacted its RPS policy in 2002. This was followed by several big policy changes: the Global Warming Solutions Act (AB32 in 2006), which launched a slew of regulatory measures, and the Go Solar California Initiative (in 2007), among others.

In many states, the legislation or regulation enacting an RPS policy itself contained or was enacted concurrently with several other policy measures likely to affect electricity prices. Some states implemented their RPSs as part of restructuring of their whole electricity sector!

The authors present evidence that the average retail price trends across all 29 states in the 6 years prior to RPS passage dont show any time trend (after controlling for state and year fixed effects).

The average pre-period trend is their main justification for the key assumption that the trends are ALSO balanced after the RPS is enacted. But we have no direct evidence of that, and it strikes me as insufficient evidence of no time variant trends post implementation of the RPS.

If there are any within state trends or other policies that impact retail electricity prices after implementation of the RPS, than they will bias the estimated effect of the RPS policy.

Beyond this we also only see *average* pre-period trend across all states. Is this sufficient evidence? Id like to see pre-period trends in each state. They ALL may exhibit some directionality, which is simply averaged out. The average of a bunch of biased estimates isnt unbaised

Concern #3: The study really leans into the fact that given their estimates, the RPS policies appear to be very costly ways to reduce CO2. Their estimate: the effective price per ton of carbon dioxide emissions reduced ranges from $130 to $460. That's a lot. But is this right?

Putting aside whether this average estimate is indicative of any specific state policy (concern #1) or whether it is unbiased (concern #2) and assuming it is accurate, just dividing total retail cost impact by tons CO2 reduced is a very incomplete cost-benefit analysis...

An RPS policy is not a stand-in for a carbon tax. They have some overlapping goals for sure, but not directly equivalent. I know, I helped negotiate, pass, implement Oregon's RPS in 2007 (also, full disclosure: note implications for my priors here!). They have multiple benefits.

Full cost benefit accounting would consider (at min): 1. tons of CO2 reduced (as per paper), 2. air pollutants reduced, 3. risk-adjusted value of reduced exposure to fuel price volatility, 4. state economic impact (positive & negative) and 5. induced cost declines for wind/solar

The air pollution "co-benefits" are likely significant, given CBA analysis we've seen for things like the Clean Power Plan (which almost entirely hinged on social benefit from reduced pollution, rather than just CO2 declines).

Additionally, costs for solar, wind, and other low-carbon resources have fallen dramatically over the past two decades, since RPS policies began to expand across the United States.

Cost declines are a function of incremental innovation, learning-by-doing, economies of scale, all of which were driven to large degree by increasing demand for renewable energy in US and across the world. RPS policies were one important policy driver of that increase in demand.

So if you're going to write a paper titled "Do Renewable Portfolio Standards Deliver?" it is worth considering those benefits alongside CO2 declines! An attempt at full accounting of costs coupled with a narrow accounting of benefits is guaranteed to show poor cost-benefit ratio!

Those are my main concerns with this working paper. I hope the authors take these as constructive. They're first-rate economists and have likely considered most if not all of these concerns and may well have good responses. I look forward to hearing more.

I'd normally prefer to converse directly w/authors or via peer review process. But since the working paper is being widely shared in the press and on Twitter (Ive also gotten numerous emails on it today) and has a lot of policy salience, I felt compelled to share my concerns here

In closing, a few things are not really at issue here: 1) an RPS is going to be less cost-effective at reducing CO2 than a carbon price; 2) an RPS is likely to raise rates all else equal; 4) a tech-inclusive low-carbon standard is likely cheaper than an RPS too; ...

...and 5) RPS policies have proven much easier to implement (& expand) in a more diverse range of states (29, including many purple and red states) than carbon pricing policies (California + electricity sector only in 9 mostly deep blue RGGI states). That last part matters too.

important amendment. apologies for mischaracterizing this

https://twitter.com/jacklienke/status/1120352897114492928

p.s. some additional perspective from @PeteManiloff, an environmental economist @mineseconbus who has published analysis of the effect of RPS policies in the past

https://twitter.com/PeteManiloff/status/1120355419233218561

p.s. #2 What does this study mean for the pending state debates over much more aggressive electricity sector requirements to ramp up low-carbon and/or renewable electricity (e.g. 100% clean/carbon-free or 100% renewables policies)? Some thoughts...

https://twitter.com/JesseJenkins/status/1120361579369967617

Important addendum: Authors DO try to account for some possibly confounding state policies. They specifically use databases on presence of net metering policies, voluntary green power purchase programs, and public benefits charges. Those are the only other policies controlled for

p.s. #3, This is what I mean about tremendous variation across RPS policies, such that discussing average effect is not very informative (and for any individual state may be quite misleading). Here is LBNL's estimate of direct compliance costs across states (ht @RichardMeyerDC)

This is from this LBNL report eta-publications.lbl.gov/sites/default/… & excludes indirect costs/savings such as wholesale price suppression (merit order effect), "intermittency costs" (as per the paper), and transmission costs (if any). But it gives a good first order pass at the large variation

A couple of states are clearly costly outliers, particularly MA & NJ, which included "carve-outs" in their RPS that required smaller-scale solar PV at a per MWh cost an order of magnitude higher than other renewable options (I'll join the economists in grumbling about these!)

I would also note significant differences in costs of compliance in states w/access to similar resource quality (e.g. NY vs NJ or MA), which underscores how different RPSs can in cost-effectiveness due to heterogeneity in policy design. These are NOT the same "treatment effect"