Thread by @Scott_Helme

Let's Encrypt identified a bug in their CAA checking and disabled issuance for 2h 12m whilst they patched: community.letsencrypt.org/t/2020-02-29-c…

As a result of this, Let's Encrypt will be revoking quite a large number of certificates: community.letsencrypt.org/t/revoking-cer…

The total number is 3,048,289 and you can download the list of serial numbers that will be affected here: letsencrypt.org/caaproblem/

For those unfamiliar with CAA you can find more details in my blog post here: scotthelme.co.uk/certificate-au…

Let's Encrypt identified the bug themselves and have reported the CA incident themselves: bugzilla.mozilla.org/show_bug.cgi?i…

They are also seeking an exception to not revoke the affected certificates given the number of certificates involved and the low likelihood of mis-issuance having taken place: bugzilla.mozilla.org/show_bug.cgi?i…

To answer some common questions I expect, I’m not too concerned about the risks here. Reading the incident report it seems the chances of something bad happening are exceptionally low. It can’t be stated for sure whether a mis-issuance actually took place.

Let’s Encrypt are handling this in exactly the way I’d expect them to, with 100% transparency and accountability. This isn’t a reason to stop using Let’s Encrypt. If anything, this is a reason to *start* using Let’s Encrypt.

After some further analysis on the list of serial numbers, it seems that 23,657,951 unique FQDNs are affected by the mass revocation.

cat ~/Downloads/sorted-by-account.txt | grep -oh "\[.*\]" | sed 's/\[//g' | sed 's/\]//g' | sed -e $'s/ /\\\n/g' | uniq > log.txt

dropbox.com/s/tp47jrpge6br…

As for guidance here, if you have received a notification then you need to renew as soon as you can. Just run whatever process you normally run and make sure you force it to renew. If you're not sure if you're affected, renew anyway as a precaution.

Possibly worth noting that while some of these numbers are really large, it represents only ~2.6% of currently active certs that are impacted. Within that ~2.6% there are *significant* numbers of duplicate certs with the exact same CN/SAN list but a different serial number.

Of the 3,048,289 possibly impacted certificates, there is a small selection whose current CAA records would have forbidden issuance. Those are listed here and as fate would have it, there are 443 of them! pastebin.com/RHXhjdyu

It's worth noting that this doesn't prove these certs were mis-issued, and it doesn't prove they weren't mis-issued either. It's just an indicator of the extremely low probability of a mis-issuance having taken place. The right course of action is still to revoke them all.

Wow there are a lot of configuration errors in those CAA records too. Example: using "letsencrypt.org " instead of "letsencrypt.org" which means Let's Encrypt would prevent issuance!

If you're interested in how widespread these errors are in CAA configs, here are the CAA record sets for the top 1 million sites in the world! There are quite a few... crawler.ninja/files/caa-valu…

Whilst on the topic of another mass revocation event, remember that Netcraft estimated 500,000+ certificates needed to be revoked as a result of Heartbleed and tomorrow Let's Encrypt is going to revoke 3,000,000+ certificates.

If Let's Encrypt ran a CRL, it'd certainly be one massive file by tomorrow evening! Whilst revocation is largely broken and does not work, for clients that are under attack, revocation is still effective and sites will probably break.

Timely that I published a blog yesterday with all the information on how and why revocation is broken, and how we might possibly fix it: scotthelme.co.uk/crlite-finally…

It'd be cool if someone could comment on the impact that a mass revocation like this would have on CRLite. I know Let's Encrypt isn't currently included as a result of not publishing a CRL, but would a mass revocation event have a significant impact on the size of the filter?

Let’s Encrypt have postponed the revocation until 2020-03-04 20:00 UTC (3:00pm US EST). You still have time, get renewing now!
community.letsencrypt.org/t/revoking-cer…

I've created a blog post with information about the Let's Encrypt revocation taking place today so that it can be shared with those who may not be on social media platforms. I will keep that post updated along with this thread. scotthelme.co.uk/lets-encrypt-t…

I'm currently burning 32 CPUs and 192 GB of RAM to scan all 12,283,290 affected hosts as quickly as possible to see who still has to renew. Scan will take ~32 mins and I will publish the list. Will also update the list throughout the day as we approach 20:00 UTC.

The resources for this (CPU/RAM/💲💲💲/⌛⌛⌛) are coming out of my own pocket so if you would like to support me in this and other research, there are links for PayPal/Patreon/Flattr/Bitcoin/etc on my blog! scotthelme.co.uk

Fast approaching 30k sites that still need to take action with ~9 hours to go. Any ideas on how we can get the message out once I publish the list?

To expedite the scan I'm not scanning a site (eTLD+1) multiple times if they have multiple subdomains or multiple certificates. If the site owner is notified they can then replace all Let's Encrypt certificates. The focus here is to get them notified.

The first batch of data is available and please note the scan is not yet complete, the lists are not exhaustive! github.com/ScottHelme/le-…

So far, ~10% of all affected sites indicate @cPanel issued/managed certs ("cpanel." subdomain in SAN). Are these likely managed or self-hosted and is @cPanel aware and preparing renewal?

Here are the Top 10 affected TLDs so far:

211 government certs still in use and the scan isn't complete yet: pastebin.com/qP7F4aPf