Share this page!

Scott Helme Profile picture
Twitter logo
29 Sep, 59 tweets, 15 min read
🚨🚨🚨 5 minutes until the Let's Encrypt R3 intermediate expires 🚨🚨🚨

29 September 2021 19:21:40 UTC
TANGO DOWN 😅
Are we still here?
I just refreshed the page on my Chrome/Windows 10 Pro install and it automatically built a new certificate chain around the expired intermediate! It's awesome when it works 😎
The problem is clients that don't have this option available are now possible seeing certificate errors on connections. Let's see if any reports surface...
If something didn't already go kablam, the DST Root CA X3 expires in a little over 18 hours, then you're free and clear!

Sep 30 14:01:15 2021 UTC
I'm getting reports of issues on devices where we didn't expect any problems... Please report any issues you're seeing on this thread 👍
A fix for Android versions >=7.1.1 if you're seeing errors for the expired R3 intermediate certificate is to kill your running apps and reboot the device.
I now have multiple reports that Bluecoat and Palo Alto proxies encountering the expired R3 intermediate will fail and refuse to connect. Even if you have a modern client behind one of these proxies that could otherwise work around the issue, those sites will be unreachable.
Unconfirmed: Cisco Umbrella Secure Gateway Web Proxy is currently reporting service issues that started ~the same time as the intermediate expiry. Looking for confirmation the two events are related: status.umbrella.com/#/detail/582
There are also many reports of iOS and macOS versions newer than expected seeing issues on sites serving the expired R3 intermediate. I've seen errors on iOS 11, 13 and 14 along with several macOS version only a few minor releases behind current. No fix on the client side yet.
🚨🚨🚨 14 hours until the IdenTrust DST Root CA X3 expires! 🚨🚨🚨

Sep 30 14:01:15 2021 UTC
Issues likely being caused by bad certificate chains are starting to surface, Catchpoint have a incident open: status.catchpoint.com/incidents/f5yl…
The @guardianiosapp (Guardian Firewall) is currently experiencing an outage:
Although, the DST Root CA X3 hasn't expired yet. Presumably this is being caused by using the older R3 intermediate?
Also confirmed issues for @mondaydotcom serving the expired R3 in their chain:
Hints at impact in other applications:
Issues in OPNsense require a manual patch and renew: github.com/opnsense/plugi…
I've created a test site to help identify issues with clients. If you can connect to expired-r3-test.scotthelme.co.uk then your client can handle being served the expired R3 Intermediate in the server chain!
As you can see, my Chrome on Windows 10 is able to build an alternate trust path through the valid R3 Intermediate to the ISRG Root X1:
Using openssl s_client or similar you can see that the sever is ending the expired R3 Intermediate in the chain:
Of course, @ssllabs is also detecting that my chain is invalid and showing the alternate, valid chain that can be built: ssllabs.com/ssltest/analyz…
@ssllabs 🚨🚨🚨 30 minute warning 🚨🚨🚨

IdentTrust DST Root CA X3 Expires:
Sep 30 14:01:15 2021 UTC
Is this thing on?
Anyone seeing issues with Google Up Time Check?
I've now had several reports of GCP monitoring reporting down across the board.
Womp Womp
OVH also having issues: travaux.ovh.net/?do=details&id…
I've seen a few providers pointing to problems with Auth0 services too, still looking for more solid confirmation.
Shopify have now joined that chorus with an incident: shopifystatus.com/incidents
Also Xero, not yet confirmed as related, but the timing is right: status.xero.com
And yes, I'm also nervously watching my own status page 😅 status.scotthelme.co.uk
Problems with apt would be not good, can anyone else confirm?
It seems that Fortinet are also having issues with Fortigate (the Next Generation Firewall) old.reddit.com/r/fortinet/com…
Many reports of issues connecting to Heroku API endpoints, but status pages doesn't seem to indicate any but metrics is broken right now: status.heroku.com
InstaPage also has an active incident where the timeline fits for it to be certificate related: status.instapage.com
ref:
Get serious when it hits the crypto(currency)! Possible related:
Can @Netlify confirm if this is related to the root expiration? netlifystatus.com/incidents/l6rk…
Seems to be the same issue that @Cloudflare pages are having? cloudflarestatus.com
It seems to be: answers.netlify.com/t/ways-to-work…
I didn’t flag this earlier, but this is a good result:
MailGun too: status.mailgun.com/incidents/brl5…
Bad day for email too, not just the Web.
If I see one more support article that contains some variation of "disable certificate warnings" I'm going to cry... 😥
Seems like a small issue at Facebook too:
The Ghost of Christmas Past: kb.fortinet.com/kb/documentLin…
To clarify, I understand people needing to do this now to make stuff work, it's how many people will forget to change the setting back that worries me.
OpenBSD 6.8 and 6.9 have dropped patches to “Compensate for the expiry of the DST Root X3 certificate”.
openbsd.org/errata68.html
openbsd.org/errata69.html
h/t @juhasaarinen
The Slack outage was unrelated to this event in the end:
Slack update: status.slack.com/2021-09/06c1e1…
OPNsense have release a hotfix to patch their issue:
Sophos SG UTM impacted: support.sophos.com/support/s/arti…
Confirmed as related:
That's a pretty big error message from cPanel! support.cpanel.net/hc/en-us/artic…
Another big one on AWS too: status.aws.amazon.com
They got DigitalOcean 😥 status.digitalocean.com/incidents/rbdc…
PostMan app had issues also: github.com/postmanlabs/po…
Voluminous devices also need an update “due to expired certificate”: community.volumio.org/t/volumio-chan…

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Scott Helme

Scott Helme Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @Scott_Helme

Scott Helme Profile picture
29 Sep
Working with @spazef0rze is never dull... 🤣 Image
Sadly, this change did not pass our stringent review process. Image
Well.... I really did ask for this didn't I... 🤣🤣 Image
Read 4 tweets
Scott Helme Profile picture
29 Sep
I bought a phone from a large retailer here in the UK and they shipped a faulty unit. These things happen, so I return it for a refund and they got it on 6th Aug: Image
They had no other phones of the same spec anyway so they said they were going to refund me. By 13th Aug, still no refund. Image
I chased a couple more times and by 14th Sep, still no refund! They say it will now take them 3-5 days to issue a refund: Image
Read 11 tweets
Scott Helme Profile picture
23 Aug
Are you using CSP on your website? You might be getting a patent infringement notice! Buckle up 😎 scotthelme.co.uk/i-turned-on-cs…
We're already working with the @EFF who will hopefully be able to support the cause here, but we need to know about other websites that have received this letter.
If you're legally and/or technically minded, perhaps you could take a look over the letter being sent out: drive.google.com/file/d/1p63IJ6…
Read 21 tweets
Scott Helme Profile picture
16 Nov 20
@BritishGasHelp @srobertson92 A few things to help you out from your friendly British security researcher:

1) Shorter passwords are easier to remember which is what makes them weak and easy to guess. This means it's more likely someone else will have access to it, not less likely.
@BritishGasHelp @srobertson92 2) Allowing someone to have an easy to remember 8-10 character password doesn't mean you need to prevent someone else from having an ultra-secure 64 character password. It's possible for both of these things to coexist, and they should.
@BritishGasHelp @srobertson92 3) Weak passwords do not protect customer data, they do the opposite and put customer data at risk. We should be encouraging stronger passwords and the use of password managers.
Read 7 tweets
Scott Helme Profile picture
16 Nov 20
There's been a lot of discussion about OCSP again recently after the Apple incident caused by Big Sur. I've written up some details about what happened and thoughts for what we could/should do about it: scotthelme.co.uk/deja-vu-macos-…
Apple published a support article to address the concerns raised, here are the details and my update based on their comments: scotthelme.co.uk/deja-vu-macos-…
Apple will introduce "A new encrypted protocol for Developer ID certificate revocation checks" but are we talking OCSP over HTTPS or something else?
Read 5 tweets
Scott Helme Profile picture
2 Sep 20
I'm not sure what's more worrying, that CAs have continued to issue certificates for >398 days or that I'm not surprised that it's happened... 🤷‍♂️
Imagine buying a new certificate that looks like this!
NET::ERR_CERT_VALIDITY_TOO_LONG
Here's the certificate, they definitely missed the deadline:
Validity
Not Before: Sep 1 00:16:16 2020 GMT
Not After : Sep 1 00:16:16 2022 GMT

crt.sh/?id=3318010380
Read 9 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

Like this author's thread?

Get emails when @Scott_Helme has new unrolls!

Check out other threads from @Scott_Helme!

Read all threads by @Scott_Helme

:(