Okay so I see people thinking Twitter will go down as it loses engineers, fail whale everywhere. But that's not how it happens. That's the worst case as things go catastrophically wrong.

What actually happens is small fires, no longer addressed, burn into wildfires users see. What a lot of people don't know is every major platform/service/whatever has people addressing small fires every day. Fires 99.9% of users don't realize happened. Maybe a failure, maybe a bad rollout, maybe...whatever, doesn't matter. Stuff goes sideways, minorly, all the time.

Lots of replies about how building the tech isn't hard, building the userbase it.

Both are hard. And both go hand in hand, you don't take on the cost of building a high scale platform before needing it, you need users, revenue, etc. to grow as you go. They're very intertwined.

https://twitter.com/Nick_Craver/status/1587100906323169282

To be clear, most people are claiming they can *replicate* something in a weekend...which still isn't true, but bear with me.

The original had to figure out how to get to the current state. They didn't know what users wanted. They had some guesses and changed directions a lot.

I think huge scale is hard to imagine for most people because human brains rely on relations and comparisons to what they've seen before. Something that's orders of magnitude past anything you've encountered thus far is hard to really comprehend quickly. It's so foreign. And I'm not claiming to be good at this either, but I have seen enough to not make assumptions about those unknowns.

I ran a medium-scale website with 9 prod webservers. My current role deploys to millions. I still haven't quite wrapped my head about that over a year in.

"Do they have WiFi?"

"Technically, yes." I hope you appreciated that tweet, it took 6 minutes to send.

I need to do a write up on the common symptom of stampeding in web apps at some point. It’s one of those problems that isn’t obvious until you see it for what it is, and then it’s super obvious and you always want to design for it. The basics are you have say 1,000 requests per second, just some number over a few. When a thing they’re all using disappears, they’re all then waiting on you to get that thing (e.g. re-caching it). That’s unavoidable unless you want to 404.

But are they *all* re-caching it?

Wow. Just...wow.

This is why you need to elect people with any clue whatsoever as to what the hell is going on.

https://twitter.com/GovParsonMO/status/1448697768311132160

If you live in Missouri, please be sure not to view source on this tweet you damn HTML source hackers.

I think the hardest part of transitioning to Microsoft is going to be scope. Previously, I was able to fit how everything works in my head from hardware and everything above.

This is way bigger though, you have to find a scope to help in and I'm going to have to figure that out. I believe that one of the best ways I can help has tended to be looking at a wide scope, what's beneath it, etc. and using that knowledge to help inform decisions, direction, and suggestions. That scope can no longer be "everything". It's too big for a dozen lifetimes.

Finally getting a few things tidied in the office. I’ve been absolutely loving the @rackstuds I’ve bought here for home (never going back), but when they saw my rack tweets they reached out and kindly sent me some blue samples to match the network gear - not too shabby at all! @rackstuds I can't remember who recommended them to me on here, but after trying a set I'm using them for all things rack now. I'm never sponsored or anything, it's just a very neat, usable, knuckle-saving thing that I discovered and now really love. Thanks whoever showed me these!

You know what they did? They wrote our own socket server. It’s still archived on GitHub: “StackExchange.NetGain”. And it’s Hard. There are soooo many nuances. When Kestrel came along, we eagerly moved over and said goodbye to maintaining all that socket code and its edge cases.

https://twitter.com/superdalgas/status/1448530041835458564

If you’re unfamiliar with Stack Overflow’s websockets: they power all the real-time bits on a page. Live voting, top bar updates, live inbox, etc.

There are generally 600,000 to 900,000 connecting concurrently at any given time. It’s quite a few sockets that thing handles.

The only disappointing thing about joining Microsoft is my new account didn't immediately get access to the SBC training and I've heard such great things. Oh snap, I now have access. Season 1 is getting good!

Yeah, we needed this break. The beach here isn’t deserted in the evening but…pretty close.

Knowing your systems front to back and what works/doesn’t can easily be exhausting over time. People look to you on what we can/can’t easily do. Over time, as the low hanging fruit gets picked off, you inevitably find yourself stating more challenges than quick solutions. It’s a weird state over a long time because you can, the entire time, give accurate assessments of situations, yet be seen as more and more negative on how hard remaining things are. People often forget: we didn’t do X often *because* it wasn’t easy. Opportunity cost was high.

PSA: if you’re thinking of getting a Tesla, you’ll want to check out the changes or wait. They’ve removed radar from some models and other misc. things like lumbar support from passenger seats. Despite price increases, what’s being sold is worse, presumably due to chip shortage. Of note: these aren’t announced changes, they’re small tweaks to website text. They can change while you have one “reserved” since that’s how their orders work, you are holding a ticket in line, with no specifics until delivery/pickup time.

Hmmm, deciding if today is small project day. Not exactly the same bracket but pretty close - thinking a 2x12 section spanning 2-3 studs to bolt these to, then I’d need to run a new outlet by that 100 amp panel for connecting the cord reel.

I am answering lots and lots of questions about why some things timeout in cloud environment when the threshold is set at 50ms. I wish there was somewhere to point people explaining some spikes and delays will happen in these environments and that's to be expected occasionally. This isn't specific to clouds either - a VM cluster with neighbors can see the same issues. You rarely 1:1 provision hardware and have full capacity for all users at all times, because 99% of the time, that's unnecessary. Over-subscribing is normal and spikes/contention happen.

I've been remotely working for about 12 years.

The past year was not in any way normal. Kids being home 24/7 and my wife working remote is...different. The days are long and intertwined.

A dedicated space is essential, IMO. Unwinding starts by leaving that space.

#DevDiscuss

https://twitter.com/ThePracticalDev/status/1387210123542736898

It's very important to realize that some people are working remote by choice. And many more (right now) are doing so by necessity. The latter is basically an invasion of work into their home.

These are not people working at home because they want to, they're camping and coping.

I still don't think a large portion of .NET developers know that adding optional parameters is a (binary) breaking change. Requests on details here - fair enough!

Let's say you have a method in a library:
Get(string foo)

Caller: Get("test")

You add an optional param, let's say:
Get(string foo, int bar = 0)

("Type is not important here")

Without a re-compile, that's a missing method exception.

I. Hate. Cryptocurrencies.

There are no redeeming qualities. None. There’s a hardware shortage everywhere, they gobble energy supply in a completely unsustainable model, and almost everyone involved is only doing this to make money…in another currency.

https://twitter.com/justincormack/status/1386614703145852929

The entire thing is based around consuming energy and occupying hardware. The value is 100% imaginary. The system is based on a perceived value, and given people are in and out to make money off that perception: it’s basically a pyramid scheme.

Cryptocurrencies are a blight.

The last 3 days is one of the longest deep dive debugs we've had in years. We finally got the culprit for some timeouts this afternoon: the compiler. It eats more CPU in a later version, not by much, but enough. Here's a segment of our build with .NET SDK 3.1.402 vs. .NET 5.0.101 Note: we've run builds on our web tier for the past decade. Why? Because they're typically at 5-10% CPU and we don't have a lot of hardware - so we used the spare capacity. Now, with Roslyn pegging all cores in that spike, it finally hurt.

Let's walk through the past few days.

We're currently working on a way to dump all threads from the current app pool somewhere when an event occurs to narrow down the blocking issue. This is getting fun, processes dumping themselves.

I explained this a few times today, and I think it's important for every dev to know: timeouts are not necessarily what they seem.

A timeout happens when an operation doesn't complete fast enough. But, what does that mean? What are the details of that check? Well... You almost never check timeouts live, unless doing so is VERY critical and worth being a primary CPU consumer. That's very expensive. Usually, what happens is one of 2 things:
- A timeout event is queued, and if it goes off first: boom.
or:
- Timeouts are checked on an interval.