(The reason I go through comments on HN/etc. is that for every 6 "the author is a platypus" and 3 "amen" there's 1 person genuinely thinking about this for the first time and it's a worthwhile exchange)
Kudos to all the geniuses who think my get rich quick scheme is to argue a nuanced viewpoint every couple years and being on the receiving end of a bunch of hate mail and a bill for the bandwidth.
Y'all got me all figured out.
I love how the Go team can say "we literally don't care about language design, anyone who does is a nerd and we don't hire nerds" every occasion they get for eight years straight and when I say "they didn't design a language" suddenly I'm the asshole
see, it's not just me.
unflagged after 2 hours. the cycle is complete
flagged again, removed from frontpage. nothing to see here, carry on!
reasoning for taking the article off the front page:
I don't know, maybe if your community rips itself apart two days in a row reading articles /I did not post there/, the problem isn't the articles.
This isn't censorship, they can do whatever they want on their site but it did seem like a LOT of people wanted to read/discuss this
see, this is what we have content warnings for:
the rules of this city are very unclear
fasterthanli.me is sustaining a DDoS attack: 6M requests in the past 2 hours, mostly from China and the US (they're just hammering the front page).
I had to turn on Cloudflare's attack mode, sorry for the javascript challenge.
Although most of it is static /content/, my site works more like a CMS: there's DB queries (sqlite) involved, templating, HTML rewriting, etc - here's what the server is spending time on.
I never needed to add caching (despite HN) until today 😊
There's about 40K concurrent connections right now fighting for access to a single SQLite database, it's time to set up some rate limiting too.
250K current connections now - server process is only using 800MB of RAM, which is nice. Setting up rate limiting soon, but I'll have to limit connections in some way as well. Luckily I've written that kind of code often before 😊
Oh, would you look at that, it's back up and fast again.
oh it was very complex, I'm gonna have to use a picture for this
(joke aside, cloudflare is dealing with most of the nonsense here)
oh, cloudflare does not cache static HTML content by default hahaha was that always like that? guess they have to get people to buy page rules
well, the attack stopped for now. if it picks up again I'll have to implement actual caching on my side, since cloudflare doesn't let me do what I want (cache HTML, only for logged-out users). but for now I have other stuff to do.
The attack came from Singapore, Los Angeles, London, Hong Kong, Frankfurt. Using 4 Chrome-like user agents. Mostly resulted in 503/403/499/524.
More details on the the attack & how cloudflare reacted to it (from their Security graphs)
Welcome back DDoS kids! I'm ready to take care of that next wave, let's see how this go.
To whoever is behind the attack: thanks for the free load testing service (doing it myself is against the ToS of several things), this is costing you more than it's costing me and I'm getting entertainment + experience hardening my stuff against it. A++
Hey no come back! the attack stopped too soon, I was just about to deploy a change :(
Just to show you what I'm seeing:
attack: 350K connections
no attack: 5K connections
Ah yes I forgot cloudflare doesn't know how to close connections so a 128 limit means the site is unavailable after just a little while. I need me some idle timeouts.
Love to pay for slowloris as a service.
attack is pretty well distributed, I'm not gonna block a whole country over it though
the logical next step is adding some observability to this codebase (and then some actual caching) but I'm gonna chill for a bit and maybe do that later
the problem if I add observability as-is is that I'm gonna blow through the @honeycombio plan limits
unless somehow those limits were lifted for the next 48hrs or so (🙏) I'd be blind until the next billing period so that's not good. (but today's the 30th? idk)
note: this is ABSOLUTELY NOT the "go team" or the "go community" reacting to my article. this is one person, a small discord, or some channers having their fun.
the go team got sad and blocked me, they'd never do a DDoS lol
front page of HN, day 3 😬
big bookclub energy
in all that commotion I hadn't noticed my video platform was attacked too
the attacker left me a lil' message in the user-agent
oh duh, my SQL code 1) doesn't cache prepared statements, 2) runs blocking code in an async context (should use spawn_blocking)
I missed a bunch of low-hanging fruits in the initial implementation, this'll be easy to fix.
also telemetry is all set now - gotta cache that feed!
this is why I love observability: I'm not happy about these SQL queries but I had no idea I was spending so much time truncating HTML.
there's a bunch of trivial ways to fix this, which is great news!
I thought my load testing tool broke, but no, it just completes instantly now
So today, even though it's the weekend, I was planning on landing some open source contributions, but fate had other plans...
My lovely wife came up to me and asked if I could help her open a little blender thingy we bought a few months back, when she wanted to start a (1/8)
habit of having a daily morning smoothie.
Except, the habit had been put on pause for two weeks, and the portable blender/jar thingy was now sealed shut — smoothie still inside — and despite our best efforts, wouldn't open.
I deployed all the brute force available, (2/8)
twisting and turning and grunting. It wasn't really about the display of masculinity: my wife (who, in her infinite wisdom, repeatedly told me that it wasn't a big deal and that we could take care of it later) had presented me with a puzzle, and I was damned if I wasn't (3/8)
One papercut right now is that executables built within the sandbox refer to a bunch of /nix/store paths, and apparently that makes nix think it depends on compile-time (1/4)
dependencies at runtime.
So, there's a "remove-references-to" step at runtime, that transforms these strings to some dummy value. The problem is it does one sed invocation per binary per dependency. So if, like me, your executable is pretty large, and depends on a lot of (2/4)
crates, it gets really slow.
I suggested a custom-built solution in Rust that does exactly what's needed, but there's some gains to be had with a sed-based solution in the meantime.
I'm still exploring nix to build my own stuff (and later generate containers, then look (3/4)
The low-level lib.rs/crates/io-uring crate seems like a good foundation, but I don't trust any of the higher-level crates in existence right now. You gotta fully internalize that as soon as you submit an operation, you don't own the outputs AND the inputs anymore.
Thing is, this bug isn't even an "unsafe Rust is tricky" bug. It's a "the lifetime of this C API is unclear", and a "we haven't gotten around to using memory sanitizers" yet bug.
I'm not even sure a memory sanitizer would've caught it, since the kernel is doing the reads/writes.
Okay my evil video streaming plans are very slowly progressing as expected and I'm still managing to do stuff on the side (there's lots of sides). I'm even getting some help, but not talking about it much yet because this is all very new and scary.
I also managed to get audio that doesn't make me want to walk off a cliff FOR NOW, I'm sure I'll hate it in a month – remember how I said I liked my last video for now? I hate it already, the echo and overall voice treatment is just unbearable. Thankfully the content holds up.
For those who are wondering, my "fix" for audio was 1) all the towels in the house 2) mixing a lav mic AND a condenser mic 3) very pricey izotope plugins
Later on I'll have other mics to play with and lights that don't go BZZZZHECKYOUBZZZZ but for now that'll do.