FAANG promo committees are killing Kubernetes: A Short Thread 🧵 For those outside my teeny tiny social and media bubble, "FAANG" means "big tech company" (originally Facebook+Apple+Amazon+Netflix+Google, now a bit broader), "promo" means job promotion, "promo committees" are the panels which decide who gets promoted and who doesn't.

Some thoughts on Elastic and the SSPL: this shit is hard. I agree with other voices that this was counterproductive and a betrayal of the community but I get it. To better explain things I think we need to split Elastic into two pieces, the steward of an immensely popular project and a VC-backed startup providing hosting services. In their role as steward, the relicensing leaves their community in a terrible place.

I've seen a bunch of "it's not that much spam, just close the PRs, being welcoming is important" takes about Hacktoberfest so I want to share why it makes me so cranky. Background, I'm a long-time maintainer, currently mostly on Kubernetes (a very big and visible project). It's not the individual spam PRs that bug me. Yes it's annoying to have the noise, and it's even more annoying we have to pay for the CI time every time someone opens one, but we could survive that. If it was a bunch of new users really wanting to contribute, I would make it work

So, Docker Inc has finally updated the FAQ for their their previously announced service limits. And I'm not going to lie, it's pretty brutal. You should consider any (unpaid) use of Docker Hub to be an operational risk going forward. The anonymous pull limits have been clarified to be by IP, so anyone running container tooling behind a NAT expect to hit those very quickly (100 pulls per 6 hours). And image expiry will apparently be per-version.

So putting aside the "I am altering the deal" of deleting images that haven't been used in 6 months, Docker added a "Data Transfer" limit section to the pricing page. I haven't seen this mentioned anywhere else. It wasn't there in the last Wayback scan from July. Is this new? And if it is new, what does it mean? Is that 100 pulls per IP? Per image? Because it reads like that's per image and if so that makes Hub 100% not viable as a public resource for open source projects.

It's time for some Friday night thoughtleading. I see a ton of people asking for help on Slack/SO/Twitter/etc with a Kubernetes webapp where each user gets their own container where they can do something. Please don't do this. In the very bad cases, some folks want to use Kubernetes as executable sandboxing. This isn't 100% impossible but it's very unrealistic for almost everyone. Container escapes vulnerabilities happen, cross-service escalation attacks happen, bitcoin miners happen.

Since I just spent 2 days explaining this to a Slack channel, I guess it isn't well known:

Probably don't use CPU limits in Kubernetes.

The Linux cpu quota system has had intermittent bugs basically forever and unless you do a lot of recon, it's hard to know if it's busted. Because of how the CFS scheduler works, they are rarely needed. Even under load, you'll get generally reasonable behavior if your CPU request values are correct because those are set as your cpu.shares for the cgroup.

Some Friday musings about the state of on-prem CI tools and Kubernetes. Right now I mostly use CircleCI and they are great but $$$, some mobile teams on Bitrise instead and probably staying there since mobile CI is a special hell. 1/9 Jenkins, the big daddy. Great k8s integration, A+ pipeline support, but really really hard to maintain in any kind of automated way. Jenkins wants to be the automator, not the automatee. 2/9

Copying this over from my work Slack since maybe it will help others too:

I'm sure the management team is working on advice and resources but I wanted to post up some of my recent favorites bits re: WFH for folks that aren't used to it. beau.blog/2020/03/remote… is from the Wordpress.com team and has some solid specifics about creating a workspace and schedule. circleci.com/blog/maslow-s-… from the CircleCI team is a bit more general, covering overall needs and structures.

Putting up a batslgnal. Rackspace terminated their support for FOSS projects at the end of 2019 and apparently didn't tell me (or anyone?), and is now saying that I owe them $850. I've stopped all my infra with them, but "non-payment accounts are able to be sent to Collections". Any remaining Rackspace folks (or ex-Rackspace) that can help sort this out? I don't appreciate them trying to strongarm me because they decided to change the terms of a deal without notifying me.

I don't normally get into this stuff on public Twitter but I am mad so here goes: Yes supply chain ethics is an important thing to think about and discuss as a culture. No, it's not okay or helpful to center it in the current software ethics discussion. To "name and shame" projects used for bad ends but that are clearly in no way directed at those ends or at the bad actors using them is not just ethically bullshit, it's abuse of a position of power in our community.

If your office/conference party planning team doesn't have at least one introverted, quiet-loving person on it, you are probably about to throw a party that is very stressful for a lot of engineers. I know this is not unique to engineers, but I've found a much higher rate of people with sensory overload issues there than in other sub-specialties. Music does not have to be "feel it in your chest" loud for a professional party. Background fill is cool, but turn it waaaay down.

Tonight I completed our final customer migration on to Kubernetes. We moved from an AWS+TF+Ansible flow to a custom operator made with kubebuilder and ArgoCD (and many other community tools and integrations). It's been a long journey. A year since our first internal tests, 6 months since our first customer migration. I owe a huge debt of gratitude to @directxman12 for both kubebuilder itself and answering my truly strange questions. We've reoriented our whole Infrastructure team around custom controllers.

About 80% of questions I see about Kubernetes make want to wave my arms and say "Kubernetes is not magic". Networking is just normal Linux networking with some virtual adapters, containers are just some processes with special flags, storage/mounts are basically unchanged. It's a great set of tools and abstractions, but they are so leaky you really need to learn most of the underlying Linux admin goop still. And that's okay, a stronger abstraction would make production outages hell. But how do I help someone internalize decades of Linux knowledge?

So, Slack has finally done something about /me so I guess I can tweet about this as responsible disclosure. While they may have also removed it due to low usage, my gut says the vulnerability in it was a factor. Up until today, you could use "/me (at)everyone" or "/me (at)channel" to trigger those notification groups even if they were restricted to admins or owners only in your team settings. We originally reported this to Slack a month ago.

No really, this thread is important. All of it.

https://twitter.com/laserllama/status/1091710068234641408

The longer I spend in DevOps (whatever that even means anymore), the more I'm convinced that a very large number of the "best practices" are just a slow death. Infra work doesn't schedule well, it's unpredictable, shipping MVPs rarely works out well, and you need to be fluid.

@dustinmm80 "The Secretless Broker uses the identity credential to obtain secrets which allow access to the Target Service." There is always a secret. Adding a network layer intercept vastly increases the complexity of the runtime path for the application. @dustinmm80 That might be worth it, but probably not just for this. Compare to something like Istio where you're using the same network plumbing for everything :)

Since I already posted one tweetstorm today, why not another? CCP Games and Pearl Abyss: A Thing. It's no secret that persistent-world MMOs in the model of WoW, EVE, GW2, etc are on a down slope. Not a steep one, but especially in The West (NA, EU, AU) that model is getting increasingly hard to pull off.

It's been a while since we had a "Coderanger talks about the law" so let's discuss the new California pretrial services law, aka no more money bail. First disclaimers: I am not a lawyer or politician and might be wrong about any of this, check responses for corrections. Next, what even is the bail system and why does it suck? @LastWeekTonight did a whole thing about it in 2015: . Last week California passed a first-of-its-kind law to completely disband money bail at the state level, to take effect in October 2019.

So, follow up on HTTP servers. Nginx is the most common thing to use, but is a big complex tool with a large footprint. thttpd is not bad software but mostly unmaintained and aging C code gives me chills. Go's included HTTP server got some shoutouts but is actually terrible (socket data copy is purely userspace for no reason that I can discern). Final vote seems to be for a custom build of Caddy with most of the plugins disabled.

DX world problems: when making a CLI tool, I have to choose between Python/Ruby/JS (high-level, solid frameworks) and Go/Rust (low-level, Go has terrible dep handling). Is there a better answer? Of those options, Rust seems like the clear winner, but it's much earlier in terms of ecosystem and while not C levels of "low-level", it's definitely not Python/Ruby.