Thread by @hillelogram

Day two of #StrangeLoop! I was doing the swagway track for a bit so came five minutes late to my first talk, "Building Observability for 99% Developers", by @jeanqasaur. We've been internet friends for a while, but this is my first time meeting her in person!

@jeanqasaur Jean opened by talking about how she left the Bob Harper software cuult that wanted to prove everything correct, and is now interested in making observability more accessible to the majority of developers who don't want to spend weeks learning a piece of tech

Proving code correct becomes unfeasible with hundreds of millions of lines of code. Then there's the rise of microservices: "once you cross the network you subvert all your guarantees". Same with APIs and SaaS. No more high level spec, everything's about component interaction

"People become programmers because they like to play god." Can't do that anymore. Observability (o11y) people think that testing happens in production whether you like it or not, so might as well learn to do it well

#StrangeLoop

Over time, more and more of our work is integrated with maintenance. And software tenures are becoming shorter; best practices that were designed for long tenures are less tenable.

[I sneezed a couple of times and Jean said "bless you", srry for talk interruption]

[Now turning my sneezes into coughs to Jean doesn't notice]

Type systems and linters do not exist with third-party APIs.

According to @mipsytipsy, after deployment, we're not testing code, we're testing systems. Biologist, not god.

Won't go deep on o11y, we're throwing the three pillars out the window

[Lots of Taylor Swift, on brand]

Developers don't necessarily know all their endpoints. Lots of reasons why logs/metrics/traces aren't necessarily viable for people

There's a LOT of developers outside the ones we think of. WordPress is incredibly popular. Maybe we should be thinking about them too.

[Man this talk is making me itch to give live talks again. Workshop doesn't count. I wanna be speaking to 100+ people at a time]

#StrangeLoop

This gives us the title of our talk: the "unseen 99%", writing PHP calendars, VB6 in Mexican factories, ASP.NET in a small office

[This is quoting Scott Hanselman btw, she's not plagiarizing, no dramatic irony for me]

Modern o11y is good "power tools" for expert teams. It's good if you know your system well, it's worse if you don't know it well, or if you weren't the original builder.

We assume tech stacks are homogenous. "How many of you heard "GraphQL makes all problems go away""

Tooling needs to work as widely as possible across heterogeneous system components

Now going to talk about what @akitasoftware does for mainstream observability.

"Observability itself is a very daunting term. I'm in the market for a better term."

#StrangeLoop

Goal: how far can we get with no instrumentation.

Sol1: eBPF-based [ebpf.io] passive traffic listening. Drops into system without requiring code changes or proxies. Pretty widespread approach people use, since it's noninvasive. But traffic info is overwhelming

It's very hard to do better than that, though. So sol2 is automatic traffic modeling: inferring API properties from the traffic, like structure and behavior.

Based on compiler and program analysis techniques, instead of machine learning. Hell yeah PL theory!

Anyway, you can join the Akita beta at akitasoftware.com

Lessons:

- Modern software is very different from old software
- "Shift right, and shift right better."
- We should make tools for the non-experts that don't require them to become experts

Q: "How do you handle encrypted traffice?"
A: This is really hard. Tried parsing TLS metadata, but that sucked, so they're experimenting with colocation and agent proxies. No easy answers.

Q: "How different is this from network intrusion detection?"
A: Security people have been in this world for a while, we're learning a lot of lessons from them. We can enhance it with more modeling of nuance, not just binary detection

Q: "Does this work on Windows?"
A: "Surprisingly, yes"

Q: "Is there still room for partial verification?" [mine, from Twitter]
A: "Every tool has an audience: still need space shuttles or cars. But I'm mostly interested in website apps. There observability is higher impact"

Q: "How far back do you consider the 99%? Does COBOL count?"
A: "Everybody should consider the WHOLE 99%. Right now we're considering, like, the top 15%. We only have so many resources"

#StrangeLoop

----

Going into the "stop worrying dead programs" talk, it's PACKED.

can't sit down, gotta tweet from phone

Wait I can type from on top of a trash can.

Okay we're going through a program to increment every element in a list. First showing an ASM codebase, then C. "C works but has lots of unecessary stuff. I need the length of the array!"

"There are only six lines and only one matters! It's simpler in LISP, and then even simpler in Haskell! In APL it's even simpler, because we have row polymorphism!"

[That's, uh, not row polymorphism]

"APL is 8 years older than C!"

[Missed a bit because a guy got mad at me]

"But we still don't have anything that does this for state and time! But Clojure fixes this!" BWith immutable data structures, "atom", and STM. All very nice. But core.async sucks apparently. Go is good for concurrency, but has problems. Erlang gets almost EVERYTHING right!

"Whatever you use should be at LEAST as good as Erlang!"

They built a spreadsheet thing in clojurescript called "Cells" and then extended it with "Maria.cloud" into a notebook like thing

#StrangeLoop

Next comes propagators, which uses actor computation which inspired Alan Kay's smalltalk!

[OH NO NOT THE ACTOR MODEL OF COMPUTATION]

Anyway propagators uses dataflow and stuff

But a bigger problem is nobody's done good program representation

Originally we had punchcards, and then teletype machines, which became VAX machines. Then there's the ADM which is what inspired the vi keys because that's where the arrow keys were, because it controlled the teletype

(He got the history wrong)

"Modern Unix is COSPLAYING as a PDP-11!"

[Applause]

Anybody who approves of text isn't thinking about REAL text like beautiful inscriptions. "I have this to say to those people: 💩"

"Visual channel is important, we don't use it!"

Text is bad for graphs with cycles, and there are lots of things that are graphs with cycles. Pictures are better. We should be able to draw pictures when we program as well.

Finally, we ignore "pragmatics" #StrangeLoop

Pragmatics is relationship between program and user of program. The only programming we should be doing is interactive programming, or live programming. No compile/run cycle, persistent state, no halt-on-failure. Gives you short feedback loops.

"What about types? This is where half of you storm off in anger." Types are fun, but speaker didn't experience any defect reduction in static programs, and empirical studies show no difference. It's fun, but there's no moral high ground.

Will make an exception for model checkers since protocols are hard, "I used Coq in the past for this", and some systems have a high cost of failure.

"That said, space probes written in LISP and FORTH have been debugged off world." Better than proving them correct

[Someone left early for lunch and the speaker said "He's leaving in ANGER at what I said." That's... super incredibly rude wtf]

"Spec is always wrong! When the spec's right, it's still wrong!" Debugability > correctness by construction

Programming is mostly debugging, so we do exploratory programming. "In a dead coding language [sic], I'd have to debug independently." In common lisp, you can see the value of the variable, and see it graphically. "This was in 1980, you should be ashamed if your language doesn't"

Obligatory Smalltalk worship, it has a browser that lets you see everything available.

"Here's the glamorous toolkit, by Fink. It takes it to the limit, we should all take inspiration from it. They're rushing to the future and we should follow them."

#StrangeLoop

[I tried GToolkit last week and it was unbearably slow. Like lag for every keypress slow. I was running it on a 64gb i9 workstation]

"You should have programming at least as good as smalltalk and lisp?"

"'well duh what if I had a real fast compiler?' 💩"

[Missed a gag about how anybody who's skeptical is a scaredy-cat weenie]

"Now most programming is still in batch mode. We tried to get past that in 1960! Then C++ happened." Ken Thompson thought C++ was a garbage heap. Too much stuff. Javascript is going same direction

Go is a lot simpler and that's great, but it's still something you have to compile.

Rust is a good library language, but a HUGE missed opportunity in terms of interactive programming. "They just went to the punchcards again. IT'S TOO COMPLICATED!"

#strangeloop

"Zig is a lot better than rust, but why isn't it interactive programming?! It's going right back to the 1970's!"

REAL future is things like:

- Structure editor for racket
- Diagram viewer for racket
- Datarabbit for clojure, box-and-arrow PL on top of clojure

- Clerk, speaker's language. No separation between notebook and production code. You can write viewers for things to see the graphical output. It improves on emacs and clojure

- Hazel has "typed holes" which do interactive type inference and improves the user interface
- "Tylr, I can't show you in a good way without video" [your slides can't do video? WEAK]
- [missed this one]
- Enso
- Hest: computes with animation

"If you see something insane and looks like outsider art, it's probably got some good ideas."

Okay lunchtime.

#StrangeLoop

-----

"Formal Modeling and Analysis of Distributed Systems", @ankushpd, #StrangeLoop

(Disclosure, I got a private demo of the tech here two days ago.)

"Saying it loud and clear, formal methods is hard. This is how to make it easier."

Distributed systems is hard. You need to satisfy scalability, availability, correctness, etc, all in the face of nondeterminisim. Node failure, message ordering, timeouts, etc. Traditional testing approaches fail to provide a strong guarantee. So it's not rare to find prod bugs.

The guard rails are super important, but we do need something more, to prevent design-level bugs from reaching production. So, formal methods.

FM means a lot of things to people: pencil&paper, Coq, TLA+. Speaker definition: any process checking the system against a formal spec.

Core idea is "thinking abstractly, formally, above coding." Leslie Lamport says coding:programming :: typing:writing. Writing is more than typing, it's also pacing, characterization, structure, etc. Same with design. What are the distsym components, how do they interact?

FM, then, is abstracting away the unwanted details. You can also do this with a whiteboard, but with FM you can also create a formal specification you can take to customers. Even better, it can be an executable specification you can test.

#StrangeLoop

So why don't we already do this? Some challenges.

- Steep learning curve: not programmer-friendly
- Developers need to reason about system in the same manner they think about it. FM uses single giant state machines, programmers use communicating state machines

- Scalability, doesn't scale to large real-world systems
- Gap between specification and implementation.

Ankush's solution: P, a PL + framework where programs are communicating state machines. Has a scalable backend checker. Compiles to actual programs, too

#StrangeLoop

P comes from the Windows USB 3.0 driver. Used P to find hundreds of bugs, went on to use in Azure and Office. Compiled code had zero bugs after a year of heavy use

[Skeptical. Evidence: I use Windows]

[Someone started yelling at me on twitter, got distracted]

[Okay, trying to get back in sync. Give me a second.]

P programs have three parts: a system, a specification, and a test harness. Programs are a collection of event types and machines. Machines are a set of states, with entry and exit effects on each state transition.

[Sorry, still rattled by the DM. Person called me a plague spreader because I have a deviated septum. I'm not following perfectly, but I saw the demo a couple of days ago and I'm running off my memories of that.]

When the state machine receives messages, it can do different operations based on the states, and then transition to different states. This encodes your program logic. It can send messages to other machines, triggering their own transitions.

The "spec" is a runtime monitor that watches the global messages being passed around and "taps into" them to compute its own properties. IE you can assert that if you see a "success" message, you didn't get any inconsistent prior messages. Spec monitors can have their own state.

P is now available on github, so you can try it yourself. Programmers using P find it a great thinking tool. It forces them to think about everything thoroughly, which itself catches bugs.

[The test component is a fault injector].

Model checking is basically search problem: you inject messages interleaved with system states. It's not as comprehensive as formal method techniques, but it catches the same class of problems that we normally need FM for, so it's a major usability win. #Strangeloop

In industry and academia, we always think about FM as being the proof, but the process is just as important. The act of thinking things through is enough to find major bugs.

Link for p: github.com/p-org/P

Q: Is there a way to generate documentation from the P model?
A: Hope is that the P model itself can be the documentation

Q: Are there annotations that connect low-level code to high level models?
A: Possibly but exceptionally hard

Q: Why is the spec also a state machine and not an invariant?
A: Programmers push back on mathematical invariants

Q: I thought specs were types?
A: Specs are a lot broader than that, specs as state machines map better to distsyms

Q: Are there any parts of P that can generate the final implementation?
A: You can generate C code from the P compiler. USB drivers did this, but for distributed systems it's hard to create efficient production code. But it does make code!

After an ice crem break with @bellmar, time for the FINAL TALK OF #STRANGELOOP. But first a big announcement: next year will be the FINAL StrangeLoop. Early bird registration starts tonight.

-----

"The Secure Software Supply Chain", @kelseyhightower, #StrangeLoop

"It's 2015, and I got a deadline. I see a USB key on the ground and wonder what's on it? So I plug it into my production computer. It has a folder called 'free software!' The world is so kind."

"This looks just like the code I needed. Has a readme and everything! And it compiles okay, so I push it to production. Cloud is secure by default right?"

"There's a saying: good developers copy. Great developers paste."

Problem, this has so many vulnerabilities!

Boss was mad, so the whole team got together and came up with a plan: pretend they were working on something SUPER critical and couldn't fix the vulnerabilities.

"Turned up my headphones to see if he'd go away"

[Started playing "Running for the Hills" by Kate Bush] #strangeloop

We're all doing this when we use software dependencies, like pull packages from github. The first part of the supply chain security is developers: what do we put in the code?

"We have to test the cake, not the eater of the cake. 'Hey I'm dead, you're cake was bad!'"

We can't get 100% security, but we should at least chip away. So try to reach all four levels of software security, as defined by SLSA.

Level 1: documentation of the build process. "Scripted build? I can do that. I know bash!"

#StrangeLoop

"I do a controversial thing: I vendor my dependencies. People think I'm a madman: can't I just pull them from the super-safe internet whenever I need!" But the internet goes down, you should be able to build regardless of the internet.

Another idea: "bill of materials"

This means knowing all the transitive dependencies. His go ping server has 9.

Now celebrate! Put a SLSA Level 1 badge on the repo! Yaaay!

[This is all livecoding btw. THAT'S JUST HOW KELSEY ROLLS]

Oh SLSA 1 also needs "provenance": showing the build system itself is secure. SLSA tooling generates metadata. Who built it, the SHA1 of the build, when the build started and ended. Golang actually puts a lot of the build info in the binary.

#StrangeLoop

Introducing "nocode": write nothing, deploy nowhere.

github.com/kelseyhightowe…

["Demoing" nocode]

Nocode has 53.7k starts. "I literally thought about quitting my job and raising an A-round". Somehow there are 4500 issues. And the community helps each other!

github.com/kelseyhightowe…

Back to SLSA, the higher levels are incredibly hard to do because you need tons of human verification. But also for some reason SLSA 3 includes "has version control". But also includes things like "strong actors" and "verified commits". You can sign commits with GPG!

You can sign commits by putting

[commit]
gpgsign = true

in your .gitconfig

#StrangeLoop

SLSA 3 also requires "thermally isolated" builds. IE can't build mutliple things in the same build environment, so that one can't somehow corrupt the environment in a way that messes with later builds

[Missed a bit because I got distracted with writing some notes for my StrangeLoop writeup]

Securing the supply chain is a continuous process. Things can be secure and then become insecure over time. Make some tools to make it more pleasant, like this:

[tool that while it scans vulnerabilities plays music, which I won't try to identify after the last kerfluffle]

I tried to find the repo he said he used but couldn't, even checked all his github repos. Sorry I failed you all as a conference livetweeter

Okay questions

Q: Can we automate recognizing levels?
A: No, we need to be mindful, and being manual at SLSA helps be mindful

Q: What about reproducible build tools like Nix?
A: When I worked with Python, the packaging situation was a nightmare. NixOS is a great general purpose solution. Problem is that NixOS gets HUGE. "This is a go app, why is there Python 3.9?!" Need to get better at removing crufts

Q: What about monorepos that create multiple artifacts?
A: REALLY hard.

Q: GDPR?
A: International politics is hard. The day is coming where the citizenship can determine if you can submit to a repo or not. Coming faster.

Q: Top three requests from language designers?
A: Not a competition, there shouldn't be language wars. Keep more metadata in the build. Make things amenable to tools. Work with the imperfect standards we have, don't make new standards we have to decide between

And that's it for #StrangeLoop. See you all next year! But not the year after 😭😭😭😭