Thread by @FriedChicken

4 years ago I started implementing an entire #linux #wayland display server in the browser because "wouldn't it be cool if ...", but I never really shared my experiences that eventually lead me to implement a #kubernetes powered cloud desktop computer. 🧵👇

I'll try to keep these posts chronologically but most comes from memory so I apologize in advance. ;)

It basically started with a discussion in #wayland on #irc where it was suggested that one should use (s)rtp for real time video stream. The browser lacking such things, only offers WebRTC so first thing was to check if that could be utilized.

Turns out it's nearly impossible to attach meta data to a video frame coming from webrtc, something you need in wayland protocol as you need to link a buffer with a commit request. (Ironically only MS err edge? browser supported this at the time)

So exit #WebRTC video, hello gstreamer. Gstreamer was suprisingly easy to implement as you can basically setup an entire pipeline using a string, input file and output file. WebRTC was still used for data transport using a datachannel.

(Note at this point, there was till zero #wayland code). Next step was to see if we can decode a video frame, and control the exact moment it is presented on screen. #html5 video offers something called MSE which allows you to feed it chunks of video.

Great, all we need to do is feed it single frame chunks and we should be set. Yeah, no. MSE doesn't tell you when it's showing the frame, it's also expects extra (useless overhed for us) mp4 container data and playback time. More complexity we don't want or need. Exit MSE.

Looks like we need to handle the video frame decoding ourself. After some trial and error journey into H264 decoders in ASM.js and WASM there was finally something that ticks all checkboxes. github.com/udevbe/tinyh264

Great. All functionallity was here. But will it be fast enough? Luckily the whole pipeline could do 1080p@30fps. Good enough for now. Time to port the the #wayland protocol to #javascript. 👀

If you want to support #wayland compositor in your #browser, you have to:
- make sure you can handle native file descriptors over network
- make sure you can deal with native wayland server protocol libraries (drm & shm implementations)... over network
- deal with slow clients
😩

But there's a solution for all these requirements:
- native file descriptors in the protocol are represented as URLs (strings)
- Native protocol libraries require a proxy compositor and a libwayland fork
- slow clients are handled by an async compositor implementation.
😮‍💨😌

After some tinkering a first version was working and was promptly featured on @phoronix
phoronix.com/scan.php?page=…
Great success! But the work was far from over. Things were still too slow, too complex not really usable. 😿

Turns out writing an async wayland compositor is really hard. Eventually webrtc was completely dropped and replaced by websockets, each applicaiton surface is now a webgl texture instead of it's own HTML canvas (HTML canvas doesn't do double buffering) and more.

Meanwhile another idea was brewing. Since we now have a compositor in the browser that talks near vanilla wayland. We don't *have* to run the apps remote. They can run directly in your browser as well using #webworkers

And so a poc was implemented and promptly featured again on @phoronix phoronix.com/scan.php?page=…

Building on this idea, there was one big blocker. There are literally no good widget libraries that allow you to render directly to a #html5 #SharedMemoryBuffer let alone to a #WebGL canvas. So to prove this idea was feasible, I experimented if I could get somethng going...

First there was the need for a good drawing library. Since there are none that fitted al the needs, I resorted to compiling #skia to #wasm. This was a challeng, not having any C++ experience but got something working eventually... and Google noticed! 😲 github.com/Zubnix/skia-wa…

Turns out they were wanting to do the same for their upcoming port of #Flutter to the browser (although they never mentioned that at the time but 1+1=...). So pretty cool, meant that I didn't have to put my spare time in it anymore and could eventually start with the next phase.

Write a custom #React renderer that outputs to an offscreen #webgl canvas, which talks #wayland protocol to a compositor in your #browser, while running in a #webworker.
No biggy.

Writing a custom #react renderer is hard. Not because it's hard (it's quite easy actually) but because there is basically no documentation about it. It doesn't help that it has 2 render modes and all documentation is about the the first one... I wanted the second one. 🥲

Eventually something was working and the basis for a #browser #widget #toolkit that can output to #offscreen (and onscreen) #webgl is there: github.com/udevbe/react-c…

Great. So now the pure browser wayland app use-case has been proven. But there is still other things lacking. Not all remote #Linux apps run on #wayland, in fact most still use #X11. So if we want to support all linux apps, we need to support #xwayland... in the browser. 😅

They way #xwayland works, is that the #X11 server presents itself as a #wayland client, but still requires the wayland compositor to act as an X11 window manager. In our case, the compositor is running in the browser. So that means the browser has to function as an x11 client.

Fun fact: there are no libraries that allow a browser/webpage to act as a x11 client. So... let's implement it ourself! Looking at #xcb and how xpyb works, xtsb (x typescript bindings) was finally born. github.com/udevbe/xtsb

Implementing the X window manager was quite a challenge but luckily #weston the #wayland reference compositor had lit what would otherwise be a dark path, and I could basically rewrite their C code to TypeScript and eventually got somethng working!

https://twitter.com/FriedChicken/status/1327016484279885831

Adding POC Xwayland support was nice (and is still not further implemented for now), but there was still another fundamental issue that was left unaddressed. The whole setup and way of running applications was extremely clumsy and messy... enter #kubernetes

btw up until now all development was done during my spare time while working full-time. I also had the privilege of welcoming 2 wonderful kids into the world. So if you tell me you don't have time for x or y, I just think you must have a very healthy sleep schedule. 😉

Back to #kubernetes. I noticed people had trouble setting everything up. So with a little help from a friend we started implementing a custom kubernetes operator that can manage browser compositor sessions and the applications they display.

A #wayland compositor proxy runs as a sidecar inside the pod that hosts the desktop application container. This means that you can distribute all your desktop apps over your entire #k8s cluster and finely tune the resources, filesystem and caps the app has access too.

Proof of #wayland on #k8s

That's it. That's the tweet. 🎤💧