donut.c update. I had a sort of dumb random epiphany this morning, which let me finally come up with an ascii donut renderer which uses no sines, cosines, sqrts, divides, or even multiply instructions. Just shifts and adds/subtracts, start to finish. No framebuffer memory either. I made a version of the donut based on @iquilezles's brilliant raymarching theory; the issue is the length operator - you need to compute sqrt(x^2 + y^2). I had some ideas, but GPT4 pointed out you can use CORDIC to rotate the x,y vector onto the x axis to get the length(!)

I reimplemented GPT-2 from scratch in C++ as an exercise to really understand the nuts and bolts of LLMs. GPT2-117M isn't a super great model, but it's extremely satisfying to get it to generate basically the same thing as other reference implementations. Earlier today it occurred to me that Flash Attention softmax aggregation can just be done left-to-right, so you only need one pass over the KV cache per query vector, without ever materializing attention weights. Neat trick!