Evaluating LLM output is hard. It's the bottleneck to scaling AI products for many teams

A key mistake is defining eval criteria w/o actually LOOKING AT THE DATA. This leads to irrelevant / unrealistic criteria + wasted effort

Thus, I built AlignEval AlignEval.com The key insight: In addition to aligning AI to human context (prompting, rag), we also need to calibrate human criteria to actual AI output.

By working backwards from the data, AlignEval helps you build better evals.

Screenshots & how it was built here: eugeneyan.com/writing/aligne…

Our paper club recently revisited some of the earlier language modeling papers. Here's a one-liner for each.

---

Attention: Query, Key, and Value are all you need*

*Also position embeddings, multiple heads, feed-forward layers, skip-connections, etc

arxiv.org/abs/1706.03762 GPT: Decoder is all you need.

(Also, pre-training + finetuning 💪)

cdn.openai.com/research-cover…

Ran a simple benchmark (Mandelbrot sets) between Mojo & Python. The speedup is impressive, and it benefits from Python's libraries.

• Python: 1,184ms
• Mojo: 27ms 🤯
• Python (vectorized): 240ms
• Mojo (vectorized): 2ms Here's a GitHub gist if you want to try it yourself: gist.github.com/eugeneyan/1d2e…

(Couldn't download the notebook for some reason)

An insider's view on China's scale and tech, the 996 work ethic, and Alibaba's acquisition of Lazada. corecursive.com/software-world…

Years later, I'm still boggled by the scale and how we had to use a completely different tech stack (spoiler alert: it's mostly Ali Java). Yea, there were one-click deploys, rollbacks, A/B tests—you name it.

Also, there were SQL commands that were both powerful and scary (and borderline questionable 🙈). Any data analyst on the street became a median data scientist.

Over the past few weekends, I've experimented with using LLMs to build a simple assistant.

Here's a write-up of what I built, how I built them, and their potential. Also, some shortcomings with embedding retrieval, with solutions from search & recsys.

eugeneyan.com/writing/llm-ex… Here's my first project dabbling with LLMs for the humble `summarize`

https://twitter.com/eugeneyan/status/1637562031233708032

This weekend, I had a blast building a personal board of advisors using BeautifulSoup, @LangChainAI , and @pinecone.

`/board` provides advice to questions on tech, leadership, and life. It also provides links to sources for further reading! `/ask-ey` does something similar for my own site, eugeneyan.com. And because I'm more familiar with my own writing, I can better spot shortfalls such as not answering based on a source when expected, or when a source is irrelevant.

This is why CS fundamentals continue to be crucial: LLaMA 30B only needs 4gb of memory if we use mmap().

Not sure why this works but one reason could be that 30B weights are sparse. Thus, lazy loading the fraction of needed weights reduces memory usage.

github.com/ggerganov/llam… Correction: I guess all we can say is that it now (only) uses the actual memory required instead of 2x memory required.

Still an awesome improvement nonetheless!

news.ycombinator.com/item?id=354000…

I'm trying to get up to speed on the subject of text2image and diffusion models. What are some good resources I should consult?

Here's what I've found helpful so far. The 2015 paper by Solh-Dickstein that demonstrates how to systematically add noise to data (e.g., images) via forward diffusion and then learn a reverse diffusion process that returns structured data from noise.

arxiv.org/abs/1503.03585

Unfortunate, but true.

“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” — Edsger Dijkstra

https://twitter.com/_brohrer_/status/1553811938886520842

Adding tweets on this theme, starting with this.

https://twitter.com/rharang/status/1554168497298903040

I'm learning how to write better Python libraries. What packages do you recommend I emulate?

sklearn, pytorch, fastai come to mind—anything else? It may be helpful to read the principles behind each implementation. Here's a paper on PyTorch's design principles

arxiv.org/abs/1912.01703

What are some design patterns that are common in data and machine learning code? Here are a couple I've identified in my work.

Factory: Interface for creating objects
• sklearn.datasets, torchvision.datasets
• torch.utils.Data.DataSet, torch.utils.data.DataLoader

What are some good papers/tech blogs sharing how bandits are used in recommendation systems? Netflix shared how they use bandits to select the best artwork for each media

netflixtechblog.com/artwork-person…

I'm excited to share something I've been working on for a while—ApplyingML.com!

It collects the tacit, tribal, ghost knowledge on how to apply machine learning from papers, guides, and interviews with ML practitioners. I was inspired by the unexpected popularity of the applied-ml repo (github.com/eugeneyan/appl…) and @vboykis's sharing of her ghost knowledge (veekaybee.github.io/2021/03/26/dat…).

While starting AWS, before writing any code, engineers spent 18 months writing documents on how best to serve customers.

Similarly, before I build anything, I write docs. Here, I'll share three docs I write and reveal the framework that structures them

eugeneyan.com/writing/writin… In addition, I'm writing about design docs for data science / machine learning next.

Have you come across relevant material? Comment here or DM me please. Thank you!

Early this year, I compiled a curriculum of the best books, essays, and videos on writing non-fiction.

I was surprised by how much I didn't learn about writing in school.

Here are some of the uncommon and best advice I've come across (a thread👇).

eugeneyan.com/writing/what-i… Writing is 80% preparation, 20% writing

“I consume information, think, and write. I would say I spend about 80% of my time consuming and thinking, and only 20% of my time writing.” – @dollarsanddata

Regular note-taking didn't work for me.

Notes stay separate—connections are not made.

I fixed this by building a Zettelkasten using @RoamResearch.

(Zettelkasten originates from highly prolific sociologist Niklas Luhmann. He wrote 70 books & 500 scholarly articles.)

Thread 👇 (1/4) Here's how a Zettelkasten works:

• Write each idea you come across on a card
• Link idea cards to other relevant idea cards (idea -> idea link)
• Sort cards into broader topic boxes (idea -> topic link)

Here's how you can build a digital one using Roam.

14 Ideas on How to Grow Your Business by Writing (THREAD)

Ideas are all from @shl and @david_perell

Notes below 👇

(1/14) Everyone is a writer

• Engineers, designers, managers, etc.
• You can't avoid writing, might as well do it well
• Gumroad relies a lot on writing, so does Amazon