Tom Goldstein Profile picture
Mar 13, 2023 13 tweets 4 min read Read on X
Here's the real story of #SiliconValleyBank, as told the boring way through tedious analysis of balance sheets and SEC filings 🧵
Throughout 2021 startups were raising money from VCs and stashing it in SVB. Deposits increased from $102B to $189B. That's an 85% change in one year. Wow! Image
Most news sources claim that SVB stashed this money in relatively safe treasury securities. This is something important that most media sources got wrong.
forbes.com/sites/billcone… Image
The balance sheet reveals that all of that money, over $80B, was invested in mortgage backed securities (MBS). MBS are far more susceptible to interest rate risk than treasury bonds, especially when interests rates get high. Image
Rather than increase liquid assets in proportion to their deposits, all this money (and more) got invested in MBS. By the end of 2021, SVB had *decreased* liquid assets (cash and short-term securities) by $7B, setting them up for a liquidity crisis if the tech economy reversed. Image
Then came 2022. The tech sector cooled off. Startups stopped raising money and began spending down their balances. Net withdrawals in 2022 were $16B - that’s more than SVB’s cash reserves, forcing them to start selling their bonds. Image
As inflation loomed, the Fed pumped up interest rates…a lot. SVB had purchased low-interest bonds and now had to sell before their maturity date. But nobody wants low interest bonds when the Fed is selling high interest bonds. SVB had to sell below face value, taking a loss. Image
By 2023, SVB was solvent...technically. They had $211B in assets (largely MBS), which was more than their $195B in liabilities (mostly deposits). $16B more, to be precise. But here’s the rub: Their balance sheet uses the face value of bonds, not the market value. Image
The market value of their bonds had decreased by nearly $16B because of interest rate hikes, making their fair market assets almost equal to their liabilities. If interest rates ticked up just a *teensy* bit more, the market value would be less than they owed depositors. Image
Interest rates didn't just tick up a teensy bit in 2023 - they went up a half point. We don’t have a Q1 report, but SVB’s fair market value almost certainly fell too low to cover its liabilities. Meanwhile, withdrawals almost certainly accelerated.
SVB's problem seems to have gone unnoticed until they announced a stock sale to raise money to close this gap in their finances. NBD, right? Unfortunately, a stock sale is a sure-fire way to get people to scrutinize your balance sheet. And the rest is history.
Some say SVB was just another victim of high interest rates. This overlooks some of the strange decisions they made. Bank of America holds 12% of its deposits in cash. Wells Fargo has 17%. SVB held only 8% after the 2021 rally, despite poor diversification and higher risk.
You’d think a bank that saw an 85% fluctuation in deposits in one year would have a plan to withstand volatility. But the storm arrived and SVB didn’t own a raincoat.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Tom Goldstein

Tom Goldstein Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @tomgoldsteincs

Jun 20
LLMs have low randomness: if you ask the same thing twice you get similar responses. Generator prompts are a way to boost the randomness of LLMs.

Using a few generator prompts, I had Gemini write an entire instruction tuning dataset from scratch. It outperform popular datasets. Image
Let’s start with a toy example of why we need generator prompts. Suppose I want a list of different colors. So I feed this prompt to Gemini 1000 times. This does poorly - I only get 33 unique outputs from 1000 runs. I need more randomness. Image
A generator prompt asks the model to enumerate a long list of execution paths, and then randomizes which paths get chosen.

Here's an example. The numbers 23 and 76 are randomized each time the prompt is called.

This prompt gives me 782 unique outputs from 1000 runs. Image
Read 9 tweets
Oct 12, 2023
🚨 This one simple trick will level up your LLM🚀🚀

Wait...don't go. This isn't a blue check grifter tweet!

Instruction tuning with this easy trick will *actually* boost AlpacaEval scores, even for large (70B) and llama2-chat base models…by a lot 🧵 Image
Ok, here's the trick: during instruction finetuning, we add uniform random noise to the word embeddings.

That's it. Nothing else.

We tried this on a bunch of base models and finetuning datasets. They all showed big gains. Image
Even when the base model is already highly refined (e.g. llama2-chat) or very large (llama2-70B) the benefits of NEFTune are still quite strong. Image
Read 8 tweets
Jul 19, 2023
The Llama2 model is pretty impressive. Human evaluators rank it slightly *better* than ChatGPT on a range of things (excluding code and reasoning).

Here's a short TL;DR on what Meta did to improve the state of the art 🧵 Image
Llama1: Small models (7B & 13B) were trained on 1 trillion tokens. Large models saw 1.4T tokens.

Llama2: All models trained on 2T tokens. This means the small models are "over trained" beyond what the scaling laws recommend, resulting in great performance for small models! Image
As a result of the long training runs, Llama2 beats other major open-source models at most academic benchmarks. Their 7B model is WAY better than other 7B options on all tasks except code. Image
Read 11 tweets
Jul 5, 2023
Nvidia’s AI products follow a weird reverse Moore’s law: every two years, you get half as many FLOPS for your money. This is the opposite of the rest of the chip market 📈

With the H100 release, Nvidia had to reverse course.

A 🧵 on Nvidia losing its grip on the GPU market.
Let’s focus in on the machine learning GPUs. You can see the value drop over time, until the H100 created an uptick. Note: I’m using today’s price for each card, but a similar downward trend also holds for the release prices.
The drop is because of monopoly power and clever market segmentation.
Example: The “server-grade” V100 is a minor variant of the 2080ti gaming card. Nvidia sells it to institutions instead of gamers, charging 5X more for the V100. This means huge profits.
lambdalabs.com/blog/best-gpu-…
Read 11 tweets
Jun 19, 2023
Training an LLM takes about 1 trillion words. That’s about 30,000 years of typing.
But where does this data come from?
And what does this have to do with the Reddit protests?
Here’s how OpenAI trains models on “the entire internet.” 🧵📜
Much of what we know about OpenAI is from urban legends. But the GPT3 paper does have a table showing their data sources. The cliché that LLMs are trained on “the whole internet” comes from the use of CommonCrawl. Image
CommonCrawl (CC) is a non-profit that scrapes the internet with bots and tries to record everything since 2008. 90% of CC is HTML, CSS, and scripts. The usable 10% contains junk that needs to be tossed out to clean the dataset.
Read 12 tweets
Jun 13, 2023
A common criticism of LLM watermarks is they can be removed by AI paraphrasing or human editing. Let's put this theory to the test! Can a watermark be automatically removed by GPT? Can a grad student do any better? The results surprised me 🧵
arxiv.org/pdf/2306.04634… Image
First, if you don’t remember how watermarks work, you might revisit my original post on this issue.
TL;DR The watermark is a subtle pattern embedded in LLM outputs that labels it as machine generated. High accuracy detection usually requires 50-ish words.
The experiment: We generated watermarked text using the Llama model, then asked a non-watermarked LLM (GPT-3.5) to re-write it. We did lots of prompt engineering to try to get rid of the watermark. Finally, we checked whether we could detect the watermark in the rewritten text.
Read 11 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(