Tim Zaman Profile picture
Mar 18 9 tweets 3 min read
Azure/Microsoft released pics of OpenAI's next-gen AI datacenter.

This supposedly powers [Chat]GPT* training (inference?). From the pics we can infer most components, power, network/ib topology, layouts, vendors and more.

A thread 🧵[1/7] Image
[2/7] The datacenter Microsoft/Azure pictures is in Quincy, WA, USA (47.23,-119.86). This is an old picture. Today, there's a datacenter in place of that parking lot.

An old parking lot as the origin of this modern AI revival. Nice: I hate parking lots. Image
[3/7] Why is the datacenter in Quincy, WA?

AI eats compute which is fed by power and cooling (+=power). This is a massive Opex to cloud providers.

But in Quincy you pay 3ct/kWh! (SF=30ct/kWH). This is bc they have hydro (green!) power.

I guess that makes ChatGPT "green"? Image
[4/7] They are using the Azure ND A100 v4.

Cost ~$150k/node.

8x A100 GPUs, 2x48 core CPU, 900GiB DDR4 ram.

8x IB HDR NICs provide a 200x8=1600Gb/s gpu-gpu fabric.
The data fabric seems separate, using custom Azure stuff. Data ingest is a wimpy 50Gb/s: for LLMs totally fine. Image
[5/7] Fabric! A thin blue cable is optical/AOC (expensive) and a thick black one is copper/DAC (cheap). DAC only works short range ~5m.

The data NICs flow into an underpopulated Arista 6070CX switch, and out-of-rack with 8x AOCs.

gpu-gpu is all AOC, maybe to a core switch. Image
[6/7] Their high-level layout is using cold aisle containment (this is the cold aisle), with amazing levels of standardization.

All nodes depicted here are just for data storage and simple cpu compute. No HPC or AI happening in this aisle. Image
[7/7] Closing thoughts - The AI tech tree is deep, and you could spend ur life in Pythonland.

AI is expensive, and if you care about performance, everything below you is relevant, down to the bare metal.

The future may be full of monolithic bare metal supercomputers.
*hot aisle containment (thx @rpoo)
Found a paper on that custom Microsoft 50GbE NIC with (lolz) usb-b: microsoft.com/en-us/research… Image

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Tim Zaman

Tim Zaman Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(