lead them to paradise |
intelligence is inherently about scaling |
be kind to us AGI
Dec 16, 2024 • 5 tweets • 3 min read
Let's review OpenAI's 12 days of shipmas so far:
Day 1 - o1 and ChatGPT Pro:
- delivered a product they promised us months ago
- the launch was horrendous because of bad, missing and out-of-date benchmarks
- despite the failed launch, still no new benchmarks for o1 models
( - anouncing o1 pro and ChatGPT Pro on this day was stupid imo, Pro Tier only makes sense when you know of Sora
- i love o1 but they should have done the price cuts on that day instead of o1 pro )
Overall Rating: horrendous presentation of good products and basically no surprise factor - 3.5/10
Day 2 - Reinforcement Fine-Tuning ALPHA:
- nice idea, could be very useful for businesses
- showed some cool applications
- just an alpha and completely useless for 95% of their users
- a surprise
Overall Rating: good presentation of good products but again just an alpha preview and limited applications - 6.5/10
Day 3 - Sora:
- very cool feature
- no surprise factor
- server issues
- extremely limited usage
- poor implementation, like no image preview (just read my post why I think that, it could be so much better)
- europoors are cooked
- competitors offer the same
Overall Rating: very similar to o1 launch cool product but very bad implementation and presentation - 3/10
Day 4 - Canvas:
- decent feature
- absolutely no wow or surprise factor
- I guess it can be used in CustomGPTs which is nice to have
- competitors offer the same
Overall Rating: honestly no remarks, very neutral - 5/10
Day 5 - ChatGPT Integration with Apple Intelligence:
- siri using ChatGPT to generate responses
- document analysis on MacOs
- vision features for iPhone 16
- could have literally been a sidenote in the changelog
- RIP to all android poors
Overall Rating: at least some features but jesus ... - 2/10
Day 6 - Advanced Voice with Video:
- video understand is a useful feature, no doubt about that but the examples were so USELESS
- HOHOHO cringe santa
- but again no wow or surprise factor
- europoors are cooked once more
- competitors offer the same
Day 7 - Projects:
- organizing information is always good
- no wow or surprise factor
- competitors offer the same
Overall Rating: could've been a post on X - 4.5/10
Day 8 - Search:
- free users gain access to search - good for the poors
- search with AVM is nice
- in app maps
- search already existed before, so zero wow factor
- competitors offer the same
Overall Rating: 5.5/10
So far ~4.4/10 - Shipmas has been slightly underwhelming, unsurprising and unfortunately overshadowed by the botched launches of o1 and Sora
o1, o1-pro and Sora could've been solid 7s or 8s
AVM is a very cool and useful feature, but the presentation just lacked substanced and good examples. I know you think it's all fun with the christmas theme, but please just show me that this product is useful for my daily life!
Like help with homework, show how to jumpstart a car, hell even get the blind guy again... Just show anything more useful
Canvas, Search and Projects should've been in one presentation - alone they are not impressive but as a whole they are a solid Quality of Life improvement in
RFT was so far the best and most surprising launch, but again no higher grade because it's just a preview alpha
Lastly, please don't ever make an Apple "launch" again
My Sora critique:
A few hours ago they released "Byte Latent Transformer". A tokenizer free architecture that dynamically encodes Bytes into Patches and achieves better inference efficiency and robustness!
(I was just talking about how we need dynamic tokenization that is learned during training 🥲
It's like fucking christmas!)
I don't want to talk too much about the architecture.
But here's a nice visualization from their paper.
Let's look at benchmarks instead :)
"BLT models can match the performance of tokenization-based models like Llama 3 at scales up to 8B and 4T bytes, and can trade minor losses in evaluation metrics for up to 50% reductions in inference flops!"
This is basically a perplexity vs training flops chart - scaling laws with compute. BPB is a tokenizer independent version of perplexity.
BLT is on par or better than LLama 3 BPE!
Most importantly they scale this approach to train Llama-3 8B model on 1T tokens which beats the standard Llama-3 architecture with BPE tokenizer!
Paper Link: ai.meta.com/research/publi…
Paper Link: ai.meta.com/research/publi…