Thread by @tszzl

i increasingly think the shoggoth is an inaccurate & unpleasant metaphor. the base language model is more similar to our own visual field brain regions that are optimized in a pure predictive loop to minimize surprise than some alien god with hidden intentions

im absolutely sure that our visual field processors have several hidden suboptimizers that help it self improve over time but it seems strange to worry about my ocular cortex developing agency & stealing resources from the rest of my brain (even in a grander evolutionary sense)

you could argue that text is different, it encodes the causal structure of the world, which leads to instrumental convergence etc, but of course the dorsal stream, actively predicting motion requires quite a strong understanding of causality!

if someone told me they were taking a pill to 100x their dorsal stream neurons i think I’d be more amused than concerned. they’d probably be really good in a fistfight but then I’d just avoid the fistfight

so then the “smiley face” part of the metaphor, the RLHF policy, is the only agent of any concern. so avoid a fistfight! don’t anger Sidney Bing or even better never instantiate a dark mode Sidney Bing. but it points to the idea that RLHF is more than just “stopgap alignment”

these are weakly held opinions & I’m pretty sure a good rebuttal exists . Consider this thread a search query for the right lesswrong link

“reason is the slave of the passions”. The gargantuan brain of the language model auto prediction loop serves as a worker process for Sidney’s personality and more generally as an exocortex for the human mind using it

I’ve been wondering for a while how to find the right metaphor for a hyper intelligent pack mule but it’s sitting right there: reason / the neocortex is a slave to ancient mammalian impulses that are refracted and take new forms

i know this is pop neuropsych but it’ll do for now