More good read/discussion on psychology of LLMs. I don't follow in full but imo it is barking up the right tree w.r.t. a framework for analysis. lesswrong.com/posts/D7PumeYT…
A pretrained LLM is not an AI but a simulator, described by a statistical physics based on internet webpages. The system evolves given any initial conditions (prompt). To gather logprob it internally maintains a probability distribution over what kind of document it is completing
In particular, "good, aligned, conversational AI" is just one of many possible different rollouts. Finetuning / alignment tries to "collapse" and control the entropy to that region of the simulator. Jailbreak prompts try to knock the state into other logprob ravines.
The difficulty of alignment is to a large extent the elimination of probability to role play a good AI turned evil, in spite of the vast quantities of related content we have collectively created. In this sense an unaligned AI would be a self-fullfilling prophecy.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
