Education reform ideas, starting with least radical: 1. Outside USA, get rid of "early specialization" in high-school/uni and switch to US flexible, liberal-arts system 2. Outside UK, switch to UK-style short degrees (3 year BA, 1 year MA, 3 year PhD)
3. Expand coding, CS, AI, and data science through the whole education system. It’s the new “reading, writing, arithmetic." 4. Allow BA degrees by open examination (fee = wage for examiner to grade the papers). Allow PhD by open submission of thesis.
5. PhD not required to be academic (e.g. require 2-3 year masters instead as in old UK system)
(Getting more radical...) 6. Reduce age segregation in school and uni. Most important, normalize people starting uni (or uni-level colleges) aged 14-18.
7. Big expansion of hands-on building/creating things. Science labs, hackerspaces, textiles, art, video-editing, software (no-code + coding). Reduce health and safety strictures.
8. Big expansion of work experience programs, internships, apprenticeships. Normalize the idea that under 18s can do real work, and can learn as much from work as from school.
9. More open/flexible system of teaching in (high) schools. Allow professionals in other fields or retirees to teach part-time (without needing a degree in education etc). Let them teach their knowledge, rather than a rigid curriculum. If not teaching, let them act as mentors.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
1.Language models could become much better literary stylists soon. What does this mean for literature? A highly speculative thread.
2. Today models have limited access to sound pattern / rhythm but this doesn't seem hard to fix: change BPE, add phonetic annotations or multimodality (CLIP for sound), finetune with RL from human feedback. GPT-3 is a good stylist despite handicaps! gwern.net/GPT-3#rhyming
3. There are already large efforts to make long-form generation more truthful and coherent (WebGPT/LaMDA/RETRO) which should carry over to fiction. RL finetuning specifically for literature will help a lot (see openai.com/blog/summarizi…, HHH, InstructGPT)
What are some domains of knowledge where big language models will be impactful?
Maybe domains with vast, messy stores of content that few humans master. E.g. 1. All US laws+regulations 2. Biological details of every beetle (>1M species) 3. All code in 787 (14M lines)
4. Function of all genes in all genomes (20k in humans) 5. Obscure human languages (Akkadian) 6. For a big company, what's the standard operating procedure for every staff role.
Let’s say there’s N items of interconnected knowledge in a domain. Even if humans can understand any *one* item better than a GPT-3-like model, the model can provide value by understanding N>100,000 items modestly well.
1/n. Will there be any more profound, fundamental discoveries like Newtonian physics, Darwinism, Turing computation, QM, molecular genetics, deep learning?
Maybe -- and here's some wild guesses about what they'll be...
2/n.
Guess (1):New crypto-economic foundations of society. We might move to a society based on precise computational mechanisms:
a) smart contracts with ML oracles
b) ML algorithms that learn + aggregate our preferences/beliefs make societal decisions/allocations based on them
3/n. We see small specialized instances today (crypto/DeFi, AI-enabled ad auctions, prediction markets, recommender systems) but the space of possibilities is large and today's Bitcoin may not be very representative.
Baseline models (GPT-3, GPT-J, UnifiedQA/T5) give true answers only 20-58% of the time (vs 94% for human) in zero-shot setting.
Large models do worse — partly from being better at learning human falsehoods from training. GPT-J with 6B params is 17% worse than with 125M param.
Why do large models do worse? In the image, small sizes of GPT3 give true but less informative answers. Larger sizes know enough to mimic human superstitions and conspiracy theories.
2/ Gerry Sussman. Hofstadter said Gödel invented LISP in proving the incompleteness theorem. Sussman shows the amazing breadth and elegance of LISP ideas. SICP, SICM, How to build robust systems. google.com/url?sa=t&rct=j…
3/ Eric Drexler. Engineering is neglected by philosophy departments (see Sussman also). Engines, Nanosystems, how engineering differs from science, CAIS. Disclosure: he's currently at the FHI (which is part of a phil dept). overcomingbias.com/2013/06/drexle… lesswrong.com/posts/x3fNwSe5…