“easy for humans, hard for ai” is not a solid design principle for evals imo
it leads you towards “judging a fish by how far it can climb a tree” absurdities
but maybe it’s one orthogonal eval style among many equally important ones
it leads you towards “judging a fish by how far it can climb a tree” absurdities
but maybe it’s one orthogonal eval style among many equally important ones
specifically arc agi visual tasks look like nonsense in JSON format and multi modality isn’t great
and the character manipulation tasks don’t work for the same reason models mess up the how many “r”s in strawberry problem (tokenization/BPE)
and the character manipulation tasks don’t work for the same reason models mess up the how many “r”s in strawberry problem (tokenization/BPE)
it’s almost adversarially constructed wrt input modalities. for a model to solve these requires a far higher level of intelligence than equivalent human score
• • •
Missing some Tweet in this thread? You can try to
force a refresh
