I still haven't heard a good answer to this question, on or off the podcast.
AI researchers often tell me, "Don't worry bout it, scale solves this."
But what is the rebuttal to someone who argues that this indicates a fundamental limitation?
- I still haven't heard a good answer to this question, on or off the podcast. AI researchers often tell me, "Don't worry bout it, scale solves this." But what is the rebuttal to someone who argues that this indicates a fundamental limitation?
Full episode, for those interested:Here is my conversation with Dario Amodei, CEO of Anthropic. We discuss: - why human level AI is 2-3 years away - race dynamics with OpenAI & China - $10 billion training runs, bioterrorism, alignment, cyberattacks, scaling...Best answer so far:Replying to @dwarkesh_sptl;dr: Maybe learning simple things (basic knowledge, heuristics, etc) actually lowers the loss more than learning sophisticated things (algorithms associated with higher cognition that we really care about), and the sophisticated things will eventually be learned as scaling
tl;dr: Maybe learning simple things (basic knowledge, heuristics, etc) actually lowers the loss more than learning sophisticated things (algorithms associated with higher cognition that we really care about), and the sophisticated things will eventually be learned as scalingI respect Jacob a lot but I find it really difficult to engage with predictions of LLM capabilities that presume some version of the scaling hypothesis will continue to hold - it just seems highly implausible given everything we already know about the limits of transformers!
