Gary Marcus called it in April 2024 - LLMs are slowing down in returns, while their costs are increasing
And here’s the thing – we all know that GPT-3 was vastly better than GPT-2. And we all know that GPT-4 (released thirteen months ago) was vastly better than GPT-3. But what has happened since? I could be persuaded that on some measures there was a doubling of capabilities for some set of months in 2020-2023, but I don’t see that case at all for the last 13 months. Instead, I see numerous signs that we have reached a period of diminishing returns.