Bryan Cantrill:
The problem is that LLMs inherently lack the virtue of laziness. Work costs nothing to an LLM. LLMs do not feel a need to optimize for their own (or anyone's) future time, and will happily dump more and more onto a layercake of garbage. Left unchecked, LLMs will make systems larger, not better—appealing to perverse vanity metrics, perhaps, but at the cost of everything that matters. As such, LLMs highlight how essential our human laziness is: our finite time forces us to develop crisp abstractions in part because we don't want to waste our (human!) time on the consequences of clunky ones. The best engineering is always borne of constraints, and the constraint of our time places limits on the cognitive load of the system that we're willing to accept. This is what drives us to make the system simpler, despite its essential complexity. As I expanded on in my talk The Complexity of Simplicity, this is a significant undertaking—and we cannot expect LLMs that do not operate under constraints of time or load to undertake it of their own volition.
So well put. Recommended reading.