Maybe Intelligence Ain't All That

> Coding, math and most administrative work are similarly benign. What makes them easy is that they have relatively smooth solution spaces and are tractably verifiable.

Code most definitely does NOT have an easily verifiable search space. P!=NP? Halting problem?

It's easy to brush this off and think those are obscure computer science trivia, but these things come up all the time. If we could prove software correct, we wouldn't need to test it.

> Most of what matters doesn't behave like that. The universe is mostly the emergent behavior of complex systems..... In systems like these, no amount of reasoning delivers the answer, because there's no shortcut hiding in the gaps. You have to run the thing

Software is clearly a complex system, with plentiful examples of emergent behavior. To understand the behavior, you HAVE to run the thing, in prod, on the open network, with real users, under real conditions, with real data. You can simulate these in testing but you can never fully reason about them beforehand - they are computationally irreducible.

"That, I suspect, is why the explosion never came. The takeoff story assumed the limit was thinking"

Fable 5 has been available to the public for approximately five minutes now; how could we possibly know already what the consequences will be?