Benchmarks don't answer the real question. Benedict Evans frames the core problem with evaluating new models like o1 Pro, Phi-4, and Midjourney 6.1: knowing a model scores higher tells you nothing about whether it can do something you actually need done. The useful distinction is between tasks where quality is a spectrum and tasks where the answer is simply right or wrong.
Generative AI has clear product-market fit in software development and marketing for a specific reason: errors are visible and fixable, and 'wrong' is often negotiable. Evans uses the analogy of 100 interns checking each other's work. But a second category of tasks exists where the output is binary, the user is not a domain expert, and verification requires repeating all the original work from scratch. For those tasks, current LLMs are not usable at all, not just imperfect.
The original piece walks through a concrete, real-world workflow Evans wants to automate, using it to stress-test where today's models actually break down. That worked example is where the argument gets specific and worth reading in full.
[READ ORIGINAL →]