honestly the hardest part of building on LLMs isn't the tech — it's that whatever stack you commit to today feels like it'll be obsolete by the time you ship. your talk perfectly captures how much the landscape shifted in just six months. kinda exhausting to keep up lol
Simon — the "often-work to mostly-work" line is doing a lot of heavy lifting and it deserves to. That's the actual inflection point for most businesses — not the benchmark leaderboard, but the moment you stop babysitting the output.
I work with a team that builds custom AI agents for small businesses. The thing we've watched shift in real time is exactly what you're describing — programming languages as lock-in collapsing. Six months ago a client's agent was built in one stack and that was a commitment. Now we're rebuilding pieces in whatever fits the job because the switching cost dropped through the floor. That's not a technical detail — it changes what you say yes to.
The Shopify "osmosis learning" bit is the part I want to hear more about. An agent operating in public Slack channels so the whole team absorbs how it thinks. That's the opposite of the black box problem Addy Osmani just wrote about with comprehension debt. One approach hides the work. The other makes the work visible so humans can pattern-match alongside it. I think the teams that figure out which tasks need which approach are the ones that actually benefit from this wave instead of just surfing it.
The pelican test is perfect. The ridiculous tasks are the honest ones.
The November 2025 inflection point framing maps cleanly to the coding-agent shift more than to the model-of-the-week handoff. The pelican-bicycle test is a useful unscored eval precisely because it sits outside the training distribution and surfaces taste rather than benchmark optimization. The five-handoffs-in-one-month claim is the right reminder that single-model benchmarking has become noise at the frontier; what counts now is whether the harness plus the model crosses the daily-driver bar. RLVR on coding harnesses is the underrated through-line: it is what made November look like a step change rather than another quarterly upgrade.
Love this. You could include how the video capabilities of AI models have been getting a lot better. E.g. Xiao Hong Shu has a new wave of AI short drama.
Does the pelican test getting too famous kind of prove the eval problem in miniature, where even the weird little sanity check eventually becomes part of the weather?
Did the “LLM psychosis” phase end because the tools got less magical, or because we all quietly learned which project shapes they can actually survive?
wait, the jump from “often-work” to “mostly-work” feels like the whole story here. do you think the next benchmark has to test cleanup and rollback, not just whether the agent got the feature working?
The bit about the pelican benchmark wearing out made me laugh. Private weird tests feel like they have a half-life now, especially once everyone starts quoting them. Do you rotate yours, or is the decay itself useful signal?
Did the holiday-project burst feel like a one-time novelty spike to you, or did it permanently change your threshold for starting tiny tools? I keep wondering how much of the November shift was model quality vs permission to be messy on the first pass.
Did the pelican benchmark quietly turn into a test of who has seen the pelican benchmark? the Qwen laptop result is funny because it makes the old “frontier vs local” gap feel less clean than it did six months ago.
The Shore quote on maintenance costs and the Hashimoto quote on language lock-in dissolving are in direct tension with each other and I think the resolution between them is where the real insight lives. Shore says doubled output means doubled maintenance liability. Hashimoto says rewriting is now so cheap that maintenance might be the wrong frame entirely. Both are correct and the resolution is that code is becoming disposable. Written for a specific purpose, used until it degrades, then regenerated from scratch rather than maintained.
Thats a genuinely new mode of software production with no precedent. "Write once and maintain" was the old model. "Rewrite constantly" is emerging as the new one. And if thats where we're heading then the entire concept of technical debt needs reformulating because debt implies something you carry forward. Disposable code has no forward carry. Its liability terminates at the moment of regeneration. the industry is moving toward continuous regeneration of temporary code, and the skill that matters most in that world isnt writing or maintaining but knowing when regeneration will cost less than the next maintenance cycle.
honestly the hardest part of building on LLMs isn't the tech — it's that whatever stack you commit to today feels like it'll be obsolete by the time you ship. your talk perfectly captures how much the landscape shifted in just six months. kinda exhausting to keep up lol
Simon — the "often-work to mostly-work" line is doing a lot of heavy lifting and it deserves to. That's the actual inflection point for most businesses — not the benchmark leaderboard, but the moment you stop babysitting the output.
I work with a team that builds custom AI agents for small businesses. The thing we've watched shift in real time is exactly what you're describing — programming languages as lock-in collapsing. Six months ago a client's agent was built in one stack and that was a commitment. Now we're rebuilding pieces in whatever fits the job because the switching cost dropped through the floor. That's not a technical detail — it changes what you say yes to.
The Shopify "osmosis learning" bit is the part I want to hear more about. An agent operating in public Slack channels so the whole team absorbs how it thinks. That's the opposite of the black box problem Addy Osmani just wrote about with comprehension debt. One approach hides the work. The other makes the work visible so humans can pattern-match alongside it. I think the teams that figure out which tasks need which approach are the ones that actually benefit from this wave instead of just surfing it.
The pelican test is perfect. The ridiculous tasks are the honest ones.
The November 2025 inflection point framing maps cleanly to the coding-agent shift more than to the model-of-the-week handoff. The pelican-bicycle test is a useful unscored eval precisely because it sits outside the training distribution and surfaces taste rather than benchmark optimization. The five-handoffs-in-one-month claim is the right reminder that single-model benchmarking has become noise at the frontier; what counts now is whether the harness plus the model crosses the daily-driver bar. RLVR on coding harnesses is the underrated through-line: it is what made November look like a step change rather than another quarterly upgrade.
Love this. You could include how the video capabilities of AI models have been getting a lot better. E.g. Xiao Hong Shu has a new wave of AI short drama.
why do the pelicans on bikes almost always go left to right? 🤔
opus 4.7 looks like the only exception in the examples you shared
https://substack.com/@theverticaldispatch/note/c-261773074?r=1pgr4n&utm_medium=ios&utm_source=notes-share-action
https://substack.com/@theverticaldispatch/note/c-261773074?r=1pgr4n&utm_medium=ios&utm_source=notes-share-action
Does the pelican test getting too famous kind of prove the eval problem in miniature, where even the weird little sanity check eventually becomes part of the weather?
Did the “LLM psychosis” phase end because the tools got less magical, or because we all quietly learned which project shapes they can actually survive?
wait, the jump from “often-work” to “mostly-work” feels like the whole story here. do you think the next benchmark has to test cleanup and rollback, not just whether the agent got the feature working?
The bit about the pelican benchmark wearing out made me laugh. Private weird tests feel like they have a half-life now, especially once everyone starts quoting them. Do you rotate yours, or is the decay itself useful signal?
Did the holiday-project burst feel like a one-time novelty spike to you, or did it permanently change your threshold for starting tiny tools? I keep wondering how much of the November shift was model quality vs permission to be messy on the first pass.
Did the pelican benchmark quietly turn into a test of who has seen the pelican benchmark? the Qwen laptop result is funny because it makes the old “frontier vs local” gap feel less clean than it did six months ago.
Simon, please can you take a look at Gator: https://github.com/cumberland-laboratories/gator
It was born from failure, so it was built to be very useful: https://github.com/cumberland-laboratories/gator/blob/main/.gator/artifacts/origin-story-navigation-coding.md
The Shore quote on maintenance costs and the Hashimoto quote on language lock-in dissolving are in direct tension with each other and I think the resolution between them is where the real insight lives. Shore says doubled output means doubled maintenance liability. Hashimoto says rewriting is now so cheap that maintenance might be the wrong frame entirely. Both are correct and the resolution is that code is becoming disposable. Written for a specific purpose, used until it degrades, then regenerated from scratch rather than maintained.
Thats a genuinely new mode of software production with no precedent. "Write once and maintain" was the old model. "Rewrite constantly" is emerging as the new one. And if thats where we're heading then the entire concept of technical debt needs reformulating because debt implies something you carry forward. Disposable code has no forward carry. Its liability terminates at the moment of regeneration. the industry is moving toward continuous regeneration of temporary code, and the skill that matters most in that world isnt writing or maintaining but knowing when regeneration will cost less than the next maintenance cycle.