MC.85: PPC stands for Pay Per Crawl

From Free Web to Paid Access

Tim Berners-Lee's original vision for the World Wide Web was radical: a decentralized network where information flowed freely, accessible to anyone with a connection. Cloudflare's new "pay per crawl" system fundamentally challenges this principle by turning webpages into potential tollbooths.

We've moved from "information wants to be free" to "information wants to be compensated."

Rethinking Web Neutrality

Traditional web neutrality meant treating all internet traffic equally. But Cloudflare's system introduces content neutrality: should human browsing cost the same as machine crawling?

Human browsing is fundamentally different from machine consumption. A person reads one article, maybe shares it, possibly makes a purchase. An AI crawler systematically harvests everything to build trillion-dollar models. The economic value extracted per visit isn't comparable.

Budget-Based Internet Architecture

Cloudflare's pay-per-crawl system mirrors token budgets in AI reasoning models. OpenAI's o3 has computational budgets allocated per task. Now websites can have access budgets.

This creates economic signals in digital consumption. When your AI agent has a $10 research budget, it forces prioritization decisions. Which sources are worth paying for? Which information commands premium pricing versus commodity rates?

The Subsidy Question

The critical question: Will AI companies actually pay these fees, or will Big Tech's advertising subsidies absorb the costs?

Google can afford to pay website owners because they monetize that data through advertising. The current AI boom is heavily subsidized by venture capital and tech giant cash reserves. When that funding model changes, we might see two distinct internets emerge:

1. Premium Web: High-quality, human-curated content behind paywalls

2. Commodity Web: Ad-supported, freely crawlable content

The Advertising Model Problem

This creates challenges for advertising-based business models. How do you show ads to an AI agent? When GPT-5 will research "best running shoes" for a user, does it see sponsored content? Does it disclose paid placements?

The advertising ecosystem assumes human psychology and behavior. AI agents don't impulse-buy or respond to brand awareness campaigns. They optimize for user objectives, not advertiser objectives.

The advertising subsidy model breaks down in an agentic world.

Market Maturation

What we're seeing is the internet economy maturing. The "free" web was never actually free, it was subsidized by advertising, data harvesting, and venture capital. We're moving toward direct value exchange.

Content creators can charge based on actual usage rather than hoping for ad revenue. AI companies access premium content by paying market rates. Users get better AI responses because models train on higher-quality, compensated data.

This forces both content creators and AI companies to optimize for value rather than volume. Better content gets rewarded. Better AI gets access to superior training data.

The web isn't dying, it's finally getting a sustainable economic model.

Modern Chaos explores the intersection of technology, business, and society in an age of rapid transformation.

P.S. If this analysis resonates with you, forward it to a colleague who needs to hear it. We're building a community of leaders who think critically about AI implementation.

Reply and share your story—I read every response and feature the best insights in future newsletters.

Until next Thursday 🎉

Cheers,
Olivier

Like this newsletter? Forward it to a friend and have them sign up here.

Until next Thursday 🎉