Podcasts · The Stack Overflow Podcast
Why Stack Overflow and Cloudflare launched a pay-per-crawl model
· 20 min · Episode 918
Episode Why Stack Overflow and Cloudflare launched a pay-per-crawl model
In this episode of Leaders of Code, Stack Overflow’s Janice Manningham and Josh Zhang sit down with Cloudflare VP Will Allen to discuss the innovative pay-per-crawl model co-launched by their organizations. They explore how the rise of AI has disrupted the traditional “open versus block” internet model, creating a need for platforms to protect their content and data from commercial exploitation while maintaining community access.
The discussion also:
• Explores the future of the bot ecosystem, emphasizing the importance of putting publishers back in the driver’s seat to decide how their content is accessed and monetized. • Explains the technical implementation of the pay-per-crawl system, which uses Cloudflare’s bot categorization and WAF rules to serve a 402 “Payment Required” message to specific crawlers. • Highlights the strategic value of data licensing, comparing comprehensive enterprise contracts with the more flexible, programmatic pay-per-use access enabled by the new model.
Notes
• Connect with Will Allen, Janice Manningham and Josh Zhang on LinkedIn. • Learn more about Stack Overflow Data Licensing here.
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Links in the notes 6
- https://www.linkedin.com/in/williamallen2050/
- https://www.linkedin.com/in/janicemanningham/
- https://www.linkedin.com/in/josh-zhang-sre/
- https://stackoverflow.co/data-licensing/?utm_medium=podcast&utm_source=blog&utm_campaign=leaders-of-code&utm_content=pay-per-crawl-episode
- https://art19.com/privacy
- https://art19.com/privacy#do-not-sell-my-info
Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.