Podcasts · Search Off the Record
What is a web crawler, really?
· 30 min · Episode 70
Episode What is a web crawler, really?
In this episode of Search Off the Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way.
Resources:
Episode transcript → https://goo.gle/sotr070-transcript
Managing your crawl budget → https://goo.gle/3IzRZxl
Dave Smart on LinkedIn → https://goo.gle/3wPSuRA
Tame the Bots → https://goo.gle/4cfCQ1P
Search Central Help Forum → https://goo.gle/sc-forum
Indexing API docs → https://goo.gle/3v8yVU0
Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
#SOTRpodcast
Speaker: Gary Illyes, Lizzi Sassman
Products Mentioned: Search Off The Record
Links in the notes 6
Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.