What is a web crawler, really?

In this episode of Search Off The Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way.

Resources:
Episode transcript → goo.gle/sotr070-transcript
Managing your crawl budget → goo.gle/3IzRZxl
Dave Smart on LinkedIn → goo.gle/3wPSuRA
Tame the Bots → goo.gle/4cfCQ1P
Search Central Help Forum → goo.gle/sc-forum
Indexing API docs → goo.gle/3v8yVU0

Search Off The Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

#SOTRpodcast

Speaker: Gary Illyes, Lizzi Sassman

[@GoogleSearchCentral]

What other questions do you have about crawling? We wanna know! Let us know below ↓

[@musluy]

__Nextdata__ -> page and query

[@MariaFernandaGomez1000]

I got a lot of spam urls like casino that are indexed, from an attack. I blocked in robots but still is indexing. I ask for deindexing in GSC but are still there. How I can deindex those url spam?

[@fdmodhia]

I’m currently managing an e-commerce product page that sells prescription-only medicine. Despite the website's overall good performance, this particular page has encountered significant visibility issues on Google search results over the past few months. I am contacting this knowledgeable community for advice on addressing this challenge. Background Performance History: The page used to rank well on the first page for several competitive keywords. Current Issue: It vanished from Google’s search results a few months ago. It is now only accessible through Google’s cache URL or the SITE operator. Content Strategy: It’s worth mentioning that my website does not utilise AI-generated content. Attempted Solutions URL Change and Redirection: My initial attempt involved changing the page’s URL and implementing a redirect from the old URL to the new one, which did not improve. URL Change Without Redirection: Subsequently, I tried creating a new URL without redirection, maintaining the same content. This approach also failed to resolve the issue. Current Consideration Given the ineffectiveness of previous strategies, I am contemplating developing new content for another new URL. However, I’m uncertain if this would effectively solve the problem. Seeking Advice This issue is new to me, and despite my research and observations of similar discussions in Google forums, I am still looking for a solution. Therefore, I am appealing to this forum’s members for any insights, suggestions, or proven strategies to help overcome this SEO hurdle.

[@fdmodhia]

Why I am not able to comment here?

[@sawpaing676]

1:35

[@MariaFernandaGomez1000]

Also, I got a los of spam urls like casino that are indexed, from an attack. I block in robots but still is indexing. I ask for deindexing in GSC but are still there. How I can deindex those url spam.

[@andreea007]

Robots txt blocks crawling. It does not block indexing. For deindexing we generally use meta robots tag (noindex). But since you say those are spam URLs, why not just delete them? Works like a charm for deindexing 🙂

[@VilmosPikacs]

0:33 ​@@andreea007

[@ETechBuy]

Googlebot / Crawling is not fetching the proper meta title for the site build on React JS (server-side render)

[@KeithGoode]

I always wondered what Gary's full name was. Now I know it's "Sometimes Gary Illyes."

[@MariaFernandaGomez1000]

Also, I got a los of spam urls like casino that are indexed, from an attack. I block in robots but still is indexing. I ask for deindexing in GSC but are still there. How I can deindex those url spam.

[@sam28407]

Deindexing in GSC helps, check them after 2 weeks again

[@MariaFernandaGomez1000]

@@sam28407 its been like that like 2 months ago.