What is a web crawler, really? | Search Off the Record

[@GoogleSearchCentral]

What other questions do you have about crawling? We wanna know! Let us know below ↓

[@musluy]

__Nextdata__ -> page and query

[@MariaFernandaGomez1000]

I got a lot of spam urls like casino that are indexed, from an attack. I blocked in robots but still is indexing. I ask for deindexing in GSC but are still there. How I can deindex those url spam?

[@fdmodhia]

I’m currently managing an e-commerce product page that sells prescription-only medicine. Despite the website's overall good performance, this particular page has encountered significant visibility issues on Google search results over the past few months. I am contacting this knowledgeable community for advice on addressing this challenge. Background Performance History: The page used to rank well on the first page for several competitive keywords. Current Issue: It vanished from Google’s search results a few months ago. It is now only accessible through Google’s cache URL or the SITE operator. Content Strategy: It’s worth mentioning that my website does not utilise AI-generated content. Attempted Solutions URL Change and Redirection: My initial attempt involved changing the page’s URL and implementing a redirect from the old URL to the new one, which did not improve. URL Change Without Redirection: Subsequently, I tried creating a new URL without redirection, maintaining the same content. This approach also failed to resolve the issue. Current Consideration Given the ineffectiveness of previous strategies, I am contemplating developing new content for another new URL. However, I’m uncertain if this would effectively solve the problem. Seeking Advice This issue is new to me, and despite my research and observations of similar discussions in Google forums, I am still looking for a solution. Therefore, I am appealing to this forum’s members for any insights, suggestions, or proven strategies to help overcome this SEO hurdle.

[@fdmodhia]

Why I am not able to comment here?

[@sawpaing676]

1:35

[@MariaFernandaGomez1000]

Also, I got a los of spam urls like casino that are indexed, from an attack. I block in robots but still is indexing. I ask for deindexing in GSC but are still there. How I can deindex those url spam.

[@sam28407]

Deindexing in GSC helps, check them after 2 weeks again

[@MariaFernandaGomez1000]

@@sam28407 its been like that like 2 months ago.

[@ETechBuy]

Googlebot / Crawling is not fetching the proper meta title for the site build on React JS (server-side render)

[@KeithGoode]

I always wondered what Gary's full name was. Now I know it's "Sometimes Gary Illyes."

[@ctomczyk]

Crawler is useful for the inventory purpose, but can't work effectively with single page applications.

[@BrianDuncan-wz3uu]

Web crawler looks like a millipede and runs across the forks . It troubles me to sele them this huge being used to violate women an children there's pics in my kids an pics of said bugs on folks who are being hamed

[@MariaFernandaGomez1000]

Also, I got a los of spam urls like casino that are indexed, from an attack. I block in robots but still is indexing. I ask for deindexing in GSC but are still there. How I can deindex those url spam.

[@andreea007]

Robots txt blocks crawling. It does not block indexing. For deindexing we generally use meta robots tag (noindex). But since you say those are spam URLs, why not just delete them? Works like a charm for deindexing 🙂

[@VilmosPikacs]

0:33 ​@@andreea007

[@DannyArlt-ly5ht]

😂😂bSa😂 1:41 ​ 2:36 2:40 @@andreea007