Crawling smarter, not harder | Search Off the Record

Рет қаралды 6,996

Күн бұрын

In this episode of SOTR, John Mueller, Lizzi Sassman, and Gary Illyes talk about misconceptions around crawl frequency and site quality, what's challenging about crawling the web nowadays, and how search engines could crawl more efficiently.
Resources:
Episode transcript → goo.gle/sotr07...
Gary's post on LinkedIn → goo.gle/3YAT55q
Crawling episode with Dave Smart → goo.gle/3WShUsf
If-Modified-Since → goo.gle/3ywXvja
About the IETF → goo.gle/3SGVVlo
Robots Exclusion Protocol → goo.gle/4dgmBSg
Proposal for new kind of chunked transfer → goo.gle/3AgMF1c
Watch more Search Off the Record → goo.gle/sotr-yt
Subscribe to Google Search Channel → goo.gle/Search...
Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
#SOTRpodcast
Speaker: John Mueller, Lizzi Sassman, Gary Illyes
Products Mentioned: Search Console - General

Пікірлер: 17

@KempHoneycutt Ай бұрын

Regarding parameters, the former URL parameters tool in GSC seemed to handle this situation. When that was taken away, we started seeing TONS of parameterized URLs showing up on SERPs and in our analytics from organic search (including URLs with advertising attribution parameters, which is especially fun). Now it's a problem????? When it went away Google told us they were able to figure out which parameters they could ignore, but....evidently not. Maybe just bring that tool back and let us take care of it ourselves? It was working.

@PreparelikeJoseph Ай бұрын

00:13 Discussion on crawling updates and strategies. 02:53 Google's crawling frequency does not necessarily indicate site quality 07:54 Site owners may want increased crawling over time. 10:19 Optimizing crawling process by understanding server limitations and content quality 15:32 Sending back only headers in HTTP response makes it more efficient 17:38 Implementing compartmentalized changes can save resources. 22:05 Optimizations for crawling include URL parameter handling 24:22 URL parameters impact crawling efficiency 28:42 Challenges of smart crawling 30:53 Crawling efficiently depends on site size and resource awareness 34:53 Hosting companies should assist customers in troubleshooting website crawling issues 37:16 Client-side connection problem solutions

@mr.nagendrayadav5735 Ай бұрын

00:13 Discussion on crawling strategy and updates 02:53 More crawling does not necessarily mean a better site. 07:54 Website owners may request for increased crawling over time, but it's not always feasible. 10:19 Optimizing crawling efficiency and server impact 15:32 Sending smaller 304 response for efficiency 17:38 Crawling smarter can save resources for the internet. 22:05 Crawling optimizations discussion 24:22 URL parameters can result in infinite versions of a page 28:42 Challenges in optimizing crawling methods 30:53 Importance of awareness and resources in crawling efficiently 34:53 Hosting companies should assist customers better in troubleshooting crawling issues. 37:16 Improving client-server communication and messaging strategies.

@vanchanges6383 Ай бұрын

Thank you too

@digita1dope Ай бұрын

Who determines the quality of content?

@sactiest Ай бұрын

This is the funniest episode, lmao John's on point with the jokes

@KeithGoode Ай бұрын

I love the personality switch between Gary and Lizzy. :-)

@Grzegorz-c8w Ай бұрын

So how Google bots honour sitemaps that contain the date of last modification in .... tags?

@JohnMueller Ай бұрын

A lot of sitemaps get these wrong (many always use "now" as the last change date), so it's a bit hit & miss. If you can do it right, then do it. If you can't set it properly, then it's better to submit the sitemap without dates (which is also fine).