System Design distributed web crawler to crawl Billions of web pages | web crawler system design

Learn webcrawler system design, software architecture
Design a distributed web crawler that will crawl all the pages on the internet.
Question asked in most of the top company interviews like GOOGLE, FACEBOOK, and AMAZON
Let's learn how to build google sipderbot or google distributed web crawler.
#crawlersystemdesign
#systemdesigntips #systemdesign #computerscience #learnsystemdesign #interviewpreperation #amazoninterview #googleinterview #uberinterview #micrsoftinterview
#crawler #webcrawler

[@sumonmal009]

estimation 5:30 HLD 6:33 queue manage 25:30 update and duplicate handle 33:40 Sim hash 39:26 storage 42:00

[@arvindaaswani1303]

Awesome explanation, As a engineer i know that, how much hard work behind the scenes. Really Appreciate 👏

[@adamhughes9938]

Makes me sad that this dude crams so much amazing content into these videos and gets 42k views but the dumbest 10 second videos get millions of views... I wish youtube had a notion of content score and quality.

[@junjiechen7341]

ikr! too much going on to be fully appreciated in his vids.

[@warriorgeneral2735]

Hey it totally depends on what people are interested in...

[@Sarah-il5dr]

Guys, please like the video, as an engineer I know how much hard work behind a video like this. This is my go to system design resources. Great work!

[@Sarah-il5dr]

Goutam Singh well, some of the audience might be “engineer-to-be”😉

[@ghisskartadchoo3618]

Fake comments

[@iitgupta2010]

Finally you start building the video in actual flow, that's really great and it will really help the viewer to understand and build the actual knowledge of SD. Great bro.

[@ksenthu]

The more detailed and clear content of crawler design I've seen. Thanks for doing this. It would be great if you can also clarify how the data transition happens between various services such as Extractor, Duplicate Detection, URL filter and Loader.

[@petar55555]

Great in detail System design. The only part I would probably skip is the heap (each queue is already tied to a thread/worker) as it looks more like a bottleneck and serves only as a timer to slow down the crawling for politeness which can be done in different ways.

[@aarushjuneja6640]

I was also thinking on the same lines.

[@NANDINIGOEL]

Think it this way, priorities based queue and then host based but you don’t know once in hosts which host to handle first ( priority is lost) so pq is filled with first elements of each of back a and then urls downloaded based on priority ensuring politeness. Merge k sorted arrays is good pointer to this , there is no point in locking threads to each queue if that is doubt because then priority is per q and not across all. Think a host has all priory 100 urls and others have 1-99 so then why should that 100 host be prioritized, it should not be unless we implement nice call something similar to increase priority to avoid aging

[@monikaa8230]

I have a suggestion to include two things in your videos which will definitely help: 1. QPS Calculation 2. Sharding key when we are planning to shard the DB

[@TheMdaliazhar]

Thanks for this. Most detailed design. No other youtuber explained exactly how the URL Frontier works.

[@jessica-mx5pw]

thank you for the video! this was by far the most helpful system design video walk through I've seen. I've been struggling a lot with system design. Thank you for putting this together!

[@venjan21]

Generally I don't post comments but this is one of the best system design (in detail) I have ever seen. It has re-kindled my thought process on how to think for a System Design question.

[@howellPan]

Great content.. appreciate the details and thoroughness!

[@sampathkarupakula7647]

making things clear and easier, thanks for your effort. I really appreciate your efforts.

[@ajaypremshankar]

It's not easy to make such in-depth content-rich video. Thank you Narendra :)

[@prathashukla6596]

awesome explaination of all the high level components. Good job

[@PeterParker-vn2hv]

Narenda, thank you for this excellent video. Much appreciated.

[@adrianliu2817]

You are the best! Enjoyed all of your system design videos!

[@anastasianaumko923]

Thank you for this elaborate design, great work!

[@manojbgm]

Awesome, knowledgeable. thank you for the video

[@JM_utube]

thank you so much for posting! i love your videos. i just got asked this in a facebook interview and i wish i had seen this video beforehand.

[@akshaymonga]

very nice n detailed video, thank you sir!

[@pryansh_]

very informative, thanks

[@heller166]

This is going to be a lot of help for my distributed systems course :). Thanks for all the hard work.

[@ashish0687]

Thank you Naren, These video's are great source of learning. Very much appreciate the details/time/efforts on your part to build the content and present/share it across. If possible can you also please make a video about Geohashing (& usecase around performing geospatial searches) ...

[@ragdoll2324]

Very detailed discussion. Thanks for making this vdo.

[@ShabnamKhan-cj4zc]

Thanks a lot for exlpaning all the modules in simple manner.. Your channel is the place where one can stop and learn everything in easy way..thanks a ton and keep doing this great work

[@rhythmPhil]

Thanks for your work. This was really interesting.

[@theFifthMountain123]

Had to watch multiple times to understand everything in the video. Thanks for the awesome explanation!

[@aleeshaali7180]

Bes channel I came across for learning about system design, Thank you and keep it up Kudos to the wonderful work!!!

[@t4ruvk107]

Thanks for your time,efforts and content.

[@iitgupta2010]

I really really appreciate your effort bro, whoever ask me I always suggest your name first. There are few others like gkcs but if you ask me there are nothing in front of your design skills. You really talk about things which matters. This is something I have not found in even paid courses. This is awesome in one word. You should have lot of subscriber. They will be soon.

[@harishaseri]

Best explained. Thanks u so much naran

[@iitgupta2010]

I crawled a word from this video is "basically" and inverted index it ....lol [don' have that much time 😝 Great video as always

[@impossible7434]

such an amazing explanation, thank you very much, keep up the good work

[@manmohanakash4222]

This is the kinda of teammate I would like to work with. So much content. Thanks for sharing

[@argstutorial2916]

Very nice conceptual explanations & tools utilizations. You have put a lot of energy with R&D. I hope this will help who are seeking to develop their own system for data processing / scraping mechanisms. Great Work, Keep it Up MaN.

[@Akashkumar-md6rg]

Thnq sir!! For such a great content. Your videos are the most practical and interesting way to learn CS. You made me your fan sir... I really appreciate your hard work. Keep going.🙌🙌

[@alokuttamshukla]

Thank you so much for these efforts. I mean 45 minutes video is not a joke with so much to grasp.

[@TechDummiesNarendraL]

I am trying make it short. But failed to do so

[@alokuttamshukla]

@@TechDummiesNarendraL No , I am in no way complaining at all. I loved it. I am so thankful to you for this.

[@TechDummiesNarendraL]

@@alokuttamshukla thanks

[@readingsteiner6061]

@@TechDummiesNarendraL Blaise Pascal, In his Lettres Provinciales, the French philosopher and mathematician Blaise Pascal famously wrote - "I would have written a shorter letter, but I did not have the time." : ) Buddy you're awesome. Keep up the good work. Wish you the best.

[@JM_utube]

after watching a lot of system design videos i really had to understand that this level of detail is NOT EXPECTED in an interview. i really stressed myself out trying to ask so many clarifying questions, and cover every single aspect of a system in a 45 minute block. this is not expected. remember - these videos are edited, shortened, rehearsed, and practiced. trust me when i say set a lower bar for yourself for interviews LOL thanks!!!

[@SkyCityInc]

This was a really, really excellent overview, thank you for putting this video together!

[@utsavkapoor6069]

Great explanation man. Loved your videos. Why have you stopped making these. Hope to see you back soon!!

[@theranajayant]

Heyy Narendra, Quite interesting topic you have chosen and it's interesting to learn this topic. You are curating really good and valuable content.

[@vishalmahavratayajula9658]

Awesome video. Can't thank you enough narenndra

[@user-hj2lb8mg8o]

Hi, really awesome videos, thanks!

[@AyushRaj-so3zh]

This was GOLD !! Amazing content

[@spyros5528]

Superb video, very helpful. Thank you.

[@sayantanray9595]

Helpful and detailed!!!

[@chaitanyareddy9848]

Dude it's awesome job.

[@pinkylover911]

A lot of great effort has been put into your videos, thanks

[@RealAbhishekSingh]

wow, such great explanation, thank you :)

[@elachichai]

Definitely helpful ! Appreciate it Narendra!

[@neoli8110]

why do you need a heap? it sounds like a bottom neck right there. why can't backqueue selector use LB like round robine select the queue and remove item from the queue.

[@w.maximilliandejohnsonbour725]

Nice info...!!!!!.

[@roooooot9545]

Great work

[@apurvasharma2853]

Excellent explanation!

[@helikopter1231]

Wow such detail and explained so well! Thank you so much! You actually made it sound interesting haha - im not a huge fan of web stuff but this actually made me curious.

[@aashnavaid6918]

amazing video thank you so very much sir!!!

[@vedant9173]

Sir, thank you so much for these great lessons

[@renon3359]

Great video man. You deserve much more subscribers.

[@PiyushSingh-vx7bx]

Amazing explanation brother 🔥

[@keshavKumar-le4df]

Nice explanation.

[@aliaksandrsheliutsin2374]

Just have to say that it's amazing content. Ket it up, Narendra!

[@hlibpylypets1333]

Very detailed explanation - best ever :)

[@SimranGupta-pz7nw]

Thank you so much for the beautiful explanation :)

[@Imkflow]

Thanks for the work on this, very helpful. Quick note, I think if every processor need to receive the same message what you need is a topic instead of a queue.

[@IdoKleinman]

Good stuff! Thank you. One suggestion, for the next video, keep the information text slides on screen for more than 300ms...

[@StormcastMarine]

Thanks a lot for the video mate, really useful

[@wellingtonrafaelbarrosamor4260]

Awesone didactic

[@Wei-up2jn]

Great content! One question I have in mind is why we want to use one queue for one host? Is it because of http connection overhead if you connect to different host back and forth is high? But in realability the URL coming from front queues might be mixed with different hosts, e.g. a.com/a, b.com, a.com/c, in that case we still have to connect back and forth (assuming we only have one back queue). Unless we could guarantee that all URLs from the same host will come together to the back queue router.

[@puravshah2342]

Hi Naren, thanks for the awesome video, can you also make a video on designing distributed scheduling system

[@TechDummiesNarendraL]

Sure

[@samirhere4341]

Great video. Keep up the good work. Can you do system design video on amazon fresh/getbojo/blue apron/plated/embrace box/trytheworld. The concept of how subscription and continues reoccurring delivery system works. Thank you

[@stalera]

Thanks a lot for taking efforts to build up the video. This was amazing. I learnt a lot from this video. Just 1 question: why would you want to store the file content in the compressed form. Is it being used anywhere later? I couldn't find any mention about it.

[@shreyasns1]

@Narendra, Thanks for the video and detailed explanation. Could you also add the links to white papers you mentioned in the video description? This would help us to dive deep further to understand the concepts. Thanks again

[@subee128]

Thanks

[@gouravkhanijoe1059]

Nice

[@shreyade5000]

Nice content but long pause at 40:31, it distracts you if you are listening with concentration. Please edit it.

[@augustoclaro]

I have watched this video so many times in the past year that I'm almost quoting every word you say

[@dharmendrabhojwani]

awesome

[@puneetpatwari]

Nice video. I have 1 question. In the URL frontier, there is a heap. I want to know if the heap is stored at only 1 place and is thread-safe?

[@pengli7213]

What is the implementation of back queue? I don't think it's a Kafka queue, right? Or there might be too many topics. I guess it can be a key-value data structure, such as [domain_name, url, fetched(boolean)] ? Each time when we want to get a url from the "back queue", we just query the key-value and get a url which is not fetched ?

[@DebasisUntouchable]

Great video! Thanks for sharing! Can you please refer me a book where I can get such great examples on System design?

[@nazmavazid9141]

Very very nice sir

[@forte9910]

one question on front queues: if a site is newly added (perhaps as the result of being linked from a site previously crawled), will you leave it in the front queues forever and periodically crawl it?

[@meetpatel5054]

Instead of coupling back-queues with threads, I would say have more number of threads for priority URLs and less for others. for this to work, we can handle the politeness at front-queues where we put the subsequent URLs in low priority queues.

[@nikhilagrawal8888]

amazing

[@kartik-agarwal]

Kudos

[@CODFactory]

a) Why not use a graph db instead of bigtable or anything b) why do those envelope calculations like 6PB or anything when we never used it and we never proved that the design will handle that amount of data c) We definitely should talk about how to make it distributed since 1 crawler cannot crawl everything, so how are we going to make sure that multiple crawlers are not crawling the same things d) how are we going to store these documents in different db and what kind of sharding we are doing to use i think those are some important things to talk about especially giving interviews

[@parupatimadhukarreddy6972]

Hi Narendra, I am basically a software developer who mainly deals with Java script technologies. I saw this videos of Distributed systems on your channel, it seems more interesting knowing the architectural front of the web space, even a newbies are able to understand the conceptual part of the subject Appreciate your efforts. What are the technologies or tools that i need to learn or start with to get to know more about Distributed Systems. Thank you

[@iitgupta2010]

I think we should decoupled the priority based crawler to normal crawler otherwise due to back queue router, all low priority crawler will be starve and never gets the chance to get crawl. We can have two/more system which are responsible for crawling every minute or less (like share market), every 5 minute or 1hr ... 1 day or week up to 1 month. This way we can scale them very easily and manage them better. This also help us to build politeness too.

[@vishalraut20]

What is the purpose of Redis? if we are pushing the entries in the queue, what is the need of cache?

[@rishabhnitc]

As always excellent. just remove the music at 46 second mark :)

[@Tony-cy2yr]

Is someone knocking on the other side of the wall at 40:46? I saw you are waiting them to finish. :) BTW, a question, at 13:56 when will the obsolete persistent storage on the bottom right be clear out?

[@ramesh4joylife]

It would have helped much better if you had gone through this entire thing with an example crawl from a scaled site

[@chickentikkasauce1301]

Heap is an implementation detail. Im being nit picky (this is a great video) but just some thoughts - Why does time stamp based priority even matter in this system? You didn’t mention that. It could be because you don’t want certain queues to get starved. A simpler approach might be to process each queue round robin and only mention the priority queue to your interviewer if they nudge you in that direction or if you want to slowly build to it to discuss trade offs. If each back queue has a priority, then just call out that we want a priority queue. You could say back queues have same priority but maybe other back queues dedicated to urls that we expect are updated at a faster rate have higher priority. But then you need a solution to the problem of other lower priority queues getting starved.

[@psn999100]

Great explanation. Yes . What I gather is that "URL Frontier" essentially implements a 1. Priority selection . -> Front Queue 2. Politeness guarantee . -> Back Queue The main issue what we are looking at is how to pick the next URL from the "URL frontier" microservice to be sent to a thread for processing. As you said, we could do a round-robin method where all Back queues get picked from in an equal - fashion. Or kind of a "weighted" method aka. priority_queue based solution to make sure the hottest websites get crawled in smaller/tighter time intervals. I think its always better to just give the simplest approach first (i.e just draw a black box tagged "Queue Selection" ) and deep dive later if the interviewer wishes to. There is a saying in system designing world = "KISS" == Keep It Simple and Stupid . Its' unlikely that you would run your interviewer out of questions, so better to even nudge the interviewer in your direction of thinking by giving out ever so slightest of hints, so that he starts asking the questions which you already have the answer to.

[@RAJESH2010able]

Hi Narendra, will it be possible for you to do a video on 'Design Online food ordering service like Uber eats/doordash and explain how to integrate it with existing (Uber) ride-sharing service'?

[@karllopes]

What is the connection between the auxiliary table and the Heap? I know the table is used to store the host/back queue # info and the heap is used to store the back queue #/last time visited info. I do not see how the two fit together in the workflow. i.e. When is the table queried/Why do we need to store the host/back queue #? Is the table only used for the BackQueue Router to know which back queue to add the current URL?

[@ambermani1667]

19:06 why we directly jumped to conclusion to use bloom filter? why can't a distributed hash table will work to know if a site is already crawled or not. its not O(n). we can hash the urls and shard the urls based on hash, then search the url in specific shred hash table.

[@nitinkulkarni7942]

Naren, if size is a problem @45:00 and therefore NoSQL is not recommended, what about Redis, how can we still use Redis as cache for the same pages?

[@mtsmithtube]

@16:38 "make it a standard convention of converting it to a lowercase" - careful because URLs are case sensitive. Maybe your duplicate detector should do a case insensitive compare but you don't want to lose the original case when saving urls.

[@NdubisiOnuora]

Great explanation! You made it seem easy. How long did it take for you to understand the whitepapers for the Googlebot?

[@scalechamp]

It's not the way google bot works, it's the Mercator Web Crawler architecture

[@RahulSathe.07]

Hey Naren, awesome video. What would be a good (& scalable) way to keep track of duplicate URLs?

[@NANDINIGOEL]

Bloom filter / count min sketch