I Made a FAST Search Engine

[@conaticus]

Start building awesome projects with $15 free credits using BrightData today: brdta.com/conaticus1

[@AWIRE_onpc]

no

[@xulaxwtf]

no

[@aryanszone4963]

no

[@noviui]

no thanks

[@user-uv3nv2bc6v]

no

[@jaymarksum6542]

I’m impressed, can’t wait to see you build a multithreaded web server in assembly

[@da40au40]

Why do I find it super funny 😅😅😅.

[@ArthursHD]

@@da40au40 Me too :D

[@DanskeCrimeRiderTV]

it's not impressive. Of course querying a few hundred or even hundred thousand web pages isn't as complicated or slow of a task than querying trillions of webpages.

[@KibitoAkuya]

​@@DanskeCrimeRiderTV google also wastes time deciding wether you are allowed to see or not certain sites

[@DanskeCrimeRiderTV]

@@KibitoAkuya what does that have to do with anything? Google is still faster at querying trillions of results than this.

[@lifeofme702]

I don't know what this guy said, and still was mind-blown of all the effort this guy puts

[@conaticus]

Thanks much so 🙏 It would not be possible without your support

[@ccost]

7:40 flashing those questionable websites in a sponsored video is quite the move

[@twitchizle]

You scared of porn?

[@coderx8634]

Love your content. You and your quality have really improved. Keep it up ❤

[@conaticus]

Thanks so much, your support means a lot ♥

[@coderan5029]

This is basically what we learned in my big data class, but we used map-reduce to do the TF-IDF calculations, so it's impressive you figured this out on your own

[@JoshuaLonsako]

W ad plug, it's 100% relevant and actually necessary to fulfill the premise of this vid.

[@6IGNITION9]

filter out JS for another 10x bandwidth savings alternatively use an adblocker. (can puppeteer do that? It's just chromium right?)

[@SG-kn2jl]

Why did you choose TF-IDF instead of word2vec or any context aware model?

[@skorp5677]

+1 Woule like to know

[@devinlauderdale9635]

The problem is this approach is susceptible to SEO spamming/invisible SEO keywords

[@conaticus]

Yeah for sure, realistically it should be moderated based on user interaction as well

[@jjwe2002]

@@conaticus How would you do that?

[@greensporevalley]

SERBIA MENTIONED 🎉🎉🎉

[@RealMephres]

​@europa_the_last_battle>goes to comments >sees meme comment >looks at replies >only a LARPer replied lol

[@jawadmansoor6064]

that name rings a bell, maybe from some kind of Serbian movie?

[@RealMephres]

@@MAXHASS-ph5ib tell that to the LARPer dawg

[@slimeyar]

​​@@RealMephrestell that to yourself 😊

[@RealMephres]

@@slimeyar you first

[@80sVectorz]

3:07 Best pronunciation of Euclidean I have every heard :P

[@CrazyDiamondo]

Where?

[@80sVectorz]

@@CrazyDiamondo I added a timestamp

[@anitaweasel]

Nice, you re-invented the lucene library

[@aryakvn6051]

You could calculate and cache TF values on the fly so you don’t fill up your ram as quickly but still get a decent response time.

[@rafaelpereiracoias1047]

Nice video and nice code, keep up the good work!

[@Raven-fu1zz]

Remember, never return an over 18 site without an over 18 word in the search request

[@polyshrub]

This is very impressive, what was the size of the database when indexing is finished? Seems like it would be quite big

[@MySachincool]

Subscribed & notifications on :) you deserve more recognition bruh

[@R_Y_Z_E_N]

Google also does the same but with disstributed computing to reduce the overall time . Just scale the database horizontally and mimic googles apporach

[@turb0004]

Please finish your file explorer in rust fully, because the idea of it is awesome. Love your videos, content is very engaging 🎉

[@foqsi_]

Love this dude and his video projects

[@conaticus]

🙏

[@rmt3589]

Awesome video! Will help immensely when I eventually make an AI RAG search engine. I wanna see if I can add blacklisted and whitelisted websites. That way things like useless citation sites and spam sites cannot come up, but things like Wikipedia and websites I get good results from ahow up more.

[@soulcpp]

Definitely a very great video, keep it up!!!

[@ExpandedCuber]

Let's go another conaticus video

[@dreamsofcode]

🔥🔥🔥

[@MortonMcCastle]

Good! The world needs a new Google Search, one that's more like how it was in the 2000s.

[@madalenaferreira3018]

great video, gave me ptsd from my information retrieval class though

[@alexmoses3215]

Programming 🤝 martincitopants…match made in heaven

[@stayhappy-forever]

thats insane, hows this only at 12k views

[@GermanTimecrafter]

such a cool video! i love the way how you explain what you are doing :) random question but what is your editor font?

[@conaticus]

Appreciate it :) I'm using Jetbrains Mono it's free to download

[@TheRojo387]

In high school, I could outperform search engines of the time. I don't think I can say the same for today's search engines.

[@jsalsman]

I believe it's "inverted indexing", as inverse indexing is something else.

[@miro5182]

You can use a chrome like TLS config to not get blocked by cloud flare in a lot of cases, using a browser for scraping isn’t viable when tracking about scanning the internet.

[@gammongaming9081]

yk what would be funny? making the slowest search engine possible without like halting the program for a set time, just with maths

[@jugurtha292]

very nice, built something similar for my info retrieval class. we have to use okapi bm25 formula for the ranking but overall very similar. scrape, tokenize, parse, inverted index, rank

[@animeworld4775]

what is things that i should to know or learn to create like these projects

[@HyperCodec]

Bro managed to memleak in js

[@yorailevi6747]

how much did you pay for the web scraping service in total?

[@ethanstewart1011]

How did you manage to get a node.js memory leak??

[@thekwoka4707]

How much did the scraping cost if it wasn't free?

[@lonelybookworm]

Well of course it is very fast, it only has like 200 websites

[@maksymilianglowacki1409]

is this engine oneline or ( wouldt it be abel to be oneline for otcher users ) so otcher also coulst enjoy it? or was it dust a peak or somthing you made cuz ( you where bored or smt )

[@errplane_]

oh my fuck i saw this on your github last night

[@iritesh]

Awesome effort ✨

[@allenfpascua]

Super good editing 🫡🫡🫡🫡

[@conaticus]

Would not possible with your breathtaking animations 😄

[@gamedirection_us]

🍎 👀 .. Apple being like "when will it be ready?".

[@gopallohar5534]

ain't see rust there!

[@guidedorphas10]

6:08 nahhhhhhhhhhh whats bro even searching 💀💀💀💀

[@joenutt1232]

Create your own database engine for shits and giggles

[@conaticus]

B+Trees 💀

[@sleepybraincells]

Why is there Rust in the thumbnail? This was written in Javascript

[@conaticus]

Used Rust for the API and TF-IDF matching - decided not to keep in much of the footage for that as it was already explained in the animations

[@MinecraftRecordings1]

whats the link?

[@a6gittiworld]

Supa dope. I would like to use this search engine of yours

[@monotonedevelopment]

If only windows file explorer could do the same

[@SandWire]

For this we have thing named Everything :)

[@callowaysutton]

Next time use the Common Crawl dataset ;)

[@--bountyhunter--]

bro thought he could scrape my web and get away with it.

[@SlimyFrog123]

Now make your own email system to go along with it. 😉

[@synapsenova299-fp7tf]

>goes to youtube homepage >finds this video >yipeee >oh >lets try it

[@FastCarsLoudMusic]

This video is so good. Instant hook.

[@mahrezjanati3426]

first time watching a vid of yours ... i have one question : why are you vibrating ??

[@-rate6326]

Cause he is vibrator

[@InioluwaFalade-Tolulope]

don't know either

[@schoolbreakyay]

Can i not use brightdata?

[@playtatus1758]

how do you edit your vids

[@conaticus]

Allen uses adobe after effects for the amazing animations - I just use Davinci to cut things up 😁

[@playtatus1758]

@@conaticus ok thx

[@daemonkisure2952]

how can i install this search engine?

[@conaticus]

Instructions are on the Github repos :)

[@Faeest]

why disallow and user-agent matter? can't you just scrap everything?

[@skorp5677]

You can but it might be illegal

[@AquaQuokka]

Rewrite your genetic code in Rust.

[@pyyrr]

i would rather be bug free so i will pass

[@Nerdimo]

Impressive, seriously!

[@humanontheinternet6510]

Auto solve captcha you say🧐

[@Miluum]

1:06 automatically solve captchas? i knew these things exist just to waste our time and energy

[@deepfan14]

Bro make a compiler programming language

[@Rudransh-hu]

You should host it

[@Serhii_Volchetskyi]

🔥🔥🔥 I was looking for that algorithm and didn't know its name.

[@_DarkLiquid]

discord clone when

[@iCrimzon]

Cant wait for you to rewrite JS in binary 🎉🎉

[@danielisop3182]

What did u mean by the websites u shouldn’t have searched

[@binpersonal]

"some fucking genius" lmao

[@neologicalgamer3437]

Bro sounds like WilburSoot

[@fangg194]

you seem ok

[@AttaaH]

0:33 🤨

[@chiroyce]

What are the consequences of scrapings sites you aren't allowed to?

[@conaticus]

Probably not much on its own as long as you're not violating copyright - however it is curtious not to scrape sites forbidden by the robots.txt

[@314cubed]

wastes their resources and yours

[@Xanmattauri]

@google acquire this man

[@StellarWeb008]

"Always bet on javascript"

[@v037_]

I found a worthy opponent

[@juniordevmedia]

what TF is IDF ?!!

[@neofox2526]

idk man but watching it makes me feel smart

[@jamesbarret4240]

Term frequency (the number of times a given word or so shows up in total) - inverse document frequency (the number of times it shows up in a specific document). The wikipedia article is pretty good: en.wikipedia.org/wiki/Tf-idf

[@larry_berry]

Lol. Got notif after clicking the video.

[@AhmedMo-ec4kz]

Great video 😊 FYI: bright data is an Israeli company 😮

[@gaimnbro9337]

Nice job :D

[@konstantinsotov6251]

we had a hackathon where we basically had to implement TF/IDF - also a search engine of a sort, but for files. we did the interface in python and all mathematics processing in C++. It would have been a fun experience if not for the time limit. we struggled really hard, on test data our solution worked faster by an order or two than most other participants, but... we somehow failed on the exam data. we failed fucking IO. and won nothing. I fucking hate hackathons since then. fuck IDF. also maybe this happened because i had written 75% of the code, while 4 other members did almost nothing. It was (their) responsibility to handle IO, and mine to handle mathematics and processing. I hate working in teams. I know noone cares but i might as well just burst out all of the rage I have towards that experience. once again, fuck team work, fuck hackathons, fuck my teammates, fuck everything and everyone

[@skorp5677]

skill issue

[@konstantinsotov6251]

@@skorp5677 exactly

[@Ayymoss]

MAKE LONGER VIDEOS

[@ph03n1x_dev]

You made a search engine for porn?! Thats disgusting... is it on GitHub?! 👀

[@conaticus]

All open source and ready to play around with 😂

[@susannerudolph8469]

then brightdata makes captchas useless

[@educacionespecialchannel3756]

Captcha's effectiveness has been in question for quite some time now.

[@Naw1dawg]

protects against amateurs but keeps it simple enough that an expert won’t breach/destroy their data to get what they want.

[@_sohom]

Make a better version of VSCode.

[@kavinbharathi]

Not to be the 🤓☝️ guy, but "Jana Vembunarayanan" is pronounced 'Ja' as in 'Jarvis' and 'na' as usual. Just fyi

[@conaticus]

Thank you, I'll do this if I ever pronounce it again 😂

[@planktonfun1]

Still not fast and scalable enough. The result is not even relevant, you made bing not google

[@LaugeHeiberg]

wow really? Im also surprised one single guy didnt manage to make a product rivaling Google

[@gamefun2525]

wow Sheldon, you got your Nobel yet?

[@Horn7xGaming]

hub 🎉🎉

[@Naw1dawg]

So you’re telling me I can access restricted data by telling it to, basically, ignore restrictions??? I Have been calling myself dev, admin, ownr, root in vain for far too long

[@lukamajcenic1172]

This is just an ad for BrightData. Compared to previous videos very low effort.

[@TheRealMangoDev]

good vid

[@ssoka-m5n]

rust is a real badass❤❤

[@igrb]

nice

[@J0Y22]

shockedd

[@Macellaio94]

Liked and subbed