Search Billions of Records in Seconds! from database | Advanced Backend

[@soumo.das23]

Please make a video on how determine bit array size and hashcount . That would be really helpful. By the way thanks for sharing your knowledge ❤

[@pankajdubey1123]

sir aapke videos bohut jyada helpful hai accha backend banane ke liye . please sir content deliver karna jari rakhna

[@sandeepdev0]

❣️

[@anuj7286]

Great tutorial. I was watching bloom filter tutorial on another channel. Thank you

[@rewindtube2406]

Keep making similar kind of videos on different algorithm too.

[@sairosestha]

thank u brother for this informative video. Love from nepal

[@mattiullah318]

Today learn new concept ..thank for this kind of videos

[@shivamdubey8]

Great video, I have heard about this algorithm first time. Thank you for this precious knowledge sharing.

[@sandeepdev0]

I'm glad you found it helpful! Stay tuned for more 😊

[@shivamdubey8]

@@sandeepdev0 yes sure

[@AshishKumar-cl5ik]

clear explaination , Greate Knowledge with good spirit.

[@jeetvadera3217]

Nice video. You have great knowledge thaks for sharing. Keep it up

[@shubhamdas6519]

Thanks for the video sir

[@kaushikroy000]

Nice video bloom filter algorithm chahiya

[@somyaranjandas4599]

please make video on redis bitmap and how to determine bit array size

[@sandeepdev0]

Noted✅

[@MuhammadAli-f9f]

Thanks for your this awesome video, I've question suppose I'm implementing your given technique and I've 10k users added in DB and now implementing the "Bloom Filter". Suppose Sandeep name is already in DB while my index has 0 value? How manage this? because if I go with your given logic with Bloom Filter it says its unique name which is not.

[@moazamali2434]

thanks for sharing

[@aviralgupta9869]

at 10:05 shouldn't we doing modulo instead of dividing the number, because dividing a number by 1000 can also have a result greater than 1000, which will be out of index

[@VivekKumar-nq2ll]

Sir, Iska advanced version ka video jrur bnaiye

[@rummanujjaman4727]

Sir, as we all are familiar with node and express js along with mongodb. But you are making blockbuster videos using hono js, Postgresql etc. I think that's the reason people are not attracting with your video and I really want your video to get hype. So please try to use node/express for your great lessons. 🎉

[@sandeepdev0]

Thank you so much for your feedback and support! 🙏 I completely understand where you're coming from-Node.js, Express, and MongoDB are incredibly popular and familiar to a lot of us. I decided to cover Hono.js, PostgreSQL, and other tools because they offer different capabilities and can really enhance our skill sets as developers. Learning new technologies can open up even more opportunities and help us stay current in the industry. 😊 I truly appreciate your encouragement and will keep your suggestion in mind for future videos. Thanks for being a part of this journey! 🎉

[@prakashchandramourya872]

Does it work like google search, example for 'san' the output should be true.

[@RandomGuy34-j1u]

but bloom filters ki ek limitation hoti hai right ki woh false positive de sakta hai ? Like maybe woh username database me exist nhi karta hai but still bloom filter may return ki that value exists...In that case hum user ko ek aisa username banane se rok rahe hai jo ki allow hona chayiye.

[@sandeepdev0]

Yes you are right, but there are further modified approaches based on bloom filter to get rid of this problem

[@mujibulhaquetanim]

@@sandeepdev0 make a video on this too, i think this is as important as applying bloom filters and mention how to determine the value should be given for it too 23:44

[@anujpandey3101]

if the index in bit array are already marked it didnt always implies that the username is already created , how to handle this also tell us about why hashing is done three times ans size of bitarray is taken 1000 btw nice explainataion and coding part

[@sandeepdev0]

You're absolutely right; even if indices in the bit array are marked, it doesn't guarantee the username already exists, due to false positives in Bloom filters.To handle this, there are some further modified approaches/variations of bloom filter. ( we will discuss it in a live stream or upcoming videos ) As I mentioned in the video, hashing three times and using a bit array of size 1000 is a practical balance for memory efficiency and false positive reduction. There are also advanced Bloom filter variations to reduce false positives even more. I’ll dive deeper into these modified approaches and how to determine the length of array and hashcount in upcoming videos and live streams-stay tuned!

[@ajithamin9196]

Please make video on how to look for opensource project

[@sandeepdev0]

sure👍, thinking about doing a live stream on this sunday/saturday, there we can discuss this

[@ajithamin9196]

@sandeepdev0 ,sure we can have a discuss there

[@blackhat5458]

@@sandeepdev0 Live Stream time? Because I don't want to miss it.

[@sachinkoli-3935]

this algorithm need to be implemented in app since inception otherwise it will not work right?

[@future_gen]

Bro can you reply me that why you don't use functions instead you use classes???

[@sandeepdev0]

Hi @future_gen, Good question 👍actually using classes over functions includes 2 main reasons for me, 1. My personal choice 2. Low level design concepts can be implemented better in classes than functions ( Personal experience)

[@mandipsingh6341]

What if, if we delete a user from db , how we can delete the same record from BF ?

[@ChessPuzzlesVideo]

Good question

[@vikasrai4915]

Just do the reverse, i.e unset the value in BF at those indexex by calculating the hash again.

[@SuryaPratap-r5h]

Sandeep bhai! where is the kafka crash course?

[@sandeepdev0]

Hi Surya, Actually I have to made that kafka Crash Course video private ( Because one confidential system key got leaked in that video) So I made that video private, but don't worry, soon I will re upload the video...

[@SuryaPratap-r5h]

@@sandeepdev0 Hey Sandeep, thanks for considering my query. I will be waiting for that video.

[@mukulkanojia576]

Agar koi unique h kisi ka hash ki index usse match kr rhe h or usne but one kr diya h to kya unique koi false dega

[@vishalgarna12]

❤❤❤

[@soumo.das23]

How to delete an element from the bloomfilter?

[@at_vlogs]

Hi Please explain why we are hashing it and storing it three thime in a single array of size 1000 instead of that we can directly store a username and search it by includes method ? Please let me know if anybody can help me

[@ChessPuzzlesVideo]

It's all about time. Think about array with size million. Will that includes work efficiently? Check internal implementation of includes.

[@at_vlogs]

@ ya got it thanks

[@rummanujjaman4727]

What if we delete a user from db ir user delete his acc or change his username then how It's gonna work

[@sandeepdev0]

well, there are some modified versions of bloom filter like counting bloom filter algorithm which allows us deletions, and also there are some other modified bloom fliters algorithms ex: segmented bloom filter. So basic bloom filter algorithm not allows us to delete directly, So, majorly we use modified bloom filters algorithm ( will be covering in upcoming videos )

[@AshishKumar-cl5ik]

request to please create advance video

[@roshandev8167]

apne bola ki database ka use nahi karna hai. mein apko bata du ki jo array aap dekh rahe ho index ko check karne ke liye wo v ek type ka database hi hai. Yahi same chiz hamlog bhot easy tarike se direct database se v kar sakte hai direct database pe index laga ke. Database automatic hi userName ko numeric format mein convert karke numberline pe arrange kar deta hai. And doing this process is lot more faster and efficient compare to hash and then store in array and then find in array index. Kyuki database data ko store karke and retrieve karne ke liye hi optimised hota hai using combination of cpu and Gpu power I suggest you to please check how database do data computation in insane speed jabki agar hamlog direct calculation c++ se v kare itna bada data toh v utna speed nahi ayega. Ye essa isliye hota hai kyuki database data to store, query and compute karne ke liye optimised hota hai like mongodb. Please like my comment if you think this is informative.

[@piyushraj7275]

First, let’s clarify: an array is a linear data structure, not a database type. While indexing can indeed speed up searches in a database to O(1), it comes with performance trade-offs, especially when modifying data. When you index a column, an additional data structure is created to store the locations of values in that column, which enables faster searches. However, this index must be updated every time data in that column changes, affecting performance during insertions, updates, or deletions. For example, if a billion users exist and you index the username column, both registering new users and allowing existing users to update their usernames can slow down the process, as the index needs continuous refactoring. Additionally, making two database round trips-one to check if a user exists and another to insert new data-is generally inefficient. For instance, if one round trip to the database takes 500ms, two would double that time, resulting in a 1-second delay for each new user registration.

[@roshandev8167]

@piyushraj7275 i agree with your point that for every new record insert we have to refactor database index but if you keep data in array then also that array is stored in ram memory. We can assume it a kind of in memory Db. Now even if we store this much huge data on RAM then cost will increase with every new record. And this is RAM cost instead of ROM cost which is costly considering billion of records. In case of ROM cost only few sectors of data is refactored which is not that much let say 500ms. but then after that time username availity is done in O(1). again with very mimimum cost. Also keeping that much huge array will slow down service startup time.

[@dev_guidance]

real life me agar implement kr k dekh skte to mza aaa jata with 1000 users or 100 users kuch bhi chlega using redis

[@sandeepdev0]

Noted✅, Probably will do in the next video

[@wafiqhosain5713]

A silly ques :- if a database has table called "User", thst has 1 milllion user entries. If i use redis as caching layer, should i cache all user entries/ a specific number of user entries in my redis cache so that while signin/signup I don't have to query the db.

[@sandeepdev0]

If you have 1million user entries in your DB and want to use redis as a caching layer, you have to store all the user's unique key (ex: email, userid)- But this will be highly unefficient, to make a it a little better, you can do sharding, replication on some basis,But all this will bring unnecessary complexity in your backend, So, Bloom filter algorithm is highly recommended for your example case...

[@rajeshbhusal1910]

You need to cache all the users. What you can do is store the username#. Whenever you need to find the username just use the get command. As Get and Set command has time complexity of O(1) the operation is faster the only cost here is the socket trip between your program and memory. I think this is even better than bloom filters as in bloom filter you need to hash the username and set the bit if the bit is already you need to query the db which is little overhead

[@wafiqhosain5713]

@@rajeshbhusal1910 is it safe to store user credentials(email/password_hash,username...) in redis, store 1 million user records in redis (for caching purpose) . As a fresher, i never work in any corporate level project. So never saw how this cases actually handles.

[@anikislam2150]

Is ke bareme or janna hai

[@rajeshp9319]

What if two of the users create at a time of the same username then how to handle this

[@Short-Story-Talk]

Race conditions can be solved by locking

[@webster.]

You will have to use mod rather division

[@sukiratwarring6219]

Instead of all this just do write the unique attribute on the particualr field in the schema.

[@sandeepdev0]

Using a unique attribute on a schema field is the simplest way to enforce uniqueness, as the database will prevent duplicates directly. However, Bloom filters are useful in high-traffic applications to reduce database load and latency by quickly filtering out obvious non-existent items in memory, making them efficient for large-scale systems.

[@dev_guidance]

please make a video in type script.

[@sandeepdev0]

sure✅

[@AjayCoding]

🙌

[@lamabinod]

bro where's the billion records? how do you feed billion records in your bloom filter? this supposed to be your content that match title :(.

[@sandeepdev0]

Bro, I insist you to watch the complete video carefully, because half knowledge is very dangerous, In video I have told that this is the simple implementation of bloom filter, but in order to make it work for billions of records, we have to implement it an advance manner, like using Redis's Bitmap, + using modified versions of bloom filter algorithm (Will be covering in upcoming videos) + a proper fault tolerance for this as well... don't judge a video by watching it half

[@RamanPratapSingh-lr4pn]

wouldn't the indexing technique optimize it ?

[@anuj7286]

Yes same question

[@gamingorbesport4920]

Billions of records buddy*

[@ImKaustub]

There could be multiple dbs

[@zdrive8692]

No indexing will only work for millions only and that is slow as well

[@sunilk9760]

Partition may help

[@rayhanislam7518]

please make video on redis bitmap and how to determine bit array size

[@sandeepdev0]

Noted✅