How to design API that display and filter through millions of products.REST API Filters & Pagination

[@vengateshan7800]

This video was very helpful and got more info . Thanks

[@karfin.trader]

Some information on how the backend queries will manage limit and offset would have been helpful to tie in the solution end to end.

[@anothermouth7077]

Same, at moment I'm making query string and based on given filters I conditionally appending where clauses. Could you let me know your approach?

[@johnrodger5467]

Can validate the request param in the api using a validation library, to only accept values in the allowed range. If the request is outside the range you can either return a response to the client telling them it's an invalid request, or send the request to the DB but with the max range on the query. Something like that. You could also write your own DB query wrappers/facades that have the limit already set there, and only use those in the application code rather than using the DB client directly.

[@Shreenidhi110]

thanks alot!I was struggling to anwer these questions in interview .I must say it's a easy way to understand these concepts.

[@sidhanshuraghuvanshi1]

Expecting implementations of concepts as well in all videos

[@UtkarshSinghPaliwal]

Please also make a few videos on Low Level Design and CDNs.

[@rjcdz06]

Great and clear explanation 👏👏 Thanks a lot

[@hakieye5649]

Wow terrific to the point with clear and clean message.

[@buckybarnes9135]

by your presentation we easily got the concepts, thanks to your editor😅😅

[@halconElectrico]

Your explainations are the best, right to the point, well illustrated, real world examples. Thank you.

[@UtkarshSinghPaliwal]

Thanks a lot for these wonderful videos. I've become a better Engineer with the help of your videos. Keep up the awesome work 👏

[@rahulsen002]

Found 2 channels useful so far .. Ur channel and another.. kudos

[@aatiqaghazali7850]

Plz mention the other one

[@rahulsen002]

@@aatiqaghazali7850 Rahul nath

[@LoayAl-Said-j8p]

Thank you so much for the effort. appreciate it Loay from Egypt

[@WOYGx1010]

You did a great job with this video, thank you

[@dragonlordsaviour7005]

you're super underrated

[@nocaoconstantino3555]

Great. Wonderful explanation.......

[@utkarshconscious]

This is really wonderful explanation. I know its not in scope of this series but out of curiosity would you mind throwing some light on how backing code for pagination can be implemented? Will it use some sort of sorted cache from where it returns results based on offset and max limit or is there a better way to get this done?

[@haseebalamrafiq5894]

mam you explain awesome. Thanks

[@m13m]

In backend for offset and limit pagination we can use db limit and offset parameter while creating a query to fetch that many rows and this quite inefficient in certain scenarios. Keyset pagination which is quite efficient but can’t directly query a particular page. Either go forward or backward in rows with a particular limit.

[@saurabhchopade3891]

Dope quality Explanation.

[@maniac1441]

greatly explained

[@nihal6290]

Love this video. Can you please create a video dedicated to API Security? How do you secure an API?

[@keshavsethi1610]

just one word Awesome.

[@jinalpatel9154]

This is nice topic. Sometime I feel that pure REST approach is not suitable for business need. It mainly revolve around single entity but business application has complex need like to filter. Let's say for example Employee Belong to specific company and organization and there are multiple attribute associated with that and we have to pass multiple filter attribute. I feel that sometime pragmatic approach will make more sense and expose end-point like /search / filters etc will make more sense.

[@saurabhchopade3891]

Thanks for posting this one

[@sergiothinks2]

amazing video thank you!

[@sudocode]

My pleasure!

[@kw5218]

Explaination is excellent please provide practical video if possible

[@shashankshekhar2736]

Amazing explanation

[@iyakkaprivatelimited5735]

This explanation is so valuable thank you for that , also I would like to suggest if possible to do a video demo with a limited paging server response for example : API like Zoho, or deluge code based API's , how to use the explained methods on a study case like this

[@naveensharma9834]

Clear and crisp explanation but I've one question. Do we need to hit db for every request, Is it a good idea ? If not then what are the other ways we fetch x amount to data in once(certain threshold) and serve in 20-20 chunks ?

[@m13m]

Definitely we could fetch a threshold number of rows and warumup your cache and grab any new data only through but offset-limit pagination is quite inefficient imagine there is no cache is such case every single query of limit and offset you are hitting db grabbing all the rows and then applying limit and offset on them. Also if you are inserting new items as well there can be cases where you could have duplicate items among your list with offset-page pagination Other technique that could be useful keyset pagination where you can’t actually pass offset but can behaviour of pagination with previous and next page with a certain limit just that can’t jump to a particular page but definitely traverse them limit you have set at a time.

[@erased1972]

Very helpful video

[@keaganferrao]

Using an OFFSET for huge data will still be inefficient for MILLIONS of rows as the database still needs to skip all the rows specified by offset. I expected you to give a solution to this problem in this video as the title specified "millions".

[@luxurymusic1051]

hey i was asked a question where i told about offset for pagination, but interviewer told the data is added on api all the time, so when i call the next page , i might get the same data in the next offset as well coz new data was appended by time i clicked on next page, so how to handle such cases???

[@II-ii2um]

simple function that checks if the previous data shown != the new data being shown this time? Source: Copy-Paste Buffers in Systems use this all the time to prevent the same shit from being copied and pasted over and over again... Have you figured out a better answer for this yet?

[@akashrulz30]

Thank you so much 🔥

[@gurusamy4523]

Hi teacher thanks❤

[@joe0212]

Nice. Any video on cursor vs offset based pagination

[@rahul2142]

Mam have you created video on implementation?

[@abhijitdixit1589]

One more way to reduce payload is sending it back as a stream, can you share use case for the same? Could it be video/audio streaming?

[@SangeeP]

Could you show a sample pagination and filtering code for an e commerce site for more clarity .. Wondering how the offset limit works when the reasons body has 1000 + record s in that case for one request pay load how you return so many by setting the server side max limit 20 ?? TIA

[@apothike]

On the server in every request we will be fetching all the data(from database) then paginating it right, assuming no caching is done??

[@UtkarshSinghPaliwal]

No. We can write the DB query which take the limit and offset numbers as arguments and fetch only the required data from the Database.

[@shailjakantupadhyay5183]

We never do that, consider server contains trillion data in table if you fetch all the data then it will go down. Read @Utkarsh comment.

[@AbhishekJain118]

You did not discuss this scenario: if client request for 100 items and server max-limit is 20, how client will be able to request items 20 to 39 in the next request? How client will set its offset ? I am assuming client logic should have offset=lastoffset+numberOfItemsReceived

[@sudocode]

That’s implied my friend.

[@unmeshvpatil]

Does api works to show products which are on low prices due to price error listed on amazon or flipkart?

[@javaeeguy]

Can you please cover REST API security also?

[@gurucharans9629]

Mam'm how could I understand my frontend before making backend server. If I created a registration page and login page are on different pages then it's important to make the same thing on the backed side. So, I will have to login and registration page (register, update, compare password pages) are on the different pages or same page in backed side.

[@stephenselvaraj8960]

Super :)

[@chollangimanisha3148]

Hi,your explanation is good. I have issue with pagination. I using one azure api to get all items. In azure portal, 500 items are there but I am getting 200 items while querying api. I applied limit and offset as well but not working. Please help.

[@cut-a-lyst]

can we send a json data in filter if filtering data is complex? I currently POST and json body for it currently

[@UtkarshSinghPaliwal]

Generally, there sbould not be a payload associated with a GET request. For a POST request, filtration might not be something that is required in the usual scenarios.

[@kunalshah2331]

What is there is no pagination.. the data is loaded based on the user scrolling the screen..

[@kapilrules]

Hi Yogita In case of pagination are all responses for all pages cached at server side at once? And then they are dispatched to client page by page as per the request? If they are cached on Server side when will the cache cleared? Thank you for all your videos 🙏

[@_tachyons]

This is unlikely, it is mostly implemented by directly executing query on database, caching layer is added to improve performance, but caching entire thing in the cache defeats the purpose of caching

[@NamathuUla]

Thank you for your video but there is some limitation in LIMIT and OFFSET. it will be huge cost involved for RU/s

[@adeshgangwar3555]

Hi I have a quick question. So if i have 50 orders fetched from api and are available in dom. Now i want to implement search the order by id. In this case shall I do a internal filter in the dataset or shall I fetched filtered data from api? What would be the best approach to do it?

[@AjaSiva]

Fetch it from the api. You may not know what ids are there in the current data you fetched. So it can give no result too if searched by id. Also id's are indexed in databases. So fetch call is not that costly.

[@praveenj3112]

It's clear ..but there is more link to fetch more data in some pagination APIs

[@typg93]

how does this compare to graphQL?

[@ChandanKumar-ou9fr]

Designing the APIs are fine, but how to implement such things? What is the right approach. Let's assume that I am building an ecommerce site and where from the database I want top 20 filtered items, how to do that in code? Thanks for the awesome content. PS: All theory and no code, makes jhony a dull boy 😉

[@DeepakJangra-up8ek]

Critical: what if you request, say first 10 (sorted by latest) orders and then 1 new order received. Now if you request next page then 1 order (1st of second page) will be repeated because thats 11th order now (not 10th anymore). How can we fix that.

[@_tachyons]

This is not necessarily a technology issue, this could be an expected behavior as well. But if you want more predictable behavior use cursor based pagination

[@pranavbhat29]

Is there any specific form of response code sent by the server when the limit set by the client in the query param is larger than the max limit set in the server. I was thinking if it will send either of 400 ( Bad request ) or 413 ( payload too large ), but again you get a response with valid set of items in the server. So it seems to use some specific 2XX code itself. Could you please suggest or throw some pointer on what that might be?

[@UtkarshSinghPaliwal]

Probably a 422 would do.

[@AjaSiva]

You can throw any status code😅

[@sujan_kumar_mitra]

Is using OFFSET a good idea to skip resources? I have heard that using OFFSET in SQL can reduce the performance of DB as if there are a large no. of records, the DB has to pull the tuples from the disk but not include them in the RESULT SET

[@yashkhandelwal9926]

Are there any issues if we use post instead of get in this and send all the required data in requestBody rather than query params?

[@_tachyons]

That will be violation of rest conventions, but if the params is too big, then you may have to use POST

[@UtkarshSinghPaliwal]

Bad Practice

[@sumanthkeith1]

I have few questions on pagination. Can you explain about the page count. How to determine how many number of pages for each query. Say the query returned 90 rows, and if the results per page is 10. I need to show 9 pages (1 2 3 4 .. 9) 2. See this scenario. I have a query which returns 100 records and on top of that I need to apply some business logic and filter some data, so after filter I got 70 records. Now how pagination work when user click on page 2. (Point to be noted there might be thousands os records returned at a time for a query).

[@UtkarshSinghPaliwal]

Pagination should be applied on the final records being exposed to the clients. In the case where only 70 records are accessible by the client, pagination should create only 7 pages in totality. Moreover, in these cases, the client should never know that there are actually 100 records on top of which a biz. logic filters them down to 70. Otherwise, this would be an abstraction leak.

[@sumanthkeith1]

@@UtkarshSinghPaliwal yup I agree. But my question is when I click on page two, should query the db get 100 and apply business logic again to filter and show the contents of next set of records?? If we do this there will be performance impact. One solution is to store the filtered logic and Show the 2 nd page and so on. Should we use Redis or java cache us my next question.

[@UtkarshSinghPaliwal]

@@sumanthkeith1 That's a pretty decent problem statement. We need to paginate and we don't have the records to send back without applying some business logic. In this case I'd say to choose either of two things, may be not use pagination at all or cache the indexes of the records and some meta if required. I'm presuming that querying data more than once is very expensive so that logic is not an option. Now with both the cases suggested, two problems arise, one with both the paths. If no pagination used, data may not even fit in a single response in which case you may provide a download link instead. In the other case of caching, you'll have to deal with all the caching related issues like data consistency etc. So you'll have to choose either of the two evils depending on your particular costs and usage.

[@gouravkumarshaw5467]

Thanks !!

[@itsme_ry]

Nic mam

[@AnikashChakraborty]

How do I get to know till what number should I make the pagination buttons? Because if I don't specify it then the next button will return an empty page sometime in other words the grt request will return an empty array

[@goikarBaab]

When you receive paginated data, the resultset should also set total records , what is the pagesize. Once you have these two data point, you know which will be the last page

[@kavinkumar6080]

how the client side decide total page count

[@umangbehl8152]

nice

[@venkatshk]

please cover sorting also

[@pradeepcreations8669]

will this work for mongodb?

[@andiamakaza24]

U looks like NAIROBI of money heist 😀

[@kumaramresh7905]

actually i learned these terms pagination, limit, offset, filters while learning DBMS and i think they're more related to that that rest Api

[@subee128]

Thanks

[@getthenetfishing7130]

Do you offer a service to integrate an API from my distributor to my website for orders and dropshipping

[@mythiyen1386]

CDN can you share

[@hammerod]

Thank u

[@jagansiram2529]

Didn't work.. getting complete result

[@mainuddinali9561]

in this video u r looking lean

[@rajeshpatil3690]

Nice explanation but only for a very primitive audience to understand. It does not contain any technical details on how it works or implemented . If you could add more details on server side services to map to your URI examples that would be helpful.

[@sudocode]

Sure, you can create a video on that and I will be happy to endorse it 😅

[@vidhanchandra3997]

💕

[@ilakkiya.s1142]

😊

[@raamch]

Wonderful explanation