MuleSoft | Design Secrets of Resilient REST API for High Volume !!!

[@hyeranbrummett7815]

Thank you for this end to end implementation based on very thoughtful design principals.

[@baidhyanathnayak192]

Hi Siva. Thanks for the great video . Very informative and knwoledgeable video . One question : Why you have chosen Database . We can achieve it by MQ also ( vm, JMS or anytpoint MQ) . Using MQ has certain advantages over database . 1- It will be fast and more reliable ( since there will be no dependency on network and network latency time is removed ) ,2- No more addition of maintenance of database which I think is an overhead . Some more pointers 1- If system API is made synchronous( reply - request ) then it will be known upfront that whether it got failed or not .We can have a retry mechanism and in case of failure after retry exhaust we can have exception handling that it will be moved to a different queue and another flow will retry sending after certain intervals say 2- 3 days. IF Order fulfillment is a asynchronous process and if they want the message after 2 days , I will say then the order fulfillment system should have a mechanism to hold back the received message . Storing all messages is an overhead and should not be suggested if there is no other requirement . The payload must be that not simple as in demo and also volume of requests makes the database usage heavy and thus maintainability is a question . Anywwhere these are some of the suggestions and I will be happy to know what you feel about my suggestions for a better design . Please let me know .

[@sivathankamanee-channel]

Hi Baidhya - The MQ and Database serves 2 completely different purposes. The MQ is to buffer the incoming input before processing, so that it initiates the order processing asynchronously. However, the DB is to log the incoming payload forever for future reference about the Date/time of request, status of order creation in the end system and will have the payload in one of the columns as CLOB. For your next question - Storing all orders forever, is the priority #1 task in the real eCommerce world for 100 different business purposes. 1 - To keep track of order submission status, 2 - Keep the original request payload for reference for any errors. 3 - For Business Analytics purpose of the requested products and its demand, region of sale, product quantity, etc. The business should be happy if they get the numbers of orders so huge in a way the DB is not sufficient, which means your eCommerce is on its peak :)

[@rameshsura5432]

Hi Shiva, You are doing so great. But,can you please explain in detail manner...like drag and drop the components.i.e very useful for everyone.

[@sivathankamanee-channel]

Sure Ramesh. I initially did that way. But I thought it would be boring and slow. Do you like elaborated way of demo?

[@shakjosh4687]

Thanks very informative and knowledgeable video .

[@narasimharaogorige3283]

Thanks for the wonderful video.

[@saurabhmalpani7095]

Hi shiva , thanks for wonderful videos. I have a question what if MQ goes down? Even after configuring reconnection strategies, if it fails to come up , how this scenario should be handled? Please reply

[@nagarjunapamulapati346]

Hi Siva when we need to use concurrency ? let me know please.

[@sivathankamanee-channel]

Hi Nag - You need to use the Concurrent web service calls or concurrent processing when you have Synchronous API within which you need to call 2 completely different APIs or segments which are not dependent on each other. This is required to improve the performance. For example, 2 different WS calls take 10 seconds each, then the performance is increased by 100%. Hope this clarifies. Cheers !!

[@nagarjunapamulapati346]

Thanks siva

[@thedeveloper2513]

Hi Siva very well video with detailed explanation. Do you think it's efficient to use Bulk Insert instead of Insert for DB?

[@itlumeesantosh4453]

Siva Garu, I am also certified MCD Level 1 and followed all your videos. But I have one suggestion, Can you numbering the videos to follow the sequence? This would be helpful for new learners. and also can you do a videos for understanding end to end project. Because few new learners not able to understand how request comes from and response going back to end user. Thank you so much for all your efforts

[@rakesh20077044]

Could you please explain the advantages/disadvantages of API led Connectivity approach

[@sivathankamanee-channel]

Hi Rakesh - There is a video already, explaining this on a high level. Please refer my video below: kzbin.info/www/bejne/aIC5iZdtiK6oqZI Instead of naming 'API Led Connectivity', I have named it as "API based approach".

[@nagendrac5323]

Hi Shiva Thanks for the video, can we get the xml of this POC. Thanks

[@eswarkumar9382]

Hi Siva, I have a question.. For processing 1000 records in 1 minutes we need 2 workers and 1v core... That means 2 workers are allocated with two different 1v cores, so records will process in parallel in 2 workers to archive the same. am I correct? Please correct me if I am wrong...

[@sivathankamanee-channel]

Hi Eswar - Let me try to answer you one by one. This is a general guidance from MuleSoft to have 2 workers load balanced and each worker has 1 vCore processor or server which provides 1.5 GB memory including server and application data requirements. It is comfortably sufficient for processing 1000 records per minute on a peak period. We cant have 1000 records per minute 24x7. Hope this helps.

[@eswarkumar9382]

Thanks Shiva..

[@balajiyadav8605]

Sir are you giving personal training

[@nagaarjuna2336]

hi shiva, Thanks for your wonderful videos and concepts. Please share the source code. thanks

[@maheshsunkari1261]

thank you very much, sir will you please send this entire project code.

[@SenthilKumar-sn5yl]

Whether this Source Code is Availale in Gitbut? how does Co-relation ID value is unique?

[@sivathankamanee-channel]

Hi Senthil - Thanks for watching videos. Fantastic question and I liked this question and am really really happy. We need to create an unique correlation ID by using GUID technique. For this, you need to combine unique or nearly unique fields from the input request combine them to arrive at the hash value, preferably combined with the Date so that you arrive at the unique hash-value. Always remember, the date-time value is unique and never repeated again until the earth and sun exist. Please let me know if you need specific code segment for this, I can help. Cheers.

[@sivathankamanee-channel]

Do you need the code? If you do it yourself, you can simply follow the video and do which is more useful. Please try and else let me know, I will share the entire code. :)

[@harsha4557]

@@sivathankamanee-channel Siva Garu, is it possible if you can please share the xml for this POC. i tried using sample data, and failing while inserting into the DB

[@rajesh1377]

Could you please show code walkthrough step by step pls

[@SenthilKumar-sn5yl]

not able to understand worker and core? how to do setup this? is this Admin Level ?

[@sivathankamanee-channel]

Hi Senthil - Yes. In general, you can seek help from Network Admin who will work with MuleSoft to set it up. This is all automated script based. But we need to know them all to design the APIs effectively.

[@ramreddy9574]

Hi Siva I think source system is system API target system is experienced api

[@sivathankamanee-channel]

Hi Ram - The APIs that deal with end system will not have any processing technique or business logic in general. For example, in System-SalesForce-API you will have code only to UPSERT the customer order, the same is the case for sending order to Oracle system, etc. Hope this helps. :) The experience API deals with enhancing customer experience by transforming in the way or format they like.

[@ramreddy9574]

Thanks