Spring Boot - Multithreading | Process Millions of Records in Batches | Hands-on Example

[@sahukarinaveenkumar3188]

We all can use this logic in realtime. Great Explanation. In most of the interviews these daya people are aksing scenario bases questions on Completedfuture. I think its a good example

[@R0hanThakur]

Nice video and clear keywords and explanation, this is also a popular system design question ....

[@Akash-tq1ui]

You are my teacher in springBoot microservices 😊 👩‍🏫 📚

[@MohamedKaanoun]

He is our teacher Lot of Love ♥

[@gopishettymahindra2713]

Thank you so much sir. I like it solving real time use cases.

[@devkratos711]

Great Explanation👌🏻

[@sujitkumar2196]

Awesome tutorial bro, thanks providing such type of video really it's helpful😀

[@yogeshchaudhary621]

Nice video! I like such kind of real problem applications.

[@dailydoseoftrendings]

Good one, Basant. Way to GO !!

[@vinodhreddy6600]

Thanks for this batch processing video.

[@iamranjandas]

Thank you sir for creating this type of video with real time challenges. Can you share your knowledge on how to take decision for how many size to be fixed for the thread execution using this concept. what is the best choice for this even considering threads of another methods of same project using completable future?

[@Rajivsingh-xb1nw]

Thanks for great knowledge.

[@AsadOnline-sz3os]

Good approach . Java introduces a new way to implement fork join that's completble future

[@aadithyagopalan2684]

Thanks. Nice one. Even my experience is that using custom executor service gives better performance when configured correctly. (Using the cores)

[@hanumanthram6754]

Thank you for your hardwork ❤

[@rishiraj2548]

🙏🏻🙂👍🏻

[@deltamakuba]

Amazing content

[@arnabchowdhury4892]

@JavaTechiice Please use this logic for creating batchsize for (int i = 0; i < size; i += batchSize) { batches.add(list.subList(i, Math.min(i + batchSize, size))); } return batches;

[@sushant6815]

Waiting for such videos

[@adarshverma3372]

which theme are you using?

[@sonicxgaming5533]

Can we use @async ? Over the method instead of completable future?

[@henrivalencia21]

thank you ser

[@deepakthankachan8443]

Won't the repository be shared across the threads that might lead to session issues? Also why do we not need the @transactional annotation in the second solution? thanks

[@pranithad1462]

can you please make a video on report generations using spring batch and oracle database?

[@hemangdave1038]

Completable future vs virtual thread what is the preferred approach....??

[@grrlgd3835]

Thanks for this JT. I'm a bit confused how this overlaps with Spring Batch ?

[@Javatechie]

Yes, there isn’t much difference. You could use Spring Batch for this as well, and I’ve already covered it in my Spring Batch playlist. However, for this use case, Spring Batch isn’t the right choice since we’re not migrating data from a source to a destination. Instead, we’re performing operations on existing entities and publishing Kafka events.

[@erbayaskn271]

Thanks for the example, but if you increased the default thread pool settings in the second method, it would give the same result as the third method. ForkJoinPool customThreadPool = new ForkJoinPool(20);

[@Saravanan-lj9so]

Could you please Publish video for Rate limit using spring cloud gateway +RedisCache+Custom Ratelimit filter with , burstCapacity, replenishRate, requestedTokens

[@vinoth2819]

How about using parallel stream for each batch. Why was it changed. Could it process it much faster?

[@Javatechie]

Yes we can try that i believe

[@azharmobeen]

I have learned a lot from you, but this solution has several issues: 1. There’s no transaction management-what happens if some records are stored successfully while others are not? 2. There’s no tracking mechanism to identify which records succeeded and which ones failed. 3. There’s no retry mechanism in place to handle failures. ...and the list goes on.

[@Javatechie]

I completely agree with you buddy . This is just a simple demo to show how can we perform batch processing using thread

[@premierde]

👏 SpingBatch does the same thing. Can you please incorporate/integrate this in your SpringBatch and updtate there. I implemented via SpringBatch in our project.

[@sumitsaurav4033]

db throughput is comparatively low if we save lakhs of record will it affect our db, how can we optimise it using kafka

[@Farm2Code]

Hi sir or brother, will you please let us know the job market in November and December. I was searching jobs I used to see most are asking for immediate joiners so i thought like putting resignation letter and search and it's 3 weeks since I applied resignation not even single call I'm receiving so please enlighten me , us .

[@sathishspeaks]

Need more videos on spring batch Asynchronous batch processing

[@Javatechie]

Satish It's already there please check spring batch Playlist

[@kaasiworld]

Can we use virtual thread to bypass the thread count declaration

[@Javatechie]

Yes we can use VT instead of PT

[@kaasiworld]

@@JavatechieHi, In one of my scenarios, we have a maximum list of a thousand.2 thousand records. I need to perform multiple operations on each record. Can I enable virtual threads and use parallel stream processing for this? In my case, do I really need to consider batch processing?

[@Javatechie]

Yes batch processing is optimize way to improve throughout in your case

[@kaasiworld]

​Thanks for the quick reply@@Javatechie I'm trying to understand the consequences of implementing virtual threads with parallel stream processing. Typically, the number of records will be below 100, but in corner cases, it could reach up to 2000.

[@maheshkumar-sl8eq]

Can we use spring batch sir for same

[@Javatechie]

No buddy spring batch used for different context where you have source and destination to migrate and transform the data

[@maheshkumar-sl8eq]

@ thank you sir

[@premierde]

​@@Javatechieyes but in each steps of (ETL) SpingBatch, it also need all this concept of throughput etc. Cant this be integrated in the steps? in better way

[@rabindradocument8934]

I think a single machine is processing all requests here. So we cant load all data into memory for batch processing. If we dont load all data then it will be latency issue.

[@Javatechie]

Single machine means Single vm right? And each vm have multiple cores so each core will initiate a new thread to execute your task . Why there will be latency issue? Can you please brief

[@rabindradocument8934]

@Javatechie a single machine won't scale considering cpu and memory limits being constraint. Here vertical scaling required with enough memory and cpu to process the requests in batch.

[@rabindradocument8934]

If I consider it is a 4 core machine then at any instance of time only 4 instructions will be running. So processing millions of records is possible but it will be slow with a single machine. Let say my machine performs good with 50 threads. But no guarantee that it will perform good with 5000 threads. Even cpu may not able to schedule those 5000 threads. Even if they got scheduled all of them may not get cpu time. Running more threads in a 4 core machine will bring lot of switching between threads.

[@rabindradocument8934]

Also as we are seeing about million records I feel we should delegate heavy processing logic to multiple machines instead doing in a single machine.

[@rabindradocument8934]

It is a nice example on batch but I have only concern of NFR and millions records. Anyway it's an informative video saying about batch processing.

[@sahukarinaveenkumar3188]

Just want to know in firm you work?

[@Javatechie]

I am a developer who loves to make videos and share knowledge with the community!

[@Mohammadsaied81]

why some contents for memebers ?

[@Javatechie]

Yes few content will be only available for members only those who joined my channel membership buddy.