Thank You! This helped a lot with writing 20mil records to solr.
@WebDevCody Жыл бұрын
Oh wow this actually helped? I’m glad 😂
@martinmingosuarez86902 жыл бұрын
there is a core module called readline that works with streams to read one line at a time, no need to write the logic yourself, otherwise great explanation and video. Cheers
@veremox Жыл бұрын
im not an expert but a chunk is not necessarily a line...
@oscarmamani5996 Жыл бұрын
martinmingosuarez8690 thank you for this comment! it help me
@LegendKopper10 ай бұрын
-----The topic "NodeJS streams----> (your head)
@pranjaldoorwar97432 жыл бұрын
This might be helpful to me i am trying to read csv data and insert it into mongodb first before parsing it
@ViktosB6 ай бұрын
Hi, I really enjoy your videos. One of the most productive publisher on KZbin. But in that video the name "overWatermark" is highly misleading. I struggled understand the whole structure only because of that naming. This would be better if you use instead "canWrite". canWrite is true when we can write and false when the buffer is full. Then we stop -> !canWrite. const canWrite = writeStream.write(`${i}, `); if (!canWrite) { await new Promise((resolve) => writeStream.once('drain', resolve)); } } I might be wrong of course. Other than that - great job again. Continue watching you content 🙂
@PhuTran-nb7me2 жыл бұрын
Hello Sir, I have a question, much appreciated If you can address my doubt. Do chunks come in sequence, in other words, does the read stream wait for the current chunk to finish the operations before it emit the "data" event to process the next chunk? You have a potentially long running iteration there and more importantly, each chunk is interdependent to each other as the last If statement is used to assign the last incomplete row to "unprocessed" variable for the next chunk to use.
@WebDevCody2 жыл бұрын
Yes they should come in order.
@empty8537empty2 жыл бұрын
It should be in order..which is taken care by your transport layer..
@officialabdulrehman Жыл бұрын
This was very helpful. Thank you
@Tony.Nguyen1372 ай бұрын
What if I want to read big data of users - User{age:20………height:7“} - and save it into a database, but that chunk of data miss some properties that a user have like age:20“ ? Could this happen if yes how to solve it
@vpetryniakАй бұрын
How to use Web stream API in node js to do the same thing?
@bipinshukla5222 Жыл бұрын
Awesome explaination
@CarlosHonorioBriones2 жыл бұрын
nice! This is good to know. Thanks!
@lucanthony22092 жыл бұрын
Hello @vJunkie, Thanks for the great content. CAn you make a video using csv-parser & fs libraries to read & write csv files ?
@LarsRyeJeppesen Жыл бұрын
Super stuff, thanks. Subbed
@rorycawley2 жыл бұрын
Great video, thanks
@ПетърТодоров-о7ф Жыл бұрын
thank u, buddy!
@YOUdudex Жыл бұрын
Cool video, thanks subscribed
@mtuchidev Жыл бұрын
How do you pass a chunk size ?
@SeibertSwirl2 жыл бұрын
Good job babe!!!
@haohong5141 Жыл бұрын
node version?
@himaliamit86982 жыл бұрын
if you show us the format of the csv file it would have made much more sense.
@majidshah482 жыл бұрын
i think opening the file will take time or it might just crash
@boople2snoot2 жыл бұрын
them jump cuts
@boople2snoot2 жыл бұрын
also no sub, your variable is overWatermark, you're waiting for an event emitter if not overWatermark, the function therefore must return true if it is not over the watermark and var should be named underWatermark. Or it would run if not overWatermark, which is under the watermark, which? shit you just ramble, bye.