How To Stream Large Files Over TCP In Golang

[@anthonygg_]

► Join my Discord community for free education 👉 discord.com/invite/bDy8t4b3Rz ► Become a Patreon for exclusive tutorials👉 www.patreon.com/anthonygg_ ► Buy me a coffee 👉 donate.stripe.com/aEU2a6ayH2uCa3u4gg Thanks for watching

[@insanity0312]

"just use little-endian and drink margaritas at the beach" "not too big because my pc is pepega" these videos are just perfect, perfect pace, perfect content, perfect tone, perfect accent

[@ArtemTimofeev-iw5cl]

man... i cant express how much this will do for my cdn in golang... Dear Anthony, bless you😃

[@anthonygg_]

Your welcome my man!

[@fredericcordier1519]

this video is really great and very educational with the different steps/tests. Thanks for your work, the days are too short to watch all your videos :)

[@anthonygg_]

Thank you!

[@otaxhu]

Your videos are very helpful, no doubt this is the best golang content creator

[@Nexjsdeveloper]

i need to see ur video over and over! Perfectly fine.

[@jatindersinghaujla]

Before watching this video I just listen what streaming is but today I have seen how actually it works. This streaming concept clear but buffer is and how stream it. Great we are expecting more exciting and interesting videos.

[@hebozhe]

Go beast! I needed this locally for a 20GB corpus, so thanks for the tips.

[@ДеянДелчев-ы9з]

Great video. I saw a few problems with the final solution. In the sendFile you append the size of file to the conn. In readLoop you read that size in the loop which is problem. And even if the size is in out of the loop, this will try to copy the entire byte data from the conn into buf in one go.

[@Alibek11221]

Like your work! Thank you! Learned a lot through your videos

[@shoebilyas9664]

your channel is a rare gem

[@anthonygg_]

Ty

[@bjugdbjk]

Awesome , Thnx for the amazing Go content.

[@Andika-hk6rd]

I'm new to programming and I think this is cool, gonna save the video, I think it's useful :)

[@BeLazyDev]

thank you for uploading great video!

[@deNudge]

Why are you reading the file size within the for loop?

[@L0wPressure]

Yes, it leads to neverending stream of random bytes. Get it above the for loop and it will do what's intended.

[@SaurovChandraBiswas]

Nice and Helpful.

[@9scott291]

Thank you for sharing

[@tonycheng1614]

I am very new in go. Can you specify the advantages of streaming file data comparing non-streaming file data in transfer? Thank you sir.

[@anthonygg_]

Non streaming is saving the complete file in memory. Hence 30 gb files will not work on a simple server

[@MarinescuEdward]

@@anthonygg_ even with the second appraoch if you send 30 gb file it will still allocate 30 gb of memory `binary.Read(con, binary.LittleEndian, &size)` since size is 30 gb

[@jondoe79]

Anthony is a Go Master ultimatum

[@anthonygg_]

The undisputed.

[@Chxrli3]

Thank you so much for this video :) its really helpful. Do you have any links or videos about parsing from the loop?

[@sovrinfo]

Awesome video. Big thanks. I have a question, if I set size: 400000 program exit with code 1. Error message: recieves 400000 bytes over the network 2022/12/16 20:50:03 EOF Process finished with the exit code 1

[@anthonygg_]

Could be a read or write timeout. You can configure that

[@sovrinfo]

@@anthonygg_ Thanks

[@lowe7372]

You are the best!

[@oscarllamas]

Great as always

[@vaibhavmewada4508]

nice video anthony

[@ruslangabitov5202]

Thank you for your video. How will this transmission work if the sender don't know the file size? Is it possible just send the EOF in the end of transmission? Does conn send EOF when the connection closed?

[@wMwPlay]

Can I download twitch stream and stream it to youtube upload API directly, using this method?

[@anthonygg_]

That will involve some more work my man

[@oleks19999]

I just don't get one thing, what condition will stop the for loop ? Cause we want to stop it at some point right ?

[@ollegio]

But your first implementation is also a streaming one: you are not reading whole file into memory, you have a fixed buffer which is updated on each Read. I think io.Copy just does the same thing

[@anthonygg_]

I think io copy is bit more effiecient under the hood.

[@simo_the_goat]

MR Anthony I really appreciate your content. Please I have a small question I want to know why I opened my activity monitor and my RAM is getting too large even though It's streaming

[@tanko.reactions176]

brother, this is the second time i see you sending stuff over the network without taking into account that the operating system can not always send N-bytes just becuase you wish to send N-bytes. it may send N/2.. or much less. if you wish to stream, you have to keep track of a pointer to how much has been send. if N/2 has been send, then you send from N/2 until N.. you can not declare the chunk to be sent, if only a piece of it may be sent. the issue does not occur locally, it occurs when you do this over the net!

[@anthonygg_]

What are you talking about my man. Read net.Conn.

[@dazealex]

I have no idea what the dude is saying either. RTFM @@tanko.reactions17

[@madhudson1]

the for loop on the read with the end solution is a bit redundant i think. Wouldn't this solution just endlessly (for huge files) read into memory anyway? the Buffer will just fill up continuously, instead of allowing you to process x number of bytes?

[@anthonygg_]

I need to check this, you could be right

[@headlikeahole6548]

How does the method "binary.Read" know where the size data is and where the file data is?

[@twitchizle]

Server sends, filesize. Binary reads filesize. Server sends filedata. Binary read file data.

[@fullstack_journey]

Time to build my own OTT platform startup

[@ghostbuster5266]

You need read file size before loop 😅

[@adamsribz]

Would be amazing if you could share github snippets with your videos

[@anthonygg_]

Im getting paid way to less for that. This is free content made in my free time. Leeched by many. Returned by a few.

[@adamsribz]

@@anthonygg_ I feel you! Your hard work shows even in your free stuff.

[@MaybeAwab]

the best

[@aianshu]

Awesome

[@mjhsaja]

rand.Reader is missing