How To Build A Complete Distributed File Storage In Golang

► Join my Discord community for free education 👉 / discord
► Exclusive Videos, Mini courses, Resume reviews, and coaching 👉 thetotalcoder.com
► 33% OFF on my Go + HTMX + Templ Course 👉bit.ly/3UFruxO
► Enjoy a 60% Black Friday Discount on My Golang Course 👉 fulltimegodev.com
► Follow me on Twitter 👉 / anthdm
► Follow me on GitHub 👉 github.com/anthdm
link to the GitHub REPO: github.com/anthdm/distributed...
SUBSCRIBE OR NO MARGARITAS
╔═╦╗╔╦╗╔═╦═╦╦╦╦╗╔═╗
║╚╣║║║╚╣╚╣╔╣╔╣║╚╣═╣
╠╗║╚╝║║╠╗║╚╣║║║║║═╣
╚═╩══╩═╩═╩═╩╝╚╩═╩═╝
Grab yourself a 33% OFF on the PRESALE event of my building production ready applications with GO + HTMX + Templ + Tailwindcss + JQuery course here: bit.ly/3UFruxO

[@anthonygg_]

► 33% OFF on my Go + HTMX + Templ Course PRESALE bit.ly/3UFruxO ► Join my Discord community for free education discord.com/invite/Ac7CWREe58 ► Exclusive Lessons, Mentorship, And Videos www.patreon.com/anthonygg_ ► 60% OFF on my Golang course fulltimegodev.com Thanks for watching

[@Support-Phalestine]

what the f bro damm project what's your networth you touch 1-million$ or not yet

[@abhisekde6841]

People building todo app demos and then this Gigachad dropped Hadoop HDFS demo. Respect.

[@3x10.8_ms]

i never watch full video of yours they are huge, i don't have long attention span. but i do like every video of yours because there aren't much go devs who create go content and i really like that you don't care about views and make quality content. kudos to you man

[@anthonygg_]

Thanks man!

[@dionverbeke1975]

@@anthonygg_ But your grandma does... Keep up the good work!

[@RustyClown-um3ek]

​@@anthonygg_ thanks for the course, very informative, might there be a GitHub link for it?

[@whiskers12901]

This is gigachad content. For someone like me who is trying to pick up patterns from experienced Go folks, thanks a lot for videos like this.

[@amlan7502]

Here are timestamps for all the different components - 03:23 🧰 Initial steps include setting up a makefile, initializing a Go module, and organizing project structure. - 05:16 📚 Detailed explanations are given about designing interfaces and structuring code for TCP transport handling. - 07:10 🛠 The presenter emphasizes good coding practices, such as documenting functions and organizing code logically for readability and maintainability. - 13:42 🏆 The tutorial promises to provide in-depth knowledge and skills to distinguish viewers as proficient engineers in Golang development. - 25:19 🤖 The presenter discusses the importance of defining a common interface for different transport types, ensuring flexibility and scalability in the system architecture. - 29:44 🚪 Implementation details include constructing TCP peers, distinguishing between inbound and outbound connections, and organizing code with clarity and documentation. - 30:12 🛠 When building from scratch, providing clear and verbose naming conventions can help avoid issues. - 31:05 🤔 Consider using interfaces for communication channels to maintain flexibility in the implementation. - 35:26 🤝 Handshakes in TCP connections are crucial for establishing trust between peers before further interaction. - 37:11 💡 Implementing handshake functionality through interfaces allows for easy adaptation and customization. - 40:37 🔧 Iterative development with continuous feedback is valuable for refining code and improving understanding. - 49:44 ⚠ Implementing error handling strategies like spam protection can enhance the robustness of peer-to-peer connections. - 55:08 🖥 Consider carefully orchestrating handshake and decoding processes in TCP connections to avoid conflicts. - 01:05:36 🛠 Messages represent arbitrary data sent over a network between two nodes. - 01:07:48 🧠 Creating a custom encoder may be necessary when dealing with non-standard message formats. - 01:11:07 🤔 Consider providing default implementations like "NotEncoder" and "NotDecoder" for users' testing purposes. - 01:15:01 💡 Using a generic message format allows for flexible communication, depending on the platform and payload. - 01:20:28 🔗 Continuing in the next episode, the focus will be on completing the distributed storage system and possibly exploring peer discovery within the library. - 01:39:38 🔍 Addressing errors in network connections is crucial for maintaining the stability of distributed systems. - 01:41:22 🛠 Handling errors gracefully in network operations is essential for robust system behavior. - 01:45:47 🗃 Exploring Content Addressable Storage (CAS) for file storage allows organizing and retrieving data based on its content. - 01:49:20 📁 Developing file storage functionality involves creating structures and methods for storing and retrieving data efficiently. - 01:54:30 🧪 Testing file storage functionalities is essential to ensure they work as expected and handle various scenarios effectively. - 01:58:17 🛠 Designing and testing custom transformation functions for file and folder naming improves organization and efficiency in file storage systems. - 02:11:54 🗃 Storing data in a distributed file system requires careful consideration of how to manage file paths and content. - 02:12:27 🔑 To facilitate file retrieval, it's essential to store the bytes of the content along with the key and hash the content for file naming. - 02:13:11 📄 Introducing a structured data type like a "pad key" can simplify file management by encapsulating path and original key information. - 02:16:56 🔒 Proper file handling includes closing files after use to prevent resource leaks and ensure efficient memory management. - 02:19:36 📝 Wrapping low-level file operations with higher-level functions can enhance code readability and usability. - 02:22:00 🧹 Implementing file deletion functionality is crucial for maintaining a clean and efficient file storage system. - 02:42:48 📁 When deleting files, ensure to remove associated directories to maintain consistency and avoid unexpected behavior. - 02:49:01 🚀 Comprehensive testing, including setup and teardown procedures, is vital for verifying the functionality and reliability of a file storage system. - 02:50:36 🐛 The code needs adjustments to handle default values properly, especially when dealing with interfaces. - 02:51:19 📁 Adding a root directory to the file system requires careful consideration, including default values and error handling. - 02:55:19 🧹 Testing helper functions like tear down and clear can streamline testing and ensure a clean environment for each test run. - 03:07:21 🛠 Regular maintenance tasks like adding new functions and optimizing existing ones contribute to code cleanliness and efficiency. - 03:08:14 🔧 Utilizing helper functions for testing, such as `newStore` and `tearDown`, can simplify test setup and teardown procedures. - 03:13:08 🧪 Testing multiple functionalities in a single test can help ensure comprehensive coverage while maintaining test clarity and efficiency. - 03:19:46 🧐 Ensure expected file names and paths match for proper functionality. - 03:20:49 🚀 Building basic functionalities like read, write, delete, and test for key presence is a solid start. - 03:21:32 💡 Refactor code for better readability and efficiency, such as improving error handling. - 03:22:15 🔒 Expose necessary functionalities as private functions for better code organization. - 03:23:08 📦 Start creating the server structure with required options like listening address and storage root. - 03:24:11 🛠 Configure server options and initialize necessary components like the store. - 03:26:54 🔄 Implement logic for integrating the server with peer-to-peer communication. - 03:28:43 📡 Establish TCP transport and handle listening and accepting connections. - 03:31:33 🤔 Consider encryption for files and plan implementation in upcoming iterations. - 03:43:17 ⚙ Design the server loop for handling incoming messages and user actions. - 03:49:13 🧹 Implement cleanup procedures like closing connections and handling user-initiated stops. - 03:53:54 🛠 Error handling is crucial when closing network connections to ensure proper cleanup and prevent runtime issues. - 03:54:38 🌐 Connecting and exchanging messages between servers is the next step, including peer discovery and gossiping. - 03:55:22 📡 Initial peer connections will be based on a predefined list of bootstrap nodes for simplicity. - 04:02:19 💡 Debugging errors, even if they seem trivial, is an essential part of development and learning. - 04:10:39 🤔 Reflecting on coding practices, like comment punctuation, can prompt interesting discussions but may not significantly impact functionality. - 04:11:09 😅 Embracing mistakes and learning from them is part of the development process, even if they're embarrassing.

[@dev-singh-kanyal]

Thanks in advance

[@gregorydaggett7444]

Any chance you could add timestamps, either in the description or add chapter breaks to the video? There might be good content people want to review that's a few hours in. Without timestamps, finding that info will be hard. Thanks for all the effort to make this!

[@AnthonyBullard]

I subscribed finally. First for you being an absolute lunatic to upload a 10 hour video, and second for you to make your Grandma proud.

[@AnthonyBullard]

Third for you making Anthony Go devs look like the Chads we are

[@akaladarshi]

I just did the same for the same reason.

[@gemsof279]

10hr golang project video is insane.....bro you are helping the community by a ton. keep it up

[@onestack-fr7ue]

I'm in 1 hour and already learned so much, this is damn huge man GG, thank you!!

[@RahulSharma-cc1fg]

that return nil part instead of return err was funny as hell but it pretty much summarises how programming is most of the time. glad that you kept it in the video though.

[@StolenXMR]

genuinely insane, this is the content that deserves to be payed for

[@dhadveyash2268]

i got exhausted making x-clone and felt like i aint gonna code for a while. i guess this is gonna make me get back to work. thanks anthony

[@raghuvamsi4879]

Enjoyed your project very much, some of the words you used are sooo funny🤣! Love you mate, this is a fire project on distributed systems.

[@anthonygg_]

Thanks man!

[@shomikhan1333]

Thanks a lot for making a tutorial on such a complex topic. Could you please make a video on high level architecture for a Distributed File Storage system? It would help me to understand what I was going to code.

[@matheuscoppi9406]

Thank you for your effort doing videos with high quality of content!! Keep it 🎉

[@redskater42]

This is the next best step for the channel. Anthony GG to the moon

[@ben.thornhill]

Thanks for putting this all in one video. This is huge!

[@leoxvic4701]

U are a godsend Anthony!! One day I'll come to Belgium just to meet u my man! I promise you!

[@clarencedu5410]

This is the most awesome go video I have ever seen in the past 2 years.

[@anthonygg_]

Thanks !

[@ashishchotani2458]

Timestamps : (will continue appending as I follow the vid) 35:00 Developing a TCP Connection via TCPTransport between local and remote nodes 56:51 Validating HandShake between local and remote nodes 1:00:53 Handling features of TCP Transport (ListenAddr, Decoder, Handshake) by creating a struct 1:10:28 Captured messages from the remote node in a buffer

[@mithilesh6793]

Hi Ashish, can you help me and provide some resouces about what this project does and how it works I am a beginner in Golang

[@ashishchotani2458]

@@mithilesh6793 to be fair, am also pretty much a beginner in Go, just learned the basic syntax via a net ninja playlist, I just felt that I should know how a DFS network is implemented and dived straight into it, hence it is taking me longer to complete the entire project. Rest for the workings if a Distributed File Storage/System you can look it up on KZbin, there's plenty of meaningful content here

[@nagy-nabil]

I don't know go, but I'm here for the vibe, great content man keep it up

[@stack.1]

Also at 4:49:00 using the peers as a multi io.Writer (since it embeds net.Con and thus fulfills io.Writer) is just chef kiss

[@user-hi3dj6wh7d]

my fear to migrate from typescript to Go is slowly fading away, and confident in giving interviews. will enroll for the subscription for your mentorship being in web3 development, i need to be ageek in go programming and your videos will help me reach my end goal

[@ShootingUtah]

Dude you put out LOADS of awesome information! I love it! Just found your channel.

[@mquanit]

woah that is some dope content man, this could literally land someone a high-paying job.

[@anthonygg_]

Spread the word my man. Preach the work of GG.

[@fabricioaraujo7642]

Anthony GG the GigaChad

[@gokalpmeric3611]

Best content I have ever seen in years. Great job! Thanks a lot!

[@anthonygg_]

Thank you

[@bumblebee9426]

Awsome content. Started watching 1hr a day. I am at 2 now. And learned so many cool ideas.

[@nicolaswolyniec1354]

Nice way to start the weekend! 🎉🎉🎉❤

[@akanimohosutuk928]

This tutorial arrived at the perfect time. Thanks AnthonyGG

[@TedoHamTole]

Thanks for this. I am already Go developer but I am going to watch this tutorial.

[@cariyaputta]

Absolutely amazing! I'll back to this after completed the free distributed system course.

[@shivamhw]

As long as you upload video, i still have hope that i can be good at go.

[@daniilkvartalnov8029]

4:10:22 Man's looking for a bug Man find the bug Man happy Me watching the other man found the bug Me happy as well

[@alikhansharapat9933]

oh man

[@stack.1]

Awesome brain exercise and exposure to socket programming

[@ibrahimmohammed3484]

This man's concept of a todo app tutorial is to build an OS hahah

[@phenvrc]

"What's the most difficult thing in programming?" Naming!!!!!!!

[@user-go8cj6ik3i]

I can not resist subscription any more, thanks for all your efforts.

[@babar-khan]

subscribed in the start buddy.

[@justadrop1317]

Anthony always make awesome projects. Thank you!

[@UzairAhmed-gn5su]

subscribing for your amazing Effort and for the promise you made with your GrandMaa.

[@anthonygg_]

I see her tomorrow

[@matthewaraujo7361]

Amazing video bro, i didnt even watch but i know will be very very very good

[@phenvrc]

Subscribed! Gonna ask questions as soon as I finish the video, prob on twitter

[@simo_the_goat]

wow, i really like your videos mr Anthony you never disappoint much love ❤❤

[@sevenreup]

His is the reason why I always have a makefile for my golang projects

[@madhavamelkote4554]

Saw the video, subscribed out of respect 🔥🔥🔥🔥

[@learntry9074]

you always pull rabbit out of any topic. my this weekend is booked now . thank you

[@xxXXuser69420XXxx]

this is madness, insanity. I LOVE IT

[@Mark2.1]

been waiting for this one

[@sakithb]

Keep up the quality content!

[@rajendrabhat8909]

!!! Absolutely amazing content....content normally you won't see anywhere else... It was fun and exciting to build this stuff with you.. I have just one req: - in intro, please can you show some figures that explains the end structure of the project or also may be directly show what the end result is gonna be also may be please increase font size a little bit... :):)🙂 Once again, great content..RESPECT

[@anthonygg_]

You are right, font size is very small. Will up that in next.

[@anthonygg_]

And thank you

[@hardikmaind9833]

thank you so much ...i got to learn so much from this

[@abdelilahou2822]

Lets gooooo, beet waiting for sooo long

[@MrBoBrilO]

Ez video to watch on saturday evening while drinking some beer

[@yaminokaze4475]

I know what I am going to do this weekend.

[@laffta6727]

ur a god send to many of us-thx

[@thecodingcult]

Good stuff brotha!

[@prashlovessamosa]

Thank you Mr Anthony very grateful to you.

[@khaledkhawajki3830]

Absolute Monster

[@pratikkulkarni891]

Hello, thank you for this. Learning Go and this is going to be really useful. Do you have a repository for this one?

[@anthonygg_]

Dropping repo in a bit

[@pratikkulkarni891]

@@anthonygg_ Thank you :)

[@brayancarter588]

​@@anthonygg_ hey Anthony! Any info on the repo? thanks in advance and amazing content as usual.

[@BensPineapplejuice]

I thought I would go outside today for the first time this week.... well screw that I'm doing this tutorial instead!

[@anthonygg_]

Go touch some grass first 🤣

[@muonx9649]

ANTHONY THE GOAT 🔥🔥🔥

[@explosivecl]

This is fire. 🔥

[@OscarRendon]

Hey grandma, blessings to your little one! 🫶

[@hueydo3522]

Is it really necessary to just type frantically the up and down arrow key? Or is that also industry practice?

[@curlybracketai5152]

Shout-out to the grandma

[@SumanPokhrel0]

Thank you for this.

[@304nokia]

Hello, could you mark up the timings on the video? It is very hard to watch such tutorials without them.

[@ankurgangwar3622]

Awesome projects, dude...

[@lakshaybansal4094]

Can somebody please tell me what are the prerequisites for watching this video, other than knowledge of go language?

[@ijazkhan3335]

Love this content. I want to buy that Toml course, but my card gets rejected every time I try for some reason, and PayPal is not operational in my country. Is there any other way? Also, I would love to see you build VPN server using golang.

[@YacineBKA]

Thank you man !!!!

[@faizannehal1]

I was planning to start learning Go for a long time and this video seems interesting to learn Go while building a project. Is this video good for getting started? I have experience in programming in other languages.

[@anthonygg_]

You can try the how to golang playlist first maybe

[@wew8820]

damn netflix just advertised a job for this then this dropped lmao

[@AlleinArk]

GOATTTTEEEEED CONTENT

[@Agility-sm1hz]

Awsome🥳keep going

[@KennethFeur]

1:43:00 this should be "if errors.Is(err, net.ErrClosed) {...}" Also `path.Join` is better for cross-platform path concatenation

[@ashersamuel958]

thanks Anthony

[@igortrifonov9339]

10h O_o. Thank you for the video, you are a monster!

[@liantianlaoli]

Can this programming process be done without always writing this function and then changing it for another method? Is this an instructional or a live recording?

[@liantianlaoli]

Don't change things around. Think about what you're writing ahead of time, okay? It's really hard to read.

[@ashutoshsingh285]

this video length == whole attention span I have for future till ii die 😂

[@johndoe9604]

You're a legend!

[@-mau]

There are two functions on the stdlib that might've been useful: file.Join to join the entire path and filepath.Dir to get the parent dir 😀

[@anthonygg_]

Yeah! 100% agree.

[@saifbenzamit5980]

True legend men keep up❤

[@HdHh-or2pu]

Before going to start i have one doubt this project that is storage is limited or unlimited file to store ?

[@anthonygg_]

As much as your hd can handle

[@kosnowman]

are there any reading materials on how to build it regardless of languages?

[@anthonygg_]

Well I coded this from my inner creativity

[@Manish-hv9qx]

i was just following your redis thingyyy, and you dropped this bomb in between

[@DeviantFox]

holy hell.. ok this is fantastic. Any chance we can get some timestamps/bookmarks?

[@zlw8844]

in real project, should engineer design the system first before jumping directly to implementation?

[@bjni]

lets goo (Golang)

[@gertrudessampaio8689]

That very good I will watch until the end and rewrite it Rust for fun 😊, but if i must say something is that expressiveness is something that Golang doesn’t care. 😣

[@miguelricardosobong4927]

Correct me if I'm wrong, but if it's distributed, are there any extra hardware requirements for this if you want to follow the tutorial?

[@anthonygg_]

No we run all nodes locally

[@anthonygg_]

You can run them on other machines though

[@miguelricardosobong4927]

@@anthonygg_ Nice, thanks!

[@omerayhan1704]

Omg! it's happening 🎉❤

[@bora514]

What are the prerequisites for this tutorial? 🙂 Is it enough to go first through your Golang course?

[@charleschen4093]

Lets go anthony

[@adityamehta2008]

Should i atleast know some basics of golang or I can jump in this project while learning the stuff ??

[@pserra94]

amazing!!!

[@kavindusuraweera5726]

This is grate 💯💯💯

[@MarkJay]

Subbed.

[@starlord7526]

what are we actually building? I want to see the output but I aint find any