Writing My Own Database From Scratch

First episode of the #mkown series.
In this video I write my own SQL database from scratch without using any libraries or third party dependencies. Only the standard library provided by the chosen programming language is allowed (Rust in this case). The end result is a basic ACID compliant database that can process only one transaction at a time, something similar to the first versions of SQLite from the early 2000s, but with less features (and probably much worse performance).
The goal of these videos is to provide a high level overview of how such complicated systems work internally, not explaining all the code in detail as that would require dozens of hours of content.
🌐 LINKS
Project Repository: github.com/antoniosarosi/mkdb
✉️ CONTACT INFO
Business Email: [e-mail]
Contact Email: [e-mail]
Twitter: / antoniosarosi
Instagram: / antoniosarosi
LinkedIn: / antoniosarosi
🎵 MUSIC
🕖 Time Lapses
/ temporal-t-lpa-once-up...
• Neovaii - The Maze
🔉 Background
/ intro-falling-out
/ silhouette
/ cerulean-skyway
/ lifted
• Crossroad
• [FREE] Chill Type Beat...
• Sine Ira
• ❰Chillstep❱ Sappheiros...
📖 CHAPTERS
00:00 What is Mkown?
00:37 What Do We Know About Databases?
02:18 Research Time Lapse
02:53 Initial Ideas
03:04 Naive Storage Format
05:31 Why Binary Trees Are Not Enough For Indexes
07:46 Why B-Trees Are Better Than Binary Trees
12:21 Software Architecture
13:17 Dev Time Lapse
14:13 Project Demo
18:23 Database Internals
19:24 File System Structure
20:31 Slotted Pages
23:45 Rows, Tables And Indexes
26:50 Overflow Pages
28:30 Sophisticated BTree Balancing Algorithm
29:53 Parsing and AST (Abstract Syntax Tree)
31:54 Query Planner
34:23 K-Way External Merge Sort
36:50 Cache Replacement Algorithm
38:19 Transactions: Commit & Rollback
39:27 Network Protocol
39:56 Final Thoughts
🏷️ HASHTAGS
#programming
#computerscience

[@BearIsAwesome97]

Finally some real software engineer who is making cool stuff. KZbin is flooded with too many leetcode tutorials.. finally someone is making cool stuff..

[@tony_saro]

Time for some real shit 😂😂

[@plaintext7288]

Tsoding must be mentioned here!

[@tony_saro]

​@@plaintext7288Judging by this tweet, I don't think he likes this kind of content 😂😂😂 x.com/tsoding/status/1802792064347668603?t=n07b-kEtMynM1VB8Ri2F-w&s=19

[@plaintext7288]

@@tony_saro 😆😆😆😆

[@plaintext7288]

@@tony_saro i meant as another good swe channel!!!

[@WarchantUA]

First video on a channel is a 7-month project - you're a legend. Please continue!

[@tony_saro]

Yeah, probably should have done something simpler 😂

[@damonguzman]

@@tony_saroNo way!! Low effort content is a dime a dozen. I just subscribed hoping for the next banger video.

[@churchers]

Brave. Databases are one of those areas of computer science that gifted experts spend their entire career on

[@tony_saro]

They're very hard, I don't have a PhD in databases or anything like that, I don't even know what I'm doing 😂

[@lyrebird712]

​@@tony_saro learning how it works is infinitely helpful though. I've never written a database before, but I do have real-world experience with quickly locating a CSV import issue because I had written my own CSV parser/writer/converter before. Even if you never use the database you've built, this knowledge is gold on its own.

[@jiauyjiauy3777]

@@tony_saro how would having a PhD be of help? My experience has been that degrees don't matter, especially with IT where basically all the information is online

[@tony_saro]

@@jiauyjiauy3777 Databases are an advanced CS topic that people have been researching for decades. Those who wrote the databases we use in production dedicated a large part of their career to just databases. This is not Fullstack web dev where you can just hop into a KZbin tutorial and learn most of what you need in one evening if you're good at coding. Sure you can find information online but it's mostly papers or university lectures. It's not necessarily the "degree" itself that is of help, it's the years of dedication to databases.

[@Omnicypher001]

@@tony_saro Game devs roll their own databases all the time. Its very simple compared to most of game dev. As long as you store persistent data and have a way to edit that data, you have made a data base. B trees and parsing query languages are optional, but many game devs use console commands to debug and edit data. Its not hard to make a database.

[@yashvardhan1158]

Finally found a real software engineer on KZbin who builds cool stuff.

[@lost_martian]

This was such a hidden gem.. lucky this came into my recommendation

[@tony_saro]

I'm glad you liked the video. More coming soon.

[@Karim-ln9dw]

It's been 4 weeks since you released your video. Man, it is unfair that you have only 1.8k views. If I stumbled upon this video, then you will find your viewers. Keep doing this!!

[@tony_saro]

That's how KZbin works at the beginning, it recommends videos slowly until it gathers enough data to determine whether the video is worth recommending to the masses. I have another channel with over 160K subs and that's what happened with that one. It's gonna be a hard journey, but I'm pretty sure it'll work out in the end. Anyway, I'm glad people like you are finding my videos. Stay tuned for more content like this!

[@OmbasaGeoffrey]

I got notification today

[@tony_saro]

​@@OmbasaGeoffrey Looks like I was right then 🤙

[@dan_le_brown]

It showed up in my recommendation today and I'm absolutely stunned 😳 Great video!

[@Laflamablanca969]

Let him cook bro, he will hit the algorithm and explode

[@yerpery]

Junior SWE here. This was such a fun watch. Your channel is severely underrated!

[@tony_saro]

Glad you liked the video, thanks for the comment.

[@chturne]

This is fantastic. Great production quality, no nonsense clear explanations. A rare gem to find in programmer KZbin. Congratulations on completing your project. I hope you'll continue to do it and be well rewarded for it!

[@melajod4wan135]

i've been wandering around youtube for a while but wait, ....how did i fall into this hidden treasure !?

[@tony_saro]

Thanks to the KZbin algorithm 📈🔥

[@mikejohneviota9293]

very rare to see a youtuber that use low level programming for content ❤🎉 im your new subscriber

[@tony_saro]

Thanks for the sub

[@agusaris5031]

Now this is real software engineering content that I'm looking for. Database engineers are on different level

[@_jfsanchez_]

It is so sad that the Spanish community mostly talks about HTML, CSS, JS and PHP. So happy to see your new channel and such a great content. Subscribed.

[@matwatgd624]

Im also from an spanish speaking country and every "programmer" is a fkin react dev, i just want to do my golang stuff and find resources

[@gemrwx]

@@matwatgd624 I don't blame them. For instance, in Latin America, focusing solely on learning fields other than web/mobile development, such as low-level programming or machine learning, is basically a death sentence (you aren't getting a job lol). Unless you're a genius, there are simply no opportunities available. Even in more advanced countries like the United States, such opportunities are limited and highly competitive. That's why it's wiser to start by mastering the most in-demand skills. Once you've established yourself in the job market, you can then consider transitioning to other fields if you wish.

[@iamwildeofficial]

That's a really interesting point. I have to say that chances of getting a job knowing common programming languages as you mention are more than knowing how to write a database engine from scratch. As Tony mentions, this video is basically reinventing the wheel, so I understand that other content might be useful for those who want to improve skills and get a job. On the other hand, having a KZbin channel explaining basic things will make you have more subs than explaining advanced stuff

[@pedromendozaaristegui5974]

This guy is native speaker spanish

[@flodderr]

random recommended video but it explains perfectly each concept and reasons behind decisions. Hidden gem!

[@tony_saro]

KZbin recommending some random stuff 🤙

[@flodderr]

@@tony_saro Im an engineer too and always wondered how databases are built so I happily clicked on it :D

[@BenMcHone]

This project seems like a huge undertaking! Kudos for following through on it. Thanks for sharing!!

[@johnyepthomi892]

This is straight forward and nothing gets in the way. Very engaging and clear. Appreciate the effort my friend. This will be valuable to a lot of people.

[@ivgadev]

Gracias Antonio! Estupenda iniciativa, algunos estábamos aburridos ya del nivel de habla hispana donde solo se habla de cosas muy básicas. Me quedo por aquí para seguirlo de cerca. Un abrazo.

[@tony_saro]

En la comunidad de habla hispana solo se habla de HTML, JS y PHP 😂

[@mikepro500]

Mas bien se debería traer este tipo de contenido al español para que el nivel de habla hispana deje de ser tan básico ¿No? 🤔

[@tony_saro]

El que vende se va donde más compradores hay 🤷🏼‍♂️, el problema del español no es que el nivel sea básico sino que esto no interesa.

[@mikepro500]

@@tony_saro hay poco interés por la misma razón de que hay poco contenido en español. 🤷🏻‍♂️

[@tony_saro]

@@mikepro500 Puede influir también pero yo no tengo la responsabilidad de hacer ese contenido, me imaginaba que me iban a llegar comentarios de este estilo, no me refiero al tuyo en concreto porque tú no me has mencionado a mí justamente pero para el que lo haga, lo único que puedo decir es que hablo en el idioma que me dé la gana. Aparte he hecho videos de este estilo en español también.

[@localhost0148]

Really well bade video and very educational. At no point did i felt bored nor did my attention went else where during the whole video. Loved it. You have just gained a new subscriber !

[@tony_saro]

That's incredible considering the video is 42 minutes 🫡

[@elipaperni3458]

This is the best video on the subject I saw probably ever, can't wait for more to come...

[@JuanGM]

Te sigo desde que empezaste la carrera, en el otro canal, but this is another level, ni si quiera me di cuenta que te lo habias creado. La explicación y el tema top, como siempre!

[@tony_saro]

No he querido darle publicidad hasta comprobar que el canal funciona por su cuenta. Pero viendo cómo está yendo pronto lo anunciaré por Instagram y Twitter. Gracias por el apoyo durante tanto tiempo 💪🫡

[@jmlopezponce]

@@tony_saro Tambien te sigo desde hace tiempo y se extrañaban tus videos en youtube. Cuando empece a verte justo empezaba la carrera de ciencia de la computacion y ya estoy al graduarme. Gracias por tu trabajo

[@dantech9924]

Toca aprender ingles porque estos videos son muy top, gracias Antonio

[@RaoulLabrunieEspinosa-qj6es]

i watched your videos of antonio sarosi and loved it but this, this is a work of art, love all you do

[@tonym5857]

Congrat bro, great project 👏👏👏👏 wow Rust, Query Plan, working with pages, transactions, MRU, etc 👌👌👌

[@oluseyeodujole4279]

I stumbled on this...watched first 5 mins and subscribed immediately! My first time seeing a programming content that resonates with me so much Great job! Keep it coming bro

[@amizednix]

Your work is awesomely excellent, and I could watch the full video without a single thought of boredom all the way down to the end while learning new things.

[@favoring-quiet-goatskin]

Wow, this is really something. You’re incredibly talented, Tony. Look forward to seeing what else you create.

[@tony_saro]

Thanks, I got some videos already planned, I'll work on them as soon as I can. Stay tuned 🤙

[@walterdiaz2003]

This is an excellent video and I'm really happy to have found it. Last year I wanted to implement something similar and I went through the same pages you went through when you were doing the research but I didn't go too far. Excellent job!

[@Weagle1337]

Happy to have you back Sir Tony Saro!, missed your content a lot 😉😉

[@Alex-kb2ws]

This is an amazingly edited and insightful video!

[@tony_saro]

Thanks!

[@slix88]

This content is gold! Gj man!

[@LenarImamutdinov]

This deserves much more visibility. Amazing work!

[@ragsbigfella]

Wow.. It blew my mind. Such complex topics explained so beautifully. Thanks a ton.

[@switchsides]

Extremely impressive, thank you for sharing the journey. Subscribed, look forward to more.

[@samals6454]

This is top tier stuff and exteremly helpful. Thanks Tony!

[@houssembousmaha3615]

Finally some high quality content, keep up bro

[@tony_saro]

Thanks, I will 👨‍💻🤙

[@protonetwork6974]

It showed up in my recommendation today when i try to make my own compiler 😄 Great video!

[@neutron_stz8894]

@@protonetwork6974 C compiler ? 😂

[@bravethomasyt]

Superb video. I use SQL every day, but I still learned a lot in this video, and will genuinely help me in my day-to-day. Also, you're super brave to take on making your own DBMS. Impressive!

[@mysticfit977]

Honestly, never commented on a vid before, but this is goated!!! make more of this, and make them as lengthy as possible, I will watch it all day.

[@tony_saro]

Thank you.

[@cajogos]

Great content, we need more of this!

[@NOCDIB]

Amazing! There are coding tutorial channels and programming concept channels but next to none do what you did, explain it so well and illustrate it so wonderfully. Subscribed.

[@shoaloak]

Wow man this is incredible. Excellent job!

[@nelsonchui0830]

❤It is really appreciated you said "I don't know ..." but you can finally find how to know them. 🎉

[@mar10reyes0]

I was also working on something like this. I decided to do it because of your Spanish videos. I may share it when I'm done. I'm jealous be cause I found your videos learning to program when I was finishing university and, at first, it was cute but now you are so much better than me is embarrassing. You deserve all the success you are going to get because of your discipline.

[@elcoco8]

This video is amazing, I love the format!

[@shrug5489]

These videos are incredibly thorough, simple, and helpful that actually taps into the minds of software engineers. Keep making these videos, would love to see this channel go to the moon!

[@AqoCyrale]

this is the best video I watched in a long time, your presentation and way of talking and explaining is top notch. will definitely watch the next thing you release as soon as you do. thanks a lot for taking the time to document your journey and share it with us!

[@supercoolezio]

I just hope you keep creating such videos. I've learned a lot in 40 minutes.

[@eladshamai]

Wow! What an amazing and detailed explanation 👏👏

[@jhonanthonyoblitasgavidia9603]

Wow the KZbin algorithm recommend you in my main activity, and I said I remember this guy, wow, you change your content to English and I understand very well 👏

[@theyadersh5263]

I come from the Instagram story what you've done about this channel and I'm excited to know that I can follow this kind of Programming Content

[@Rafael4]

Great content, would love to see future projects!

[@lemonadeintech]

Yo this was a treat to watch! Great vid 💯

[@roycechua]

I'm glad the KZbin algorithm recommended you in my feed. Great content especially since I don't see many programming content on KZbin that are this deep. Subbed

[@jeremylemans3005]

This is absolutely awesome! Thanks you for your efforts

[@Varunshrivastava007]

Great video brother... enjoyed every bit of it.

[@UzairHayat]

Subscribed! Looking forward to more content like this!

[@MauricioDavid77]

How fantastic to find someone who also enjoys creating their own database. Excellent project, congratulations. I did this same process of creating my own database called LiteDB (written in C#). Congratulations for the initiative!

[@AlmirBispo-CSV-Comp-DB]

Congratulations !Very good project !

[@MaskedEngineer-kj5kt]

Keep going Bro!! This is legendary

[@trendysupstar]

This video is so good that this is the second time I'm watching it in 3 days. Keep up with this good work.

[@alexwexler1257]

What a GREAT high quality video. You keep making videos, I watch simple as that!!!!! 🎉

[@Jasonlhy]

Very good idea, I often find looking at some low level detail actual help in doing high level staff in the long term

[@NguyenNam-yx5mw]

Haven't watch the full video yet, but this is so good, thank you for gathering and sharing this useful content

[@brunopm5138]

You’re just doing everything I wanted to do after university, re invent and understand software and tools that most of developers use all the time. Respect bro 🙌

[@rubensferreira7736]

Very good stuff man, Waiting for your next videos, keep up the good work. :)

[@chockman3833]

Happy too see KZbin is pushing this video a bit now. Incredible video !

[@felicytatomaszewska2934]

This makes me truly appreciate the teams that developed databases especially opensource ones like postgres. Though it is not a tutorial but more of a quick rundown and demo of the project, but still a lot of new concepts can be learnt.

[@caiolucas9136]

I'm very surprised that almost all of it I needed to do in my graduation! I'm a student from UFOP, Brazil and its delightful to see how my teachers were able to explain such concepts while we had to implement the code per ser

[@brdevll]

The moment I saw this video in my feed I knew it would be absolutely insane. I was not disappointed

[@ryantaylor6470]

Great video! Keep up the good work!

[@gyimihendrix]

I'm glad the algorithm helped me find this gold mine, super educational & inspiring, I might just try to build something of my own.

[@demian.succs0]

Hi tony, ive seen all of your spanish videos, ive learned a lot with you, i hope this new channel to become successful and keep the great work man!

[@andresaguilar3055]

Man this is pure gooold! I really was missing you in your spanish channel. But this is amazing! Thank you for this and for all the spanish content. I'm the kind of person that likes to learn the deep concepts of everything we use everyday and you have helped me learn about them. Muchas gracias!

[@tony_saro]

🫡🤙

[@leucthinh3226]

Cool stuff man! Keep going

[@MRECoelho]

Just came across your channel by accident and this seems amazing! In my opinion reinventing the wheel is always a good learning method to understand how 'we' got to the point we are now. Keep it up!

[@tony_saro]

Exactly

[@KryptoBrah]

You will blow up! You can explain very good. Just continue!

[@imcorfitz]

Amazing video! This is a very impressive watch. I freaking envy that commitment. Also - I can only imagine the level of complexity in implementing UPDATE commands with variable length data. 😳

[@tony_saro]

Here's the update command execution, judge by yourself 😂 github.com/antoniosarosi/mkdb/blob/bf1341bc4da70971fc6c340f3a5e9c6bbc55da37/src/vm/plan.rs#L874

[@raresgabriel8991]

Please keep producing quality content like this

[@zanfet]

This content is amazing ser. Very Refreshing

[@markostojkov7763]

Great video! I respect the time spent

[@LeleSwapper]

Really interesting and good video, especially since its your first video. keep up the good work!

[@tony_saro]

It's my first video on this channel. Not my first video ever.

[@hitach2000]

This is so cool , i'm so excited about the incoming stuff , dude don't ever ever think about anything other than making videos like this , this might sounds so selfish of me , but you are one of the legends

[@cheeto4950]

Genuinely an amazing format of video. I know the algorithm doesn’t like this format but as a watcher it’s helpful and technical. Big fan!

[@tony_saro]

Short & quick content is popular today but things like podcasts still work, I think this kind of content does have its place

[@DataPains]

Very cool man! That's a lot of work, respect.

[@r1nlx0]

This is amazing rust journey !!!

[@Israel220500]

Amazing work dude. I had the idea to try to implement a database from scratch as well, but I always either didn't have time or was to lazy to start. I guess I'm gonna use this as motivation to get into harder and more complex CS problems.

[@mplm28]

Low level stuff are amzing and rare at the same time. Great video ! Keep posting this kind of content

[@xDeltaDoggox]

I followed you since you opened your Spanish channel. Every time you upload a video you blow my mind with something harder. You're the best

[@DevelTime]

Pure gold, and the quality is so high that I cannot even find a thing to complain about 🙂 Maybe one, not a complain though, more a wish -- since I worship Rust, but does not have time for learning, it would be great (from my perspective) if you could keep screens with code (slowly scrolling or fixed) more often and a tad less with you (like full screen screen code + you in the corner). But this is little non-issue even (but still it would spice things up). Many many thanks for you stamina (7 months is no joke) and sharing! Liked, subscribed and now heading to your next video 🙂 All the best to you!

[@tony_saro]

I know it would seem like code helps with understanding what's going on but it doesn't at all. All the algorithms I implemented, I didn't come up with them, I copied them from SQLite, I had the code right in front of me and yet it was such a struggle to understand. You can check the source code any time, it's linked in the description, but if you don't understand the ideas and concepts behind the code it's not as useful as you might think. The content that helped the most was actually the lectures from CMU Intro To Database Systems, which don't show code, they just explain the concepts like I did. Regarding Rust, I wanted to decouple what I'm explaining from a specific language, because you can implement all the data structures and algorithms I mentioned in any language.

[@mykhailo_klym]

This man is on fire 🔥 Keep producing more content !

[@yashS4201]

Bro this is just pure gold and the resources you provided are damn , very helpful for the graduated students like me to understand databases internally .

[@tony_saro]

Thanks bro

[@ricardom860]

Great Job!! excelent content!!

[@larkohiya]

This video was worth the time it helped understand intricacies of the process that I knew but never understood

[@snarkyboojum]

I've been wanting to do something similar for a while. Thanks for putting this out there!

[@brazenintellect364]

THIS!!! This is real engineering!! Excellent stuff, much appreciated.

[@morrisroberti1205]

Thanks, this kind of videos are majestic, keep up with these 👍

[@tony_saro]

Thank you

[@snof7602]

Thank you so much for this explanation, KEEEEEEP GOING 🙏🏻🔥

[@daniel-ro2jm]

Volviste Profe Sarosi ya se te extrañaba

[@tomjerry5772]

Found a Gem. Subscribed. Please continue these

[@tony_saro]

Thank you

[@michaelutech4786]

Very well done! Kudos! This is like a time machine to an era where people didn't fear learning.

[@tony_saro]

True, makes even more sense nowadays with AI

[@FelipeV3444]

Demystifying DBs with awesome explanations graphics... amazing!

[@filippomenchini]

This video is just perfect. Keep this up!