Optimizing Communication and Networking in Database Systems - The Backend Engineering Show

[@codedoctor3265]

Wow , I'm listening to your videos while working !! No more listening to songs while working nowdays !! Your videos are really helpful for senior engineers :) . Hard Core Stuff !!

[@TheRDB46]

I dont want to tell you what to do, but would love to see you build out a system from linux distros to networking entire backends, im sure you can throw in some curve balls or ask the community, appreciate everything you do mate

[@batman4797]

"Once you add Django, mango, fango and flask and react" HAHAHAAHAHAHAHAH

[@ritwizsinha1261]

Yeah I also laughed hard at this ; )

[@julienviet]

On the concern of databases being able to support more concurrency, I think it is not related to the actual database concurrency like you explain, e.g you say that row level locks would be an issue. I believe if we are concerned by pure scalability, the important point is how the database is actually implemented. Some database can be implemented with blocking model and therefore the scalability limitation will be related to system resources. Some other databases might be implemented with a non blocking model (i.e event driven) and in this case multiplexing would make lot of sense because the database would be able to perform more work.

[@hnasr]

lets discuss this more because its interesting. you either do pessimistic locking or optimistic locking, you can either do row level locking or lock escalation to table / page lock. Optimistic locking will consume less resources but in case of conflicts the db has to fail the transaction which will cause a retry that puts more load on the database. row-level locking consume more resources (the smartest implementation was postgres where they don't keep track of row locks in memory but in the tuple) but in case of sqlserver or other dbs its just sucks in memory very quickly. I think it will come back to the user's data modeling technique to do as little locking as possible to make this scale better. given that the database doesn't lock in itself .. we will have to wait and see how more transactions and concurrency can be handled by databases.. if you have examples of certain implementations do share. thanks

[@ofek73340]

your channel is a gold mine. thank you so much!!!!

[@AbinashPattajoshi]

Super video! I applauded for ₹1,000.00 👏👏👏👏

[@hnasr]

thanks Abinash for the support

[@MrSunil9999]

This channel is like comedy show for computer nerds

[@gradientO]

Pls do a video on that Instagram story hack by @pgtalal, MrWhoseTheBoss did a video saying he did HTTP proxy interception or something. But I think you can cover more on that. Pls look into that

[@bitethepain1]

Important video with a lot of value. Thanks 🙏

[@autohmae]

21:32 you assume it's encrypted. Their are no guarantees for that as far as I know. It's just 'separated from other customers'. Take for example VXLAN, not encrypted.

[@Taklsdjw093]

If it is not Already exist, we need to have hardware acceleration for streams/channels. Just as we have fireworks celebration for AES which allowed encryption to be more ubiquitous and cheaper Computer resources. Multiplexing support at the water level would solve a lot of this

[@ikonxTech]

thanks man you the best

[@nitishbhatia25]

Wishing i can give more than 1 like to this video. 1 doubt. UDP is layer 4 and they built quic over udp which is also layer 4? Can u please clarify this? And how do they get away with the problem of ossification when introducing quic at layer 4. Any good resource to understanding all this more clearly would be awesome. Please keep making such videos where you think out loud. It’s beautiful.

[@hnasr]

TCP and UDP are layer I think this video helps kzbin.info/www/bejne/bXq2aJqfnNqhr6s

[@utsabbanerjee9672]

Food for thought

[@sivasaianem382]

I read portgres won't allow multiple operations on a single connection. Is it that we can’t extend this idea to applications using postgres ?

[@HarshKapadia]

Really nice video, Hussein! Thank you!

[@Marek13nagy]

Hello, have you considered doing a podcast from these kind of longer videos? Maybe some people would like that idea. What do you think?

[@hnasr]

I do actually have a podcast, check it out husseinnasser.com/podcast Maybe I should start announcing it more thanks for the reminder

[@sumitrawat2037]

27:24 that's not a silly example. I remember in junior school our maths teacher would encourage us to learn multiplication tables of upto 20 numbers BY HEART. And in any case a student cleverly regurgitate it using logic in his viva, they would deduct the marks. NOW THATS SILLY !!

[@darshaniaudhish8240]

Hey Hussein, unable to view parts of your database engineering course on udemy. Section1 video doesn't play.

[@VivekYadav-ds8oz]

Is it necessary to use networking to access DBMS, even if the application and database are running on the same computer? Wouldn't the overhead of ACK packets and handshakes be unnecessary in that case? Also in real world, how often is it that database and applications are hosted on different computers? (If you couldn't already tell, I'm a complete n00b).

[@shravandhar6169]

Almost all the time. The DB's IP should not be known to the end user. It can be configured in a VPS and isolated to a decent level.

[@khalid5d3]

Even if your application is just a webapp or a webservice, you might think you can just host both the app + db on the same host machine. But the thing is, if your app needs to scale to serve more requests you then start to upgrade the specs of your server host, this is called vertical scaling. But at some point vertical scaling will not be enough, if you reach your max performance on a single machine. You will need to horizontally scale, that is to run the same app binary on multiple host machines. And here you have multiple app hosts connecting to a single dbms hosts. How to do that? Well, TCP or whatever. This is just one aspect (the performance). Not to mention other problems which, in order to solve them, you need to have multiple hosts running your app. Example: uptime SLA, resilience, etc....

[@chandeeparora.7165]

Hey Hussein, Can you please do a video on system design for a stock ticker application(like the one you did on twitter design)? Impromptu works :)

[@farooqmss]

@9:55 do you deliberately exclude java based servers?

[@hnasr]

No, Java works too

[@farooqmss]

@@hnasr was just wondering, but I see you talk rather little on java and MySql, but you seem to know these deeply just my observation, might be wrong! cheers!

[@MrEtronic]

Hi nasser i think the problem has been solved only one level lower in the NVMe protocol and NVMe over fabric , although i dont know how they do it . I tried reading the spec and they dont go into MVCC. but it would seem they were successfull in implementing concurrent writes over a network.

[@dvskr1]

NVMe is on top of PCIe ..It is for peripheral connecting right and not a network I/o protocol..I am not sure if things have changed recently,I have worked on NVMe storage connectivity few years back

[@MrEtronic]

@@dvskr1 nvme over fabric is a thing . So I believe they might have something like iscsi for nvme.

[@charliesta.abc123]

Lol yesterday I had a weird error connecting to sql server from my api. "The server does not support the requested protocol."

[@autohmae]

7:38 euh... SQL Server supports SAML ? Yes, it support SSO with Windows Authentication (SSPI/Kerberos usually). But SAML is a web protocol (yes, used for SSO, but not for SQL Server as far as I'm aware). 18:15 My guess has always been, it's also to avoid TCP slow start

[@hnasr]

Ohh I misspoke thanks for correcting me,

[@autohmae]

@@hnasr Yeah, I assume you did. But it does show how much 'web-centric' we really are :-)

[@hnasr]

So much webby stuff

[@autohmae]

@@hnasr If anyone doubts where the future is going: Microsoft is planning and working on making Outlook a Progressive Web App (PWA) and deprecating regular Outlook.

[@lambdamax]

I hope we also learn about the KZbin outage some day.

[@Serpher1]

So HTTP/2 for something like Tomcat/postgresql combo is bad.

[@Euquila]

I just want to hug your mind lmao

[@hitmusicworldwide]

What does Netflix have to say about this