Congrats man!! You're an absolute legend! And have fun at Google!

I think just about all DB indexes will use some sort of write ahead logs. I think what you said is basically correct, but the main reason b-tree indexes are good for range queries is that similar keys are next to each other on disk. Also, in reality, there's lots of in memory caching of B-tree pages that is done.

Yep!

Thanks! Any reason in particular that you think Excalidraw is challenging? Unlike google docs, I imagine that there aren't enough concurrent edits to make a single leader infeasible here

Nothing. You have multiple instances of ham now. On the front end though, we just tell the user that we have one instance of ham.

In theory you can maintain an in memory heap of size k with the counts of the top k elements as you compute the count of each while using the count min sketch

C'mon man it wasn't even spirit or frontier!

@@jordanhasnolife5163 if it was spirit they would seat you on the jet engine none the less pick your poison

Metadata basically just stores the link of the clip in S3 as well as it's sequence number and resolution. The CDN is used as a globally distributed cache of the video clips.

I mean in theory kafka, but I tend to imply that our Kafka cluster has replicas. CDC does make things slower, but I suppose in the cases where I use it I don't actually care (hence why I use it)

Not sure what you mean by synchronous but basically a write must be fully committed in the WAL for it to count otherwise we throw it out

So I'd say it really only has to write to two places (since the caches should be content that is pulled in, not pushed in). To avoid two phase commit, we can just write the metadata row once the S3 clip is uploaded. If for some reason there are some orphaned S3 files that's no biggie

Hey I tried to explain this one more in the video. Cassandra - as fast write throughput as you're going to get (other than maybe like kafka or something and just using a log, which actually now that I think about it isn't the worst solution here). HBase - fairly fast for writes, but important for reads since we only want to access a couple columns of our data at once. You call both of these columnar dbs. To my knowledge, they both use column formats, but HBase uses column oriented storage, which is a significant difference.

Feet incoming?!

I appreciate that man! I already have no life though so you've paid your debt. In all seriousness, once you get that job, no need to keep watching go have fun and socialize :)

@@jordanhasnolife5163 i kinda watch em for fun at this point XDD

She may still be sleeping

@@jordanhasnolife5163 That's OK, the system is fault tolerant. I will eventually achieve consistency via an air horn

@@scottmangiapane lol, I see you're the primary in your relationship then and you're asynchronously replicating your sleep status to her

​@@jordanhasnolife5163 Oh yeah. And if she's still not up, I'll implement sharting I mean sharding ;)

@@jordanhasnolife5163 Oh yeah. And if she still won't wake up, I'll implement sharting I mean sharding ;)

I imagine this would be database specific but I don't see why they wouldn't

Oops, typo on my part. If this were a bigger deal I'd revise the video, but I don't think anyone is losing their job due to using the wrong terminology on cassandra key names.

I'm sorry man I'm pretty pressed for time these days, perhaps you could find one amongst my other gigachad viewers or go asking on blind/linkedin

The server itself doesn't need to be physically near it's location but it should be responsible for the area of the map that the potential rider and all nearby drivers could connect to

Chat server can just expose an http endpoint that the flink node can hit

You're right that there's not a direct ordering, but for a given geohash I do know all of the 8 hashes of the surrounding boxes so I can just check all of those with a Pythagorean equation to check things are in the correct distance from my focal point

1) Yes, let's increment the version vector. 2) Yes, we'd get siblings again lol, hopefully this doesn't happen too frequently and the two leaders have time to synchronize.

So the point is the height index is the secondary one. So his primary hash puts him on 2 but his secondary puts him on 1, now we need a two phase commit.

What about atomicity of transactions...

I don't know a database like we are discussing that has atomic transactions - though there may be some. I mean a database that immediately stores the key-value on disk with an in-memory hash index pointing to offsets on disk. ( Note: There are some well known databases that store sorted key-values pairs on disk as SSTables that provide ACID transactions, but I don't mean those. Note: Cassandra is a database that uses SSTables but doesn't/didn't support typical transactions. But since it buffered entries in memory first before disk, it has a subset of a WAL that was just the "redo" entries. Some call that log file a commit log to make it clear that it isn't a full blown WAL, but I usually call it a redo log. If Cassandra started supporting ACID transactions, it could possibly repurpose the commit logs and make them complete WALs. So my initial comment to you really should have said I was referring to that redo part of a WAL (which is what we were discussing I think in the video). That isn't really needed if you are going to persist all changes to disk immediately) If such a database was created, where hashed key-values are immediately written to disk instead of being buffered in memory for a while... and you wanted to implement atomic transactions -> you could use a stripped down version of a WAL that just has the undo entries. I would call this an undo log. But that's still a subset of a typical WAL, but most people would not object to a undo log being called a WAL so long as context makes it clear. You made a great point above though, which is forcing me to write these thoughts down.. Now, _if_ we were to introduce a WAL (i.e., with redo entries in it) to a database with immediately persisted hashed key/value pairs, the logical step is to actually not write things to disk immediately. That is, you want to spend some time buffering the hashed key/value pairs in memory with redo entries going into WAL first in case the power gets cut. TMI: A WAL is really a combined redo log and an undo log. 1) Redo logs refer to the durability in ACID. They are not in general needed if writes are persisted to disk immediately, because we already get durability when we persist writes. Adding a redo log allows us to program performance boosting features into the database which amount to buffering recently modified items in memory. 2) Undo logs refer to the atomicity in ACID. These are used to aid in aborting or rolling back partially committed transaction.

@@slover4384 Thanks for your detailed response, I appreciate it!

Hey man! Wishing you the best - unfortunately unlike my time at google, I'm now a bit short on time, so I don't think I have the availability to mentor at the moment :(

Seems reasonable to me

I actually found tech lead pretty entertaining until I realized he wasn't being ironic