F2023 #06 - Database Memory & Disk I/O Management (CMU Intro to Database Systems)

[@indavarapuaneesh2871]

It seems like Postgres is due for big architectural overhaul. 1. Moving away from per process arch 2. Using Direct IO instead of OS page cache.

[@redixhumayun]

That snide remark against JavaScript developers didn't go unnoticed 😏

[@MichelCarroll]

Sounds like he has a history with them

[@nanunsaram]

Another great lecture!

[@mystmuffin3600]

22:25 "We don't need to have a latch for the whole page table. Assuming it's fixed size, we can have latches for individual pages/locations of page table" Okay, if the latter is possible, why concern ourselves with multiple buffer pools? If these fine-grained latches for individual pages are still a bottleneck, then no matter how many buffer pools we segment our memory into, we will still suffer from contention...

[@avlepikhov]

The idea of attaching one SeqScan to another is quite impressive. But as we know, queries can do such scans with different speeds for some reason. So, should one scan have finished reading a page and wait for all other scans? And if not, How to be with a lot of pinned buffer pages if one scan will be too quick, according to another? Or maybe it's just an implementation issue?

[@李洋-i4j]

I think another reason for not so easy to sharing scan cursor is: when q2 read all the pages with q1, then the missing parts are modified by other threads, then the result will not be consistent neither.

[@mystmuffin3600]

That can't be because every new thread/query starts scanning from the existing cursor. But, if q1 was an UPDATE query which is still unfinished before q2 begins, then we would have inconsistent results because q2 wasn't aware of the latest state before initiating.

[@ahmedalirefaey3219]

@@mystmuffin3600 i think there exist some kind of implementation to read meta-data before q2 start like background worker in DBMS

[@oz5219]

I have a question with regard to the AllocatePage method in bustub when doing projec 1(can't paste link because it keep getting deleted). The way it try to retrieve a new page id is just return next_page_id_++ with next_page_id_ initialized to 0? But should it not consider what other page ids already used in disk? I look around in the codebase but did not find any code try to keep track of the page ids used in disk. Please let me know if I miss anything, thanks a lot

[@oz5219]

an follow up question is that, judging from the ReadPage method, the page id will basically the physical index on disk. Eg if page id is 10, then the method will try to seek the 10th block of the page file. Is this a industry practice? In my mind it seems to be more reasonable if we have a metadata of page id to their physical offset

[@andypavlo]

15445.courses.cs.cmu.edu/fall2023/faq.html#q8