File I/O for Game Developers: Past, Present, and Future with C++ - Guy Davidson - CppCon 2023

cppcon.org/
---
File I/O for Game Developers: Past, Present, and Future with C++ - Guy Davidson - CppCon 2023
github.com/CppCon/CppCon2023
If you have played a game on a computer in the last few decades, you will most likely have encountered a loading screen. This is used to advertise to the player that, among other things, data is being loaded from a storage device into the game. But why does this exist, why does it take so long, and how can we improve matters for the player?
In this talk we will discover the history of games and their development environments, the relationship between address space, RAM and storage hardware, how C++ abstracts file I/O and why it might not be the best fit for game developers, how a 64-bit address space changes everything, how #embed will change everything, and how C++29 (yes, 29) may upset the applecart yet again.
Although this talk is aimed at game developers who are often Windows programmers, it is relevant to anyone who has to read large amounts of data from local storage. Expect tales of woe, discovery and jubilation as I describe every surprising little thing I've learned about file I/O over the past 40 years and how C++ is heading for a great future in games.
---
Guy Davidson
Guy Davidson (he/him) is the Head of Engineering Practice at Creative Assembly, one of the UK's oldest and largest game development studios.
Guy started writing games over 40 years ago and has spent the last 24 of them at Creative Assembly. He is the co-author of Beautiful C++: 30 Core Guidelines for writing clean, safe and fast code, published by Pearson in 2021. He is a frequent conference speaker and organiser, as well as being a co-founder of #include and a regular C++ committee participant.
---
Videos Filmed & Edited by Bash Films: www.BashFilms.com
KZbin Channel Managed by Digital Medium Ltd: events.digital-medium.co.uk
---
Registration for CppCon: cppcon.org/registration/
#cppcon #cppprogramming #cpp #gamedevelopment

[@franciscogerardohernandezr4788]

The power suit of the software development world (beach shirt) makes a glorious comeback. Quality talk is guaranteed.

[@Alexander_Sannikov]

Let me save you an hour: #embed (which is basically a binary #include to embed data from files into static arrays compile-time) is proposed, but not happening yet. Fastest way to read data from the hard drive is still mapping it to avoid buffered reads, there's still no standardized solution for that in C++, although there's a bunch of proposals submitted (std::map_file is still not happening). On windows 11 and RTX3090 you have DirectStorage that allows direct transfer of file data into the VRAM, bypassing RAM/CPU.

[@fredlament]

This is by far the worst cppcon talk I have ever seen, his information about DirectStorage is also wrong, it won't bypass Ram/CPU, but just bypass the majority of the traditional IO stack on windows.

[@billgrant7262]

So what are and what's wrong with buffered reads?

[@Alexander_Sannikov]

@@billgrant7262 buffered reads copy data into a temporary buffer before giving it to you, which allows easier logistics but copying to an intermediate buffer has a cost associated with it.

[@bravefastrabbit770]

"I should have taken it as a sign of things to come when, in 2015, Jon Kalb tried (emphasis on tried) to introduce me to Bjarne. Bjarne looked me up and down, put on a face of disgust, and walked away."

[@PanzerschrekCN]

It's unbelievable that C++ still has no std::embed. This feature is very useful and easy to implement by compiler authors. Now i need to implement my own embedding via external script that generates a header file, which i later include.

[@heavymetalmixer91]

Talking from ignorance here but I think it's because most embedded devs preffer C to C++.

[@SianaGearz]

@@heavymetalmixer91 That argument doesn't make much sense. People have been wanting to include stuff into executable for pretty much ever, and C only got this facility in the future. C software tends to be super conservative with features since it often has to target remarkably odd and wonky toolchains. And C++ is exceptionally big in embedded since you can have true zero cost abstractions - just no STL on microcontrollers, no RTTI, no exceptions, usually limited to no dynamic allocation, some oddities.

[@SianaGearz]

Depending on which toolchains you have to target, you can have a macro that emits inline assembler with 2 symbols, one being incbin of the file and the next being an end-marker. Difference is length. So you can possibly have it all within your C++ code even though it's strictly speaking not implemented in C++. Haven't watched the talk, don't know if he mentions it.

[@ElPikacupacabra]

Excellent talk!

[@xarcaz]

Phenomenal talk from Guy as always. And looking fabulous

[@therealvbw]

Very interesting talk

[@Alexander_Sannikov]

the talk starts at about 37:00

[@peterevans6752]

Lots of history in this talk leading to presenting summary description of three papers p1040 p1031 and p108? all dealing with improvements to fileio for c++ or the c standard. Could be some better details on the costs, but its only an hour format.

[@goeiecool9999]

43:16 I feel like reading a memory mapped file on a different thread is a hacky way to do it. Linux has the mmap system call option MAP_POPULATE. Granted this might not be portable, but at least your program doesn't need to make assumptions about how the page faults correlate with disk IO requests and performance should be similar if not better than doing it manually.

[@__hannibaalbarca__]

Oh, what’s file; this my first question in 1990 when start first time learning computer programming in club, Dos and GWbasic.

[@binary132]

Nice talk. Standardized mmap would be rad!

[@TheFeralEngineer]

I work for DMG Mori, so I got a chuckle at 49:18 😅

[@gapinggawker822]

fat32 is still used to format some usb for BIOS updates

[@mytech6779]

FAT16 and 32 are still common but they are so small they get lost in the shadows. UEFI firmware ROM, EFI boot partitions, SD cards, Flash drives that need to be compatible with many systems, nearly all small embedded devices use it. All because FAT is very simple to access, low complexity with very efficient use of space and computation(energy). Of course FAT has many limitations that make it unsuited to large tasks but FAT certainly still has benefits.

[@theo-dr2dz]

I do understand that doing file I/O in a way that made sense on 1970's hardware does not necessarily make sense on present-day hardware. But I wonder if this can't be solved under the hood. I mean I don't expect that the number 4k for the buffer size is in the standard. That could be increased. And does the standard say anything that prevents an implementation of querying the file size and than set the buffer to that number (with some cap for really huge files)? So that in 90% of all cases there is only 1 chunk and the buffering is purely virtual? You can be a bit creative can't you? And I expect that the iostreams library is specified in quite abstract terms. I don't expect that it prescribes to be implemented in terms of fopen and friends. It could also be implemented in terms of that fancy OS API he was talking about. If you really want to be nice to people running ancient legacy software on ancient legacy hardware, or arduino and other embedded hardware, there could be an opt-out through a compiler option that selects the ancient implementation. So I wonder, isn't he barking up the wrong tree? Is it really necessary to push the committee for new features and could he not just as well lobby with standard library implementers to reimplement the existing interface but optimised for modern hardware?

[@Alexander_Sannikov]

A person from the audience claims that on windows there's currently no way to reserve the address range for a mapped file without actually allocating RAM corresponding to that address range. If I understand that comment correctly, it means that if you want to mmap a 32GB file on windows, you'd have to allocate 32GB of RAM just to reserve its address space. The way it's supposed to work is that mmap should only reserve the address space corresponding to the file size and then virtual memory is supposed to redirect all memory reads from the mapped memory range to the hard drive, which has to be implemented on the OS level (but the commenter claims on windows it's not). Can anybody confirm this or correct me?

[@GabrielGonzalez2]

I think the way it's supposed to work is what they were trying to describe, but using windows terms. Like if you check task manager in the memory section, you should see "In use" vs "Committed" where in this case, you would add 32GB to the committed section but not the In use section. What they were probably trying to say was that there isn't away to map less of the file and that it *must* map the whole file to the address space.

[@XYZW12]

The commenter is wrong. When file mapping view is created then only the address space of view-size is allocated without any committed physical storage. When any portion of the view is accessed by the CPU then the regular 'page fault' mechanism is triggered and the kernel maps physical memory to the accessed address space pages with the content of the appropriate file section. The only difference to the regular virtual memory is that the address space of the view is not backed by the common pagefile but the file being mapped, so when the physical memory is paged out then it can simply be discarded and writing-back-to-the-hdd is only needed if it is modified. The whole mechanism is not much different to allocating address space by VirtualAlloc (flag MEM_RESERVE) and then allocating physical storage behind the individual pages by VirtualAlloc (MEM_COMMIT). File mapping and regular memory allocation work the same way but it's completely automatical. What the commenter probably thinks of, is that the address space is always allocated which can be very large for certain file mappings. If you map a large file portion but only use a small portion inside it then address space is needlessly wasted which can lead to out-of-memory conditions when allocating plain memory later in the application. This was indeed a problem with 32 bit applications in the 2GB address space. But it shouldn't be with 64 bit applications.

[@douggale5962]

No, you are totally wrong about DMA and MMUs. The I/O driver creates an array of PRD structures (physical region descriptors), which give the device an array of {start,length,flags} structures which tell the device where the kernel has mapped that range of address space in physical memory. Also, on top of that, there is another layer called an IOMMU, which is capable of making the physical addresses go through another translation layer, so a virtualization platform can make a guest OS think it is dealing with physical addresses, when those are transparently remapped. Almost every device uses DMA, but they call it "bus mastering" now. Same thing, bus mastering is just more general - you can do I/O port accesses, too, and it might not be memory access, you can bus master MMIO ports. You can do "peer accesses", where one device can access another, you could write an image straight into video RAM from an AHCI host controller. The CPUs RAM isn't necessarily involved. I have implemented many modern hardware drivers since my VIC-20.

[@simonfarre4907]

Having listened to the talk, I couldn't find a place where he said anything differently? Could you be so kind to provide a time stamp?

[@csanadtemesvari9251]

Least hostile c++ developer

[@SianaGearz]

FAT32 has a strong use case: SD-Card connected to a microcontroller. If you have a 3D printer, or a data logger, or one of many kinds of gadgets of this type, they're going to pretty much need to use FAT32. You MIGHT implement exFAT these days but just a couple years ago it wasn't at all an option for intellectual property reasons, so it's still a little rare.

[@atahirince]

i couldnt sleep again, here we go!

[@xbmarx]

When I see a C++ developer with a Hawaiian shirt, I know I'm in for a good talk.

[@luisvazquez183]

If I know C++, what would be a good path to start Game Development?

[@ChrisCarlos64]

There are libraries that you can use to help you make your own game engine if you want. RayLib, SDL2, and a few others if you look up "game development C++ libraries" will pop up. They vary on what they do. RayLib is perhaps you're more complete "package" because it has many different modules you can select from to help with everything from Input, rendering, and more. You can pick and choose what you need. Lastly if you want to do something a little more than build the engine, like get started faster, Unreal Engine 4 and 5 are C++ driven. You can use the engine to make your scenes, import 3D models, etc. Then write your game logic in C++. It has a learning curve but it is widely used so you'll be able to find most answers to the engine online.

[@chainingsolid]

Google "Lazy Foo SDL", goes over using SDL, to make a game in c/c++. Don't fixate on making something impressive just do things to learn for a while.

[@mapron1]

Better to pick on Rust today.

[@mangoman5864]

use unreal engine 5 to develop simple games using c++ % "Blueprint". unreal engine 5 has a steep learning curve. but once ur over that its pretty interesting what u can do .

[@MenaceInc]

Look into the Godot engine or the Unreal Engine if you want to stay closer to C++

[@pingkai]

Attacking C stdio with C++ stream is not wise.

[@vladimir0rus]

RAM is not an another kind of cache. All kinds of cache made in purpose to have access to the RAM. You can't address cache, you only can address the RAM. Only RAM maters.

[@mytech6779]

Not all addressable memory is random access, and ram is often used as cache for yet deeper layers of data. Nor are instructions required to be addressable in RAM or cache. On the other side there was a time when main memory was called "storage" and the only persistent records were made on paper (Printing, automaticly punched cards and punched tape.)

[@GabrielGonzalez2]

I don't think he meant it literally is a cache, like L1 cache, L2, etc but rather the abstract meaning where its just a cache of data from, say, the hard disk. Which makes more sense in the context of virtual memory where unused memory can be paged out to the disk which is much slower than RAM.

[@SianaGearz]

Your memory hierarchy, not from the machine architecture perspective but from the data perspective, could be registers - L1 L2 L3 - DRAM - disk - network. At this very moment, you're interacting with files on disk which cache Internet requests. And those in turn run through a CDN which can be seen as another set of layers in the cache hierarchy, since you're getting local copies from a server within 2000km of you of distributed worldwide data.

[@jeremycoleman3282]

please give me my hour back