Arcanum "Dat" Files

[@TheGameStarter]

I love it when you do a bit more technical video like this one. Lots of great insights. Thank you, Tim!

[@SCARaw]

He used bajts and explained you what dat file is... World changer

[@talideon]

Another reason, especially useful for spinning rust, is that it helps avoid disc seeks by grouping together data that's likely to be needed together.

[@Anubis1101]

Man, it's surreal to hear you talk about this stuff. I remember back to my modding days, unpacking the DAT files, playing with custom backgrounds and item data. Thanks for going over this stuff!

[@motyd]

Entries is later files overriding the same entry from an earlier file is also the method used by the classic Doom games when loading data from the .wad files. Same technique.

[@imALazyPanda]

I just want to thank you for designing with modders in mind. Ive created a few in my time and am working on a big one with a friend for a game we love but is not mod friendly, they allow them but dont have a framework really for them. Devs who publish a modding tool are greatly appreciated.

[@extremepayne]

You say space is no longer at such a premium, but these days a 256GB drive can only have 1-2 modern AAA games on it. Optimization is always appreciated

[@DBagg-zz4ip]

Yeah, it feels like installing one of those would just be inching my ssd that much closer to death

[@OrangeGenerator]

that's because you probably should avoid playing huge latest release AAA games on a 256GB drive in this day and age. there is a reason terabyte and higher NVME M.2s are becoming more common. I am not saying i love games that are 150GB each and who cares about optimisation, no, more just wanting to point out that as games have gotten larger (as have most software), our storage capacity has too.

[@extremepayne]

@@OrangeGenerator yeah but 256 or half terabyte drives are still really common, even in devices marketed as gaming oriented. and I’m sure game file size will continue to balloon as terabyte and larger drives get cheaper and more available

[@OpenGL4ever]

I am using NTFS compression for some game data folders. For example, the game ARK and its folders "Content" and "SeekFreeContent". Before compression: Content = 46.8 GiB SeekFreeContent = 64.6 GiB After compression: Content = 31,0 GiB SeekFreeContent = 34,9 GiB So overall i saved about 45,3 GiB by using NTFS compression for these two folders. I didn't measure the loading times afterwards, but it felt like there was no difference. Today's CPUs can usually unpack the data faster than the storage device needs to read it.

[@DreadDoom]

To be fair modern game sizes are because of large texture- and sound files, not large data files. And saving a few hundred bytes per file is inconsequential if the files are several Gb large in the first place.

[@joshuakirkham9593]

This is really useful info (for thinking about gaming files). Thank you for the break-down as to why this was chosen.

[@mementomori771]

I don't understand all this tech mumbo jumbo but enjoying listening to people speak about their passions

[@igorthelight]

As a software engineer I understand all of that. And those are very valid points! Especially for older PCs

[@z4ckhyd3]

this is one of my favorite videos of yours because the information is so specific and concrete. What you are saying helps me understand computer hardware better. May I request a thread of videos where you talk about how big-name games use memory and hard drive space and how/why our favorite games perform the way they do in their hardware environments? Also, please do an on-screen introduction to your dog or dogs.

[@CainOnGames]

Have you seen my twelve shorts on The Dog, starting with kzbin.infoGTx-4XV-iE0?feature=share

[@GeezNutz]

Love hearing The Dog snore through the whole video lolol

[@vieleanimations]

really valuable insight. loving those technical explanations

[@joaolimaoyt]

Great technical video as always. The number of file descriptors a process can have open at the same time is typically limited by the OS so a process opening too many files at the same time may run out of file descriptors and then that process would need to close files before it can open new ones. Packing assets into DAT files helps with this problem too. Regarding compression, in addition to whole file compression it's also possible to compress individuals entries within an archive even if the whole archive is not compressed. This helps reduce the amount of IO performed. IO is typically slow and there are fast decompression algorithms out there so compression can actually end up speeding up loading times.

[@OpenGL4ever]

Who remembers the good old FILES variable in the CONFIG SYS file of MS-DOS, which was usually set to 20 or later 30 and determined the maximum number of files that could be held open at the same time?

[@TheLeadZebra]

I remember hearing that Titanfall used uncompressed sound files in order to free up processing for some improvements in graphics. This was probably the first time I considered how things like that might come into play for development, before then I just knew of zip files for improved downloads

[@PhantomV36]

I was reading about pointer fix-up tables and how to store game data in a binary file and reading them using their file offset yesterday, but I couldn’t understand why anyone would do this. Then, this video came out… Creepy, but much appreciated!!!

[@Zeropointill]

I'm working on a multiplayer card game, but needed a "bot" to test gameplay. Making a separate build with the bot code to run on was my workaround. Now instead of having two builds and connecting them together, I can just test my build directly and have the binary automatically connect in a pseudo-singular player mode.

[@DBagg-zz4ip]

That’s good info about sectors and seeking. I’ve come across the “how many files to use” question in my own project.

[@seanothepop4638]

I remember being on the lookout for an arcanum module that had the player create a "team" instead of just one character, usually 4-6 for some games, though arcanum allowed lots of followers script wise (endless with that bug where you mentally control the npc, then meta magic on them, then cancel the control then meta but they permanently are yours and the limit was when your game crashed) but a good party where you governed the growth of everyone and their origins would be neat. Loved artwork too and was my first and favorite middle modern rpg.

[@NotoriousBlake]

Thanks for the video, Tim! Super informative and I've always thought it was a little weird seeing the big files, but thought it was more for anti-modding purposes to be honest.

[@WastelandChef]

Invaluable insight as always Tim!

[@SianaGearz]

I remember playing some cheaply developed go-kart racing game in 1998, and it took an ETERNITY to load, because of all the tiny files. Most games that use some sort of pak/big/dat files, they actually have a 2kb placement pitch for individual sub-files within those archive files. Apparently the reason is 2kb block (similar to sector) size on optical media, chosen to be less than a single revolution of a CD, makes seeking or file operations quicker or something on optical media in particular. You don't really care about space wasted on optical media, at least not back then, but you care a lot about placement, since the pickup transport takes almost a second end to end and a third of a second between random spots, it's super slow. In turn how much space the smallest file takes on computer is not only dependent on sector size but on FS cluster size. If you built your computer in 1995 with a 1300MB hard disk, which was extremely generous at the time, you'd have cluster size of 32KB on FAT16B if it was a single partition. With several partitions or smaller drive, 16KB was a more common value.

[@Janonas]

This proves an Arcanum remake is in the works.

[@Anubis1101]

Certainly a missed opportunity.

[@Gouleur]

Sounds very similar to Mike O'Brien's MoPaQ format for World of Warcraft, as well as the latter DAT format for Guildwars 1. I had a lot of fun reverse engineering those 20 years ago and exporting models and audio, less so trying to understand their custom Huffman encoding lol. There is something incredibly satisfying about building structs to try to reverse the binary packing built by the original devs, as a player. It feels like you're getting a look behind the scenes and that some of your code might be very similar to what they wrote for their in-house tools.

[@hpph7133]

I used to do a lot of Doom editing as a kid and the developers of course kept graphics, sounds, and levels in the WAD file (Where's All the Data). For most instances, to edit, one could simply run a file with a patch WAD file (PWADs we called them before "mods" was in vogue). The one odd exception was sprite files for monsters, items, weapons, those required either patching the original file or distributing everything found in the original Doom.WAD except with changes already made, which technically was piracy, OR using a patch utility to partially modify the existing Doom.WAD. The executable file is where the data for monster hit points, behavior, etc was housed and for editing that, unfortunately you couldn't do a patch, you'd have to outright edit the values there (although the easy restore solution is to just reinstall). I've since looked out for that in other games I play and if an editor exists or not and get a rough idea as to where it data is stored. I've noticed a tendency with a lot of contemporary games to outright leave OGG music files out in folder. I still have the mindset of a kid ith a 600 or so MB range and get very concerned at the idea of uncompressed file formats even if I have terrabytes free these days

[@theninedivides6851]

I love these videos so much not because I am in game dev but I am in IT, so its interesting to listen and learn about this experience

[@JFrancoe]

Exactly the kind of video I wanted to see. Thank you from a person who does video game reverse engineering and modding as a hobby!

[@Pedone_Rosso]

I will always remember when my cousin and I went to visit a friend of a friend of mine, in the early 90s, when we were around 15 or so. This guy, who's parents were loaded, had just put together his new top of the line gaming PC, and he had clearly gone far beyond overkill with the specs. After a few seconds with our mouths agape, just trying to grasp the unfathomable amount of space his HDD had, we asked: "One GIGABYTE !?" (Yep, the reference to Back to the Future was still second nature at the time... LOL!) "... What the hell do you do with 1 GIGABYTE of disk space?" His answer: "Well... I just enjoy splashing around in it!" I mean, that's maybe nothing compared to the jump to Pentium with HDD from systems like the Commodore 64 with its tape recorder. But still, hardware related numbers didn't slow down their improvement that much in the last 30 years. Thanks for your videos!

[@nathanlonghair]

Fantastic info, thank you! Extremely concrete and useful.

[@igorthelight]

3:42 - for modern NTFS file system it's 4 Kb. So you would waste even more on big amount of small files xD

[@Almeida0140]

@3:29 One way for people to see this in action is the size vs size on disk (for example on Windows: Right-click file -> Properties)

[@EiriSanada]

Having a RAM pointer load faster than the Windows file system sounds so obvious to me that I'm surprised I didn't realise it until now. When I made mods for Arcanum and whatnot, the ingame loading screen was just something I didn't think about and accepted, though I certainly notice the amount of time it takes to write or delete those files. That said, even though you said it doesn't matter as much anymore, I've noticed some games still use packaged files, and sometimes it feels like a detriment. A 400 MB Arcanum dat might seem big, but having a bunch of fragmented 4 GB or 30 GB paks makes the load time slow anyway, especially with frequent patches. At least we're all moving to SSDs now.

[@MarekAndreansky]

Good explanation, I never realized all the reasons why most games were using large archives to store data, thought it was about size only. What is interesting is that using zero compression may be more beneficial now with fast ssd disks. I ran into it with one game, No Mans Sky. They store all resources in large archives, but whenever you exit a planet or enter a new area, the CPU spikes as its trying to decompress the data and your game stutters for a few seconds. But if you repackage the files (there is a mod that does that, NMS Decompressinator), the data does take more disk space, but the stutters are gone, and loading speeds in general are improved. I wish new games had an option to toggle this, so depending on your hardware, the game would use no compression as long as you are willing to sacrifice the space and have the hardware that makes the disk I/O bottleneck no longer relevant. I wonder if you tested this on any games you are consulting for, or if it is something that you would consider suggesting as an option to include.

[@michaelbolland9212]

Love the relaxing iron-lung sounding doggo 🐕🤣

[@spudd86]

Actually, on spinning rust especially, but slower SSDs too, fast enough compression algorithms can make reading data in faster too. Pretty sure this has been the case since at least the mid 2000s, if not earlier. More modern filesystems will store small files directly in some other data structure, rather than allocating a sector for them. How small and what data structure varies, but usually it's the data structure that holds the list of files and their attributes within a directory. Also "Modern" here basically means after the early to mid 90s. FAT is basically the stone tools of filesystems.

[@Jaqinta]

Hi Tim , first off all before start i would like to say thanks for giving us some logical things about software programming also , i hope i understand the situations right , if not i would like to hear anything from the things i didn't understand it right. 1)All the things that holds informations on these .Dat files , the list that points out the information inside of these .DAT files loads in first execution sequence ? So in each time when program needs a certain thing , goes to list that i need like X information then list checks that X information but also checks that if this X information overwritten by another .DAT file or if something like that happened that file overrides previous information says to the list that, if you need that information don't look that previous file , you now suppose to look my file with this line ? So overwrites also the information from that list ? 2) "1000 100 byte files for every perk and skill , attribute you may have little file describes how it Works so that designers can go in and mess with the parameters" on that thing ? Is this things like also for example you did some calculations on which things are used in damage output for lighting bolt skill , so there is like variable stores in there like naming might be wrong but something like -> lightingboltDamageoutput = ( X +Y) /2 … something like that and in your calculation side of your file something like if lighthingbolt cast call lightingboltDamageoutput , so designer can work out those of things kind of also ? If this was the some of the things like so is the right way for doing that compress into less files as much as possible and inform the designer about these things ?

[@CainOnGames]

1) yes, as the DAT files are opened in order, the master list is updated, so if a file appears in a later DAT, its entry is the one used 2) yes again. A lot of design info is kept in those files so the designers can tweak variables to tune skills, perks, etc before the game ships

[@liaminwales]

It's a fun topic, thanks for the video.

[@adamc117]

Love the shirt Tim!

[@benhickson6149]

Amazing video, thanks tim.

[@davidmuldowney]

Dat Arcanum content..

[@techieg33k]

All of these Arcanum videos have actually got me wondering, especially this one. Do you have any thoughts or opinions on the mods and unofficial patches for Arcanum? Do you have certain ones you think are absolutely must if play today so you should turn on, other ones are kind of not necessary for the core gameplay and experience? Arcanum is a game I've been looking at going back to. And going back to it, these are questions that pop in my mind and I think about all the time. So it'd be awesome if it's something you could answer, either here or in a video. Thank you, Tim.

[@malik740]

Its a little off topic but I noticed you keep up really well with the pace of technology over the course of your career. Did you specifically do something 'to keep up' like getting books or journals, go to presentations or courses etc. or was it more a passive process due to your work and hobbys? If its an active process to you have tips or techniques? And if its a passive process, do you feel thats enough (maybe a little hard to judge because 'you dont know what you dont know')? Anyway, love this technical type of video especially yesterday ones about data. I live for and breathe numbers and statistics haha

[@CainOnGames]

It was very much a "learn by doing" process. School did NOT teach technology details but it did give me a strong foundation, so I learned things as I needed them. Example: I needed a millisecond timer to track animation frames in Fallout, so I found a tech paper that described how to do it. It was assembly only, so I wrapped the assembly into a C function, added it to my OS abstraction library, and bam, Fallout had a millisecond timer. Finding tech papers, writing in C and assembly...that came from both school and from earlier projects.

[@elduderino1287]

Hey Tim, I'd love to know your thoughts on the new Fallout London mod but I guess more specifically, fan made mods in general. I think it's incredible that they've made what is essentially a pretty polished full game. I'd love to hear your take as a developer and whether you think Bethesda should endorse the project and the team behind it. Many thanks Tim, keep up the excellent work. P.S Love the popsicle shirt.

[@CainOnGames]

My thoughts on modding: kzbin.info/www/bejne/gnmXnZ-DhtJ9Z6c

[@elduderino1287]

@@CainOnGames Hey Tim, thanks so much for taking the time to read my comment and pointing me in the right direction to one of your previous videos, it’s very insightful. Judging from your comment saying you like mods that add new areas, quests and npc’s I’m sure Fallout London is one that might be up your alley, especially given that you are the godfather! Again many thanks, you rock!

[@SableLeaf]

Tim, if you were to make DLCs for Arcanum, what would they be like? What are the systems and themes that would be emphasised in these DLCs?

[@nathanlonghair]

You can always add stories of course, but I wonder if there’s anything left to wish for on the systems side - it’s already a kitchen sink game 😅

[@Kermitable]

when I saw this in my feed initially, i thought you had retitled the OG video to be more specific.

[@AnnCatsanndra]

Tim time!

[@DamianReloaded]

Ahh dat files therefore visions of Old School 🙏

[@stewartdahamman]

Can hear your cat breathing heavy on this video.

[@Dylanface8791]

I see arcanum in the title I watch.

[@dempa3]

Interesting to get some insight in to how and why! I'd like to ask about the development team composition. For a game like Fallout, Arcanum or Temple, what are the proportions between programmers, designers, writers, graphical and sound artists, and other coworker groups? Many thanks in advance!

[@CainOnGames]

We didn't stick to our assigned roles back then. On all of those projects, I'd estimate 4 programmers, 6-8 artists, and the rest were designers...who also scripted. And several of the programmers and artists also did design...some did a lot of design. I will say to you what I said to guide book authors of my games who asked for the main steps through the non-linear, reactive storyline...."good luck with that".

[@dempa3]

I must admit that I imagined that there'd be a lot more programmers than artists, since programming can take quite some time. Very enlightening! Thank you very much for your answer to my question and for the Herzlichen glückwunsch! :)

[@ricar2]

pkzip! now that takes me back

[@bluefalcon04]

I thought you were maybe referring to "dis file, or dat file"

[@CainOnGames]

Dat isn't what I meant in dis video.

[@OpenGL4ever]

And then another question. Would you combine data that compresses well (TXT, XML, JSON) with data that is already compressed, e.g. PNG, JPEG graphic files, in such large data files? Or would you store the data that compresses well in a separate DAT file and the others in another separate DAT file? In my opinion, there is no benefit in compressing already well-compressed files (e.g. PNG, JPEG) again, e.g. in a large ZIP archive. That is why I would save such files in a second large DAT file. Then I could save the other data that can still be compressed well in a large DAT file and then compress it. But how would you see that? What would it look like with 3D models that could consist of both? For example: a 3D mesh, that could still win with compression and compressed textures, if the 3D model has such individual textures and you don't put them somewhere else in a generic place because they are not generic textures.

[@CainOnGames]

I stored all files in the packed files, because even with files that didn’t compress much, there were other advantages to having them in there.

[@OpenGL4ever]

@@CainOnGames Thank you for your answer.

[@MannerdDesert7]

Is it true that any single stat bonus can only have a certain amount of the same bonus. I was recently modifying some of the files trying to change the all fathers blessing to give +2 to all attributes but i noticed that there was a developer note saying that & I noticed the Vivifier potion actually gives you +1 to all attributes in two seperate bonuses.

[@resb64]

you could almost say it's arcanum datum

[@Killaaqu33n]

Tim! Is there a way you can post the link to sign up to be a member in your video descriptions? The button doesn’t appear on iOS or consoles :(

[@CainOnGames]

If open KZbin in a browser, a Join button should appear. Sorry for Apple and their dumb iOS policy

[@Killaaqu33n]

@@CainOnGamesit’s okay…gives me a reason to fix my PC!

[@AlucardNoir]

I think there are a few file systems that no longer use sectors as you describe them. But they're far from the norm so your point still stands. And honestly, even on something like ext4 you'd still want dat files or equivalent for games. Optimization is the name of the game, and modern game designers suck at playing. EDIT. it should probably also be noted that this concept is why some games have 40-80GB updates.

[@NavJack27gaming]

there might be file systems like that but think about the underlying physical device. the memory cells still need to have a defined physical addressable size. EDIT: yes i know random access but the closer you are to the native design of your physical device the better. even if its minimally different. just go big sectors for big throughput with big files and small sectors for fast tiny file throughput.

[@AlucardNoir]

​@@NavJack27gaming I don't think you understand what a file system does. Not if you wrote that. The underlying physical device does 1s and 0s. The file system is what imposes any sort of logic on it. Standard size sectors have been replaced with clusters but again, there are file systems out there that use neither. Or, to be more precise use per file metadata as opposed to per sector metadata. That tends to make them more efficient, for both small files that would fall under a sector in size and result in wasted space and for large files that would require multiple sectors. And it's not like the memory is random, it's as orderly as any sector based file system, it just doesn't bother with sectors. The begining and end of the file are marked the way sectors would be on a mare traditional file system. That results in a far more flexible and efficient system overall. Edit. Ok I might have been too dismissive. You do know something, just not enough. Modern HDDs and SSDs do use minimum block sizes but your OS is not talking to them, it's talking to the file system. And the file system decides what gets written and how. If you have an SSD block of 4k and you decide to use 512 sectors your file system will force a read and write of each sector 4 times, once for each 512 virtual sector. Similarly, if you use 8k sectors, your SSD will still need to read 2 blocks of data for what the os will see as one. Sector less file systems do the same. They write the file at whatever size it is, and if it's under a sector they treat it like they're writing sectors smaller than what the drive technically supports. Again more space efficient.

[@Dawwwg]

You mixed up the term 'cluster size' with 'sector size'. Offropic; you mentioned SecuROM in an older video about compiling Arcanum for Windows 10, as being a build-step. For the best protection it should usually be 'deeply integrated into the game code'. With all kinds of potential issues. Can you talk about your experience - both positive and negative - with implementing DRM solutions ?

[@techieg33k]

Arcanum means I watch

[@EB-cz4te]

The last time I played Arcanum I wanted to see how I could go with installing my own portraits, even though it might not quite be .DAT file related it was interesting to and configuring the thing with the note pad, blew my mind that it worked. On a side note with Arcanum modding have you played the Drogg black tooth patch yourself? I wonder how much you notice is different from the game that was shipped.

[@KeiNovak]

Man, that (.dat?) was an awesome video on info about DAT files.

[@Dawwwg]

You mixed up 'sector size' and 'cluster size' it seems....

[@FlyingPenguino]

Dat was nice

[@OpenGL4ever]

I have a question about packing all the data into one large data file. Would you make it larger than 2^32-1 bytes or about 4 GiB or would you split it into several data files of that size once it reaches that size? After all, you would then need integer values ​​that are larger than 32 bits for the offsets. There are also old file system formats, such as FAT32 with a 4 GiB file limit, that the user could still use. This question is primarily relevant if the target system requires 64-bit binaries anyway, e.g. because you want to address more than 4 GiB (or 2-3 GiB) for the game without PAE and want to use the extended registers of the AMD64 instruction set.

[@CainOnGames]

I would base the decision on the target specs of the machines I was making the game for. If they were greater than 32 bit, I’d let the packed data files be larger then 4GB. Otherwise, I’d make a set of 4GB files.

[@OpenGL4ever]

@@CainOnGames Thank you very much for your answer.

[@r.rodriguez4991]

That raises an interesting question for me. Are games so big now because there's simply more data or did people give up on saving space with the advent of the $50 1 TB SSD?

[@genericpersonx333]

I hope some more developers see stuff like this because the disconnect between load-times and the actual content of the games is getting very bad in many areas. I have wanted to enjoy so many independent game titles over the last few years but almost ALL of them have atrocious internal data management, leading to load times I haven't seen since the 1990s on far inferior machines. A good example is the game Black Geyser: Couriers of Darkness. Black Geyser is functionally a Pillars-of-Eternity clone, with about the same level of graphics and gameplay mechanics as POE with a somewhat shorter campaign. In theory, it should run better in 2023 than POE did in 2016. That is NOT the case. Black Geyser takes several times longer to load most everything, leading to very long load screens and even notable FPS drops in active play, turning a relatively engaging little story into a slog simply because you have to add in the load screen time to every activity. A five-minute fetch quest becomes ten minutes or more because you have to transition between several load sequences.

[@blighthornsteelmace820]

well, I just reinstalled from steam - and I do not have any modding tool there I think.

[@StrikeSven]

Will you release some kind of an update?

[@CainOnGames]

Arcanum As A Windows 10 App kzbin.info/www/bejne/sF7Sn35ogJeSoZI

[@thescatologistcopromancer3936]

dat file 😏

[@IndusRiverFlow]

Hey Tim - Random question: The scene in the intro of Fallout 1 where the American soldier executes the Canadian - Do you know if that scene was inspired by the famous Vietnam War Saigon Execution photo? Thanks.

[@wesss9353]

How many megabytes was the game?

[@ComissarYarrick]

I always lament that Arcanum never grew it's own modding comunity to the same level as for example Baldur's Gate 1&2 did. Partialy it is due to it beeing a niche ( if beloved by these few people ) game, partialy due to it never having a good modding tools, nor comunity large enough to develop them. Temple has it circle of 8 team, Bloodlines have (massive!) comunity Patch, Arcanum has, well not nothing, but not nearly that much. Then again, I also saw examples of modding scene thriwing many years after seemingly game had it's prime. Fallout 2 had whole slew of mods that are prrety much whole new games ( Ressurection, Dayglow, Sonora), wich is possible only because /someone/ in around late 2010's esencialy reverse engineerd tools that devs used to make the orginal game. Starcraft 2 never had any in depth campain mods untill 2022 when comunity custom campain mod manager was created. It's late, but maybe, it's not too late ?

[@JavierBonnemaison]

Why don't you remaster Arcanum? The market is there, and your brand is rising.

[@sean7221]

He's explained this already, the IP is not his to touch

[@JavierBonnemaison]

@@sean7221 where there is a will...

[@JPH1138]

I don't know if you've seen his videos, but he kind of is. He's updated Arcanum to work as a native Windows 10 app and removed all of the multiplayer code, and has been cleaning up bugs in the build. It's just that he's doing it as a hobby atm and doesn't have the legal right to release anything. The people you should be talking to about a remaster release should be Microsoft, I'm pretty sure they own the rights after acquiring Activision. (Arcanum was published by Sierra, which was acquired by Vivendi, which was acquired by Activision which was acquired by Microsoft)

[@totallynotthecia]

ass.dat