BINARY vs TEXT File Serialization

[@TheCherno]

Hope you enjoyed! ❤️ Don’t forget you can try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/TheCherno . You’ll also get 20% off an annual premium subscription.

[@ZulnorineP_Coding]

Sir , continue the C++ playlist if not then tell talented teacher for teaching in more advance and has been better as you.

[@LordHonkInc]

Ah, that brings back the nostalgia of opening an .ini file of an old game for the first time and changing things just to see that it actually changes what the game does. It broke things more often than not, but that was my first interaction with the concept of variables.

[@arushford]

My first experience with variables was in HTML which I picked up almost immediately after getting online when I was about 18, by the time I was 25 I was learning c++, it'll be another 15 years before I could actually code well, cherno helped renew my interest and set me in the right direction

[@craftydoeseverything]

Serialisation is a topic I've wanted to see a video on for so long, I got so so excited when I saw this video was released!

[@lerdi]

You have a whole playlist on his channel about serialization

[@cnb_kevin]

The best solution is USE BOTH. On my current project(s), I use only binary because they are small projects. If I had more time, the level files for example would be saved as xml or json. That way merging is easier in git, and if I break loading/saving code and save a broken file, I can edit the broken data manually. Then I'd write an exporter to export those text/json/xml into binary when I export the project data. Only the problem now becomes that you need an additional QA pass to check that binary data also works. Although, player options are good to export as xml, so users can change things. Which is what I used on my previous project. And if you're going to use binary, as ThePrimeagen said recently on his stream: ALWAYS start with a version at the start of that binary format! I'm using this in the current project and I can modify the saving/loading code for level data and still be able to load an older level to save it in the new format.

[@rmt3589]

This is what I was thinking. (Haven't finished the video yet, but am close)

[@arushford]

XML is old as f***.. I understand the purpose you're using it for I would look for a different format. That's a resource hogging dead language. Microsoft invented it but won't even use it.

[@rmt3589]

@@arushford What should be used instead? Is there a better markup language?

[@arushford]

@@rmt3589 json pretty standard now. otherwise Dictionaries encoded in XML are unnecessarily verbose and complex due to overuse of purely structural markup. • Much data in lexicography is inherently headed, but headedness is difficult to represent in XML. • JSON and YAML are no better than XML at representing headedness and at serving the needs of lexicography. • Two existing languages do provide good support for headed data: XML’s historical predecessor SGML and a less well-known language called NVH.

[@kubaofc123]

Text format design is very human

[@mrcobalt124]

'the design is very human'

[@tawandagamedevs]

😪

[@MehmetSarkd]

at the cost of speed

[@princeali316]

😂😂😂😂

[@anon_y_mousse]

One thing to remember is that when using "plain" text that the encoding and file formats still matter quite a bit too. For instance, whether you use ASCII or UTF-8 can make a huge difference, and while YAML is an okay choice, TOML would be better because whitespace really shouldn't be significant in a user edited format.

[@KojoBailey]

13:27 Absolutely! That's the kind of stuff I work with atm with modding, and optimisation is always something I'm seeking to learn

[@SomeRandomPiggo]

I've only ever done binary serialization in C/C++, in Go it is super easy to serialize to text (JSON) so I use that in most of my Go projects. Interesting video!

[@WladaK]

For me one more good reason for text is future changes in the data format. If you add/remove some variables in your format text parsers will easily handle old versions by using defaults/ignoring some keys. But binary ones will need added logic for each change and there needs to be version of the format saved in the data. Of course you should always have format version in all data even the text ones.

[@zoltanujszaszi]

Nice comparison, I would love to see the code as well in a future video!

6:47 I don't have to imagine. I had to edit an Eye of the Beholder II (90s game) savefile to cheat on the game. It was something like "save the game and write down the number of you have. Now throw one and save again. Compare both files to find the hex values and there is the byte you have to change... maybe".

[@madhavgoyal6093]

I was just looking for this. U really read my mind. Thank you so much

[@JoelJose12345]

Can you do json vs yaml? Which is better for storing data, readability, commenting, which is better to use as config file? Are there better formats ?

[@DemoBytom]

in most cases json and yaml are pretty much exchangable. Technically yaml is a superset of json, as in it has more data types supported out of the box, and supports things like references within the document, but overall serializing/deserializing it will, in most cases, be more or less the same. In the end both are key-value pair, human-readable documents. Readability is personal preference. I saw people preferring yaml, but personally I vastly prefer json format, but tbh.. But that's very personal opinion.

[@Brahvim]

@@DemoBytom For some reason I feel that YAML must be easier to parse, given that it doesn't exactly use braces like JSON.

[@DemoBytom]

@Brahvim nah, for a serialer it would be easier to parse json than yaml. Json doesn't care much for indentation, nor white characters, while if you screw indentation in yaml the whole document is invalid. Granted - both are pretty much non issue anyway, because both json and yaml can be validated by tools to be correct, and I in prolly 99% cases nobody writes their own serializers/deserializers for them, since better ones already exist in any language/framework already.

[@anon_y_mousse]

@@DemoBytom Just use TOML, whitespace isn't significant since it's basically just an extension of old INI files, but it adds some extra functionality.

[@asandax6]

Use TOML and stay away from YAML (Curse you flutter) and use JSON if you don't want TOML.

[@ZulnorineP_Coding]

My best programming and my favorite teacher in my whole Life.Thanks to you Sir very Thanks....

[@KoraktorXV]

1:35 no there is no link in the discription

[@Voidload]

Nice vid! Always has thoughts on this topic - Unity serializes in YAML text file, UE serializes in binary. Personally I think the most ideal engine would store assets in text format in editor and during cook process would generate a binary version for that build for smaller size and some "encryption" of data, merging blueprints sucks merging scenes in Unity is somewhat doable from my experience

[@mikeweathers5726]

Great Video! Can't wait to see the follow up. Keep up the good work!

[@Akronymus_]

Personally, for hexeditors, I prefer imhex, because it provides a sort of scripting interface to automatically annotate data in the files.

[@m4rt_]

you bringing up that it's a hexadecimal representation of a binary reminds me of the bell curve memes. (beginner) It's a binary file (middle) It's hexadecimal (expert) It's a binary file

[@ITR]

In the game I'm currently working on we store quests as json in the editor, but use MemoryPack for the runtime

[@iAmTaki]

This and memory management is one of my favorite topics in programming. Thanks for the video!

[@andresilvasophisma]

Another reason for using text format is if your software runs on multiple hardware configurations/operating systems. In web development it's common practice to use json serialization for almost everything.

[@cheesiestmaster879]

could you serialize to text during dev, then automatically serialize to binary during production, so during dev you can easily edit the save file, but in production it is just stored as a .dat

[@custard131]

while it still comes down to whether the format is intended to be read by a human or not, another reason/side effect for using text formats in the shipped build is modding on i can still remember is in GTA San Andreas the data for vehicle specifications was in a text file (csv iirc) and allowed for some fun things like insanely fast tank while determined users will still find a way to parse and + attempt to edit anything the text serialization does make it easier which depending on whether the game devs want to encourage/discourage may be another factor one way or the other

[@andrasbradacs6016]

Binary files contain data of a fixed length, and text files are also binary, with a lot of line breaks and a lot of human readable numerical values, which must be converted back from ASCII to numerical values ​​that can simply be placed in memory.

[@seinou7471]

I don't understand, why(at least in 2024) all the code editor and even notepad decompile or decode idk then compile or encode when you're done editing, so everything stays in binary, it's just test surely it the cost is negligible right?

[@CC1.unposted]

Use binary if you wana store some data (which is not needed to be used by hummans) it can be effecint at reading time, storage etc While use some Text if it's humman designed,it will be slower but readable You can use JSON, or custom fornat or any other Theres tons of them

[@Shineglow]

Does Hezel support something like Unity's Addresables system? I'm meaning a system that allows content to be packaged separately from code into packages.

[@cygnus987]

Haven't needed to do anything with binary for files in a while, largely setup any config type files as text exactly for the reasons described in the video. However if I had to make a binary file format now I'd probably consider using something like Protocol Buffers, mainly for maintainability.

[@feintha]

Im working on a game engine and am currently using binary serializing, similar to how you would if you were pushing/reading a stack (ie packet deserializing) and integrating a reverse dump so the runtime can be the editor

[@delusionalaar4031]

Can’t wait for the next video. Let’s get it!

[@spaceplayertv3789]

do the shipped files get enrcypted or otherwise protected against "malicious" changes/extracting game assets besides being in binary

[@TurtleKwitty]

fun fact your user needs to have decrypted files to use them so it literally doesnt matter in any way

[@custard131]

a waste of time both in terms of development time trying to implement it and loading time having to decrypt at least for the stated goal if you encrypt the assets, then you would also have to provide the user with the key to decrypt them so they could play it not with games but i have seen people try to do this exact thing with software and its a complete waste of time in almost every scenario except for malware avoiding detection

[@ensuretime]

@@custard131 Most games that don't have file encryption are hacked, attributes are modified and can generate a lot of strange behavior on the client or server itself, assuming that the client/server will trust the client's information "a little". e.g.: A weapon has an ID and the client sends that it is using that ID, but the weapon's ID has been modified on disk and now the server "thinks" that the client is using another weapon, that's it, you've exploited a problem... Many games encrypt and decrypt files in parallel, others prefer to hash the file and send it to the server to confirm that the file hasn't been modified. If you don't do either of these things, your game will be destroyed by cheaters. If it's an offline game, no problem, but an online game can't do without this.

[@custard131]

@@ensuretime that is a problem that needs to be solved for online multiplayer games, but encrypting of assets doesnt solve that in such a setup the server needs to keep its own copy of the attributes and use those rather than relying on ones submitted from the client there is essentially nothing you can do to prevent someone modifying how thing appear in thier client, but in realising/accepting that there are ways to design it in such a way that it doesnt impact other users signed data packets can be useful in some cases but i see that as a completely separate topic to encyption the simple fact here is that in order for a user to be able to play a game where the assets are encrypted, they are going to need to be able to decrpyt them

[@NihongoWakannai]

​​@@ensuretime if you're making a multiplayer game then you simply cant trust the data sent from a client. There is no encryption trick to stop hacking, you just need to receive the least amount of data from the user as possible, send them the least amount of data about other players as possible and analyze their data over time to observe impossibilities in the player state.

[@bsdooby]

ASCII is binary data as well; it is just interpreted/read and/or written differently.

[@MrJdcirbo]

It would be really cool to see the implementation differences between text and binary serialization. Do you use a lot of bitwise operations with binary serialization? Is bitwise logic actually faster than standard logic in C++? I have wondered about these things for a while. Lol. Thank you for your videos!!!!

[@malekith6522]

It depends on the problem you're trying to solve. When you're dealing with communication (protocols) where bitwise operations are very much in use and you need to splice your number into bits, then bitwise operations will be a faster and better choice by far. It's just a tool that could be useful, and there is no separation between bitwise logic and standard logic of CPP

[@gelis07]

What notes/whiteboard software did you on this video?

[@zdspider6778]

I'd be interested in seeing how a GLTF/FBX/OBJ can be converted to an "engine-friendly" format for faster loading.

[@custard131]

i will use OBJ as example because its the one im most familiar with thugh i expect the same logic applies for most other file formats when you load an OBJ file into an engine the engine is going to have to loops through the data in that file and build up an array of vertices in memory with whatever data you have for each vertex (eg position/normal/UV coords) one option that would be relatively simple and i think would result in the fastest loading would be to simply dump the resulting block of memory out to a binary file, then rather than needing to perform any processing at all you could just load the file into memory and point a variable at it for a setup where the originals are in text based format and only the shipped version is binary that feels like it would be pretty close if not an exact explanation to what is done

[@Mystixor]

Not packing the assets in the development phase is definitely understandable, however I believe keeping all data in binary on the disk is fine. Instead of keeping the files in text and parsing them on runtime, you can also parse the binary in the Game Engine editor and show the developer a text version or any kind of human-readable UI which gets automatically converted and saved back to binaries on disk. The big downside here is that extra work needs to be put in to create a nice editor, but it may be a nice middle ground for some scenarios.

[@arushford]

Name one time that old story made things better. That's literally the hill that game developers die on.. I'll make a nice editor they say I'll have the best games out there they'll say in the quickest amount of time... Only to quit shortly after and never code again.

[@BinaryCounter]

I feel like the merging and version control point is the most important part of this. Git, SVN, Plastic... Whatever you use, it doesn't like binary files and will likely not be able to merge changes of two contributors into the same file in your repository. If you're working in a team, trust me, this becomes a huge nightmare.

[@abacaabaca8131]

This is a good video..what about a video about cpu vs gpu in game programming. Can gpu execute algorithm? Are they only used for vector multiplication?

[@CD4017BE]

Modern GPUs can mostly run arbitrary code but there are some restrictions: - no recursion (the maximum size of the call stack must be known at compile time and is very limited) - no indirect function calls (for the same reason as no recursion) - no dynamic memory allocation (so you can't use things like lists or hash maps) - branches are potentially slow (because multiple vector cores get fed from the same instruction stream and can't execute different instructions at the same time) - sequential code execution is much slower on GPU than CPU so in order to achieve a performance advantage, you must design your algorithms to run heavily parallelized. There might be some more things I haven't mentioned here but these are the main limitations for GPU algorithms.

[@NihongoWakannai]

Look into compute shaders for executing code and receiving back data from the GPU.

[@mintx1720]

A thing to note is text serialization is really not that bad on strings, it's only in numbers where the perf can get a bit worse.

[@DM31415]

I would like you to discus xml reading writing which are useful for small configuration data store.

[@christianm4906]

Almost everyone in is moving away from xml and adopting JSON instead.

[@malekith6522]

@@christianm4906 Agree, usually on the team, we will never approve the use of XML if we are not forced to.

[@carljalal3855]

I'd put my cutoff at around 100MB. If you can save the file as a JSON string or other text format in less than 100MB, go for it.

[@intptointp]

I just realized that serialization is really not that different from code compilation. Just like when you compile your human readable code into machine code, you compile your human readable text into a binary format. (Serialize your values into a binary format.) That’s all it is. Is it for humans or machines? Your choice will make it easier for one of them. Make it for humans and machines need to translate. Make it for machines, and humans have to translate.

[@urisinger3412]

it really is diffrent, serialization is supposed to be lossless, while compilcation is lossy. compoilcation is more like deserialization, which is more lossy.

[@njnexgen]

​@@urisinger3412 ahhh so your reply was compiled then

[@tedchirvasiu]

@@urisinger3412 Combobulation

[@tedchirvasiu]

@@urisinger3412 what is being lost durring deserialization?

[@urisinger3412]

@@tedchirvasiu depndes how you serlize it, for exsample json deserlization losses some information like indentation

[@true7563]

You didn't link the hex editor!

[@TheCherno]

Thanks, added!

[@andresnexuschamarra6991]

The idea of content files not being human readable on a released product hits a snag if you factor in Modding, if you want games to be mod friendly a large part if not all of your config style content files (aka not 3d models or texures and so on), should be human readable, but, yes, that increases load time, so why not have the best of both worlds? provide a "development" build, might also include more verbose logging, command console, asserts and content / asset validators, which works with text files as well as the release build that pre-processes those files once into binary and from then on will load faster? remember, any modding support and / or tool provided will also likely benefit designers and content creators on the dev team, so everyone wins

[@KarimHamdallah-gc2el]

when I serialize gltf model to binary it loads extremely faster but it turns from 70 mb to 300 mb on desk !!!!

[@Dexterdevloper]

Amazing Contant , always Unique.

[@overcritical304]

Finally, we are getting there

[@dXXPacmanXXb]

Why did you go with YAML

[@nahuelfernandez2352]

I would love to see a vidio of the serialization code

[@ThatJay283]

for text data that you don't want to be text data in the published game, why not just represent it in some entire c++ constructor (or other language) in some file? that way its human readable, but the compiler does the job at serialising it for you

[@parabalani]

I think always use binary for assets. But have an option to convert to text if human needs it

[@keshavraj9518]

Informative video. I would love to see the code as well :)

[@PeterfoxUwU]

I'd love to see how you do good Serialization in C++ Have tried many times, but never got a satisfying and safe result

[@s.boucher9101]

Yes please make a video on binary SerDes!

[@SeanLumly]

Pro-tip: Have the best of both worlds by making an editor that can directly visualize and modify the binary format.

[@marco_martin]

if"other humans come in and change it"... very alien like

[@neil_m899]

I use binary serialization in my game engine, similar to how unreal engine does. I prefer binary serialization because generally, it is faster to deserialize binary data.

[@paradox8425]

That's old school and one of the reasons I kinda don't like UE anymore (Even though I regularly worked with it for past 5 years). 1-) You are forced to use editor at all times which is usually not open since you write cpp 2-) Since it's a development environment frequent changes and crashes occurs. This sometimes ends up in binary corruption without any way to recover. If it was a text file, at least I could see what went wrong to prevent it in future. I might even be able to recover 3-) Using text files doesn't meas you can't use binary files. A good approach is storing only metada as text (depends on asset type)

[@neil_m899]

@@paradox8425 well I already made my custom binary asset serialization format LOL. I'd rather focus on the engine itself. Plus, the asset loading (deserialization) in my engine is super fast. And writing 2 different asset formats (text + binary) is too much for me. Currently, I have a class called Object, and a class called Package (which is subclass of Object). An object can have children objects in a hierarchy format. And Package is the object that is serialized to disk with all the children objects serialized under it. So it's very much like Unreal engine lol.

[@paradox8425]

@@neil_m899 You can do whatever you like. What I'm saying is, reliable and easy to use tooling is much important than most of the people realize. Sure, it might not increase performance of shipped game, but it can make you ship it 2x-5x faster or maybe even more depending on the exact situation. You say "binary assets is fast", but did you benchmark it with the text assets? Exactly. Just like cherno said most difference it can make is 2-3 seconds to load an entire scene which is usually differs much less. And by doing that you gain ability to review chanhes directly from source control and edit without editor open

[@vaijns]

An important thing to evaluate about performance tho, is how often will the reads happen? If it's just once at startup where you have to wait say 1 second instead of 0.1 seconds, who cares (of course if it's many files, that adds up)? But if it's inside some kind of loop that's executed regularly this difference does matter a lot (best is no file io in loops at all ofc).

[@RelayComputer]

I think this novel introduction of background music is kind of distracting, which in my opinion is not good for the content of this particular channel. Please take that as a constructive suggestion. Said that, just to share my experience as a retired software developer: storing and loading application memory models in binary is virtually instantaneous but slightly more prone to corruption due to subtle bugs. Also versioning is more difficult to handle. On the other hand, text based memory models are arguably slower but a lot more robust against bugs (particularly versioning bugs). But please do not use standard json libraries or other library based stuff because these tend to be slow as hell. Just do your own parsing and you will be fine and a lot faster than ANY existing serialising library. In at least one occasion I chose to use both alternatives simultaneously in a single file bundle. Saves created both formats, but opens only used the binary form. However, the text version was always there as a fallback in case something went wrong as reported by the appropriate data integrity checks. But I mostly developed document based apps, not games, so these files were updated by the users by using the app, adding further risk to their integrity, and the need for robust versioning. Otherwise, I understand that loading heavy textures and 3D models for a game is just too much demanding for text.

[@yassine-sa]

My brain ( him not me): IF I HEAR THE WORD THUMBNAIL I'M GONNA BE LIKE 🙎🙆🙆

[@easydoesitismist]

Modding text files is easier. Binaries will need exporters or converters

[@PS3PCDJ]

TOML. Use TOML

[@DM31415]

Sometimes libraries force you to choose between these two formats

[@swapansaha2368]

Sir pls start a series on vulkan with c++

[@Fangamer1254]

High level: Ez "Low" level languages for humans: too hard

[@nahuelcutrera]

wouldn't be great if we could go from text base programming to something even easier for humans like something visual or something we can't figure out yet... I always thought that when I was in college studying this stuff. 😆

[@asandax6]

You mean nodes? Like the ones they use in shader scripting or visual scripting.

[@mobslicer1529]

i can open binary in notepad, i just can't edit it

[@Nikita-kt2oo]

quite useful

[@ic6406]

2k lines will slow down a couple of nanoseconds, not worth to optimize it at all

[@ItsTheSebbe]

Nice Peruvian sweater!

[@CrazyWinner357]

I do text most of the time. Binary is pain in the ass to extend

[@ВалерийЖмышенко-ю6м]

and Unreal Engine still uses stupid binary serialization for all its assets. as for me, the biggest disadvantage of it.

[@zanagi]

tried to learn unreal using my c++ knowledge but the compile time ruins my whole motivation... Not forget to mention that it crashes when the components are not loaded properly, and has two different method of compiling 💀

[@zdspider6778]

But encryption, tho. 🤔 In Blueprints you can use Ctrl+A to select all the nodes, Ctrl+C to copy, which copies it to the clipboard so you can send it to someone else (with Ctrl+V), in text format. I think it works with other node-based assets, too, like materials. But sending the .asset files themselves is a... philosophical decision that Epic made a long time ago to prevent people from stealing in-game assets.

[@dj10schannel]

👍

[@tedchirvasiu]

On modern computers I think the difference between binary and text for a 2500 line file should be in the order of milliseconds. We're talking about kilobytes of data.

[@bits360wastaken]

In the order of milliseconds is incredibly slow for asset loading

[@tedchirvasiu]

@@bits360wastaken Can you observe a 2ms difference?

[@vaijns]

@@jamesmcmanus I think the "how often it happens" is the most important thing. If it only happens at initialization it's fine, if it happens during the game regularly it's a problem.

[@tedchirvasiu]

@@jamesmcmanus we are talking about files which would make sense to be human readable / editable: scene files, prefab files, material files. All of these would probably sum up to a few megabytes for a decently large scene. The big files such as mesh files, textures (images), audio files would never be edited by a human in notepad because nobody would adjust the position of a vertex in a 100k triangle mesh like that. And nobody would care to see how those vertices changed by looking at git diffs. And when I said milliseconds I was extremely conservative, we're likely talking about microseconds. I highly doubt parsing files measuring a few hunderds of Kb would amount to anything in comparision to other tasks.

[@PopescuAlexandruCristian]

If you need serialization there is nothing better then chunks with headers that you just memcpy + relative pointers

[@arushford]

Memcpy is an expensive operation, I'm pretty sure literally every method is better than that, but please prove me wrong, most convenient perhaps, your favorite even but I don't think better is the correct word at all.

[@arushford]

Also even my Alexa seems to understand that men copy can have data format errors where is binary serialization will always allow compatibility

[@DynamicalisBlue]

I wish Unreal was more text-based. Stupid ass shit. I don’t understand how even Data Assets of all things are not in a text-based format. Makes no sense.

[@tgirlshark]

me using base64 encoded textures so I can use git without lfs 😎

[@blamechickenman7434]

serialisation with a 'z' is a big crime

[@flflflflflfl]

Oh no, guess I'm going to jail

[@0xbinarylol]

Me too going to jail 😢

[@professionalcat9928]

both are correct

[@kotofyt]

programming war crimes, yay

[@Redfrog1011]

Says who?

[@kotofyt]

Imagine Serializing into c++ file...

[@Simple_OG]

I code in binary

[@eoussama]

I code by manually manipulating the ions inside of my CPU using magnets

[@vaijns]

I code by making the world/earth itself my game and manipulating people

[@maxgordon3194]

please make more videos on how to use a hex editor. please