File Systems - IO from Scratch - Part 10

[@arieloh1998]

Finally good video on the internet about FAT filesystem and some practical examples.

[@weirdboyjim]

Glad you found it useful!

[@randyt700]

He's so incredibly good at explaining in simple terms and complements it with great visuals.

[@weirdboyjim]

Thanks!

[@prawtism]

ChatGPT comment :D

[@randyt700]

@@prawtism chatgpt sucks ass

[@KerboOnYT]

Very cool! It's neat to see the low level detail of all the hoops the computer has to jump through just to read a file. With today's fast systems we take a lot of this for granted.

[@weirdboyjim]

Yeah, one of the big reasons for this project is shining a bit of light on all the things we take for granted!

[@HeadCodeMonkey82]

Loading code directly from the SD card, such a satisfying step forward!

[@AndrewShevchuk]

And also by uploading this code to the SD card via an FTP server, which will require a network adapter on some ESP8266. Then it is possible to provide terminal access. Multitasking and... :)

[@weirdboyjim]

Next step is make the monitor support executable files directly!

[@fabian999ification]

This is amazing. Impressive how you got it to work with the official FAT filesystem! Now the hard part is reading and writing files at the individual byte level, you'll have to know the byte-seek value (32-bit number telling you where you are in the file in bytes) in relation to the cluster size to know exactly how many clusters to access and where you are in the cluster. Writing 500 bytes could mean crossing a cluster barrier if you're close to the end of a cluster.

[@weirdboyjim]

That shouldn’t be too difficult to build on top of what I have, it’s all about creating the right layers to build on.

[@fabian999ification]

@@weirdboyjim For me it took quite a while to figure it out how to do it reliably (especially with edge cases like single byte writes right at the edge). I had to first psuedocode it in C and then translate to assembly language (since I wrote it for my own homebrew 8-bit CPU with 64 KB of RAM), however it is the first time I've made filesystem drivers ever, so it takes a while anyway haha. The filesystem I have made has 16-bit linked-list FAT entries (0xFFFD for bad/inaccessible cluster, 0xFFFE for end of file, 0xFFFF for free cluster), 512-byte clusters (making 32 MB of storage). It's basically just a root directory (32-byte directory entries with 8.3 filenames), FAT table and data region from scratch and is incompatible with FAT. No boot sector, no subdirectories and no long-file-names yet. The hardware disk drivers in the BIOS are capable of IDE/CF-card or high-capacity (HC) SD-card in SPI mode (later I'll support 4-bit mode). I actually followed your tutorial for SD-cards!!! I'm thinking of making a boot sector next with a linear allocated space for the OS (before the filesystem-layer at the sector level) so that it can be loaded from disk rather than being in ROM, allowing for a very simple 2 KB BIOS to then load the 16 KB OS from disk. Subdirectories would just be a root directory structure but instead of being hard-coded at the start of disk, it's in a file (or a folder, which is a file with the folder attribute set to 1) and can be dynamically loaded in for recursive tree directory structures. Long filenames would have the LFN attribute set high, and would then allow for null-terminated ASCII strings amended at the end of the file entry. This is different from how FAT does it, but I'm making mine from scratch and solving the problems as I go.

[@DavidLatham-productiondave]

Awesome. I always think the moment you can load and save to storage is a turning point in any homebrew project. Suddenly you find yourself looking to standardize the filesystem interface. This is also usually when the idea of a C runtime occurs to me. It's the promotion of a project from a hardware one to a software one. Seriously, well done!

[@weirdboyjim]

I agree! Can’t wait to share the advances I’ve made since!

[@m1geo]

I really enjoyed this episode, James. I've always been fascinated with how a hardware project and some code actually become an operating system and a functioning general purpose computer. This series is just starting to get stuck into that now! 👌

[@weirdboyjim]

Thanks George! Glad you are finding it interesting. Loading and running programs from the SD card is about as far as I'll go, but it will make the system much more capable!

[@m1geo]

@@weirdboyjim hehe! I hear ya. But that's the kind of feature set that ties the entire system together. Makes it from a bag of bits to a thing.

[@querela92]

How to make a stone think. 🪨 💬

[@your_utube]

I will need to watch this a few times, because that is what you do to something this awe-inspiring and well done. Thanks James, you are a star!

[@weirdboyjim]

You are very welcome!

[@Packbat]

That's awesome! Really educational, too - you did a great job of explaining the basics of FAT16. It really seems like a great fit for the jam-1 - can't wait to see you expand your library's capabilities and write programs that can take advantage of it!

[@weirdboyjim]

Glad you enjoyed it!

[@luandkg]

The quality of this series is incredible, every day I'm more in love with the details, and today to my surprise you brought a top theme, file system, damn, I want more haha, now seriously, I think it's also time to think about creating a small language at a slightly higher level for your architecture, something like a small compile for your assembly, would make it a lot easier, in any case I'm looking forward to the next video for file manipulation, this video was incredible, congratulations to you .

[@weirdboyjim]

Someone in discord has a c compiler working, I may do a video on that at some point.

[@OscarSommerbo]

A decent dive into the FAT filesystem. Just about right for this format! And impressive that you got it to work without any remade library, but rather rolling your own, as it were.

[@weirdboyjim]

Glad you liked it. I do tend to do everything myself from the ground up!

[@OscarSommerbo]

@@weirdboyjim Coming from a games programmer background where every cycle counts, that isn't surprising. Funny thing, mostly when a project winds down or changes direction I usually lose interest, but your blend of continued hardware development and software to make it work, shifted the focus in just the right way.

[@damouze]

Impressive! I'm currently writing a boot loader in C for a 32-bit hobby kernel and it has both been educational and frustrating because while FAT is a relatively simple filesystem, it has been giving me grief to no end, but I've gotten it to the point that it will now traverse a path and display one or more directory entries in a (sub)directory. However, I can tell you it took me way longer to get to that point than you did! So far my FAT FS testbed has been a FAT12 DSHD floppy image and I've decided to put the FAT filesystem driver aside for a bit and focus on MBRs and partition table scanning, so that I have more options. This has turned out to be its own can of worms. Scanning primary partitions is easy. Scanning extended partitions is hard, because the wise people at IBM and MS back in the day decided that and extended partition table should actually be a chain of partition tables, each with only two valid entries: one for the "current" partition and one pointing to the next partition table. This has led me to wonder if my hobby-OS actually needs to support this scheme, or if instead it should use its own, simpler, scheme. I'm merely doing this for the kicks, but also because after 30+ years of programming in various languages, both machine and higher level (except for Java, which I truly detest ;-)), I had never truly done some x86 protected mode programming at the bare metal level. It was always a great mystery for me and I decided I needed to learn it. The hobby kernel will probably never get finished, but for me it not necessarily just the end result that counts, but also the journey towards it. Ok, sorry for the rant. I really enjoyed this video. Keep 'm coming!

[@weirdboyjim]

Yeah, you’ll find everything to do with these volume “standards” to be a bit tricky, it more evolved than was designed and much of it only makes sense in the original context.

[@damouze]

@@weirdboyjim So, in the meantime I have decided not to support "classic" DOS extended partitions and instead use a simpler scheme for logical volumes. Besides, strictly speaking, there is no need to be able to boot from extended partitions, DOS certainly never was 😛.

[@nahkamursu]

this is so thorough tutorial for FAT that even my monke brain can understand the basic logic, maybe make it bit easier to find with tags for it. I'd hate to miss this information when searching for it.

[@weirdboyjim]

Glad you liked it. What tags would you suggest?

[@alst4817]

This is very cool, great work man!

[@weirdboyjim]

Thanks!

[@GoodWill-s8j]

He did it! 🥷 He did it in assembly! 😶‍🌫️⚡️

[@weirdboyjim]

It could be argued that I take the very long route to get anywhere!

[@GoodWill-s8j]

​@weirdboyjim it was great!

[@TheRealBobHickman]

Nice one James! Reminds me of reading through that massive ST Disk Drives: Inside and Out book back in the day. I got distracted making boot sector demos that were around 400 bytes - fun times!

[@weirdboyjim]

Ha! I did some similar things! Really long iteration time when you only have one pc though!

[@starc0w]

Amazing Stuff! Very interesting! Thank you very much James!🍀

[@weirdboyjim]

Glad you enjoyed it

[@any_name_is_really_fine__]

Love this! the algorithm did me right recommending your channel

[@weirdboyjim]

Awesome! And Welcome, I hope you enjoy!

[@spinbizzy8421]

This was wonderful! For your next trick: synchronizing on $4489 and MFM decoding. You know you want to!

[@weirdboyjim]

I’ll settle for a basic dir command 🤔

[@mark879]

So awesome! You amaze me and I really appreciate you. This was an absolutely fantastic breakdown and it answered many questions for me (about how I thought it worked).

[@weirdboyjim]

Glad you found it interesting!

[@MobiusHorizons]

Really enjoyed this! I love that you have it loading from a filesystem instead of just reading from some specific offset on disk. This may be out of scope for your intent, but If it comes up I would love it if you would talk about how some systems chainload bootloaders from rom to disk without using a filesystem so that the filesystem code is stored on disk instead of rom. Obviously this is how IBM PCs booted but also many embedded arm systems as well. I specifically find mixing magic offsets with partition tables to be quite mindbending (how do you keep them from stomping on each other)

[@weirdboyjim]

Glad you liked it! I briefly mentioned the "reserved sectors" after the boot sector. The code in the boot sector would normally be loading from inside there.

[@KludgesFromKevinsCave]

At long last, much of a BIOS, and much of the filesystem layer! A DOS cannot be far behind!

[@weirdboyjim]

Watch this space!

[@R.Daneel]

Something that's confused me for a while; I frequently see you push registers to the stack, but then pop them off in the same order you pushed them. The code works, so I'm missing something. The systems I've used, you have to pop them off in reverse order since the stack is a LIFO. (Eg. PUSH A, PUSH B ... POP B, POP A). Does this architecture "know" which register is on the stack letting you pop in any order, or is the stack a FIFO here? Is there a video where this is discussed? Cheers! Love the vids.

[@weirdboyjim]

Data comes of the stack in reverse order as it’s pushed. Do you have a specific time instance? I sometimes do this deliberately because I want the data in a different register, I also may have made a mistake with register preservation.

[@R.Daneel]

@@weirdboyjim Oh, ok. That's fair enough. I think it may have been a simple bug not worthy of adding to the edit, or a cut that just implied something not happening. Not worth worrying about. I just finished binge re-watching the whole series, so I'm amazed I found an example! SPI (Bit Banging) - IO from Scratch - Part 4 @ 15:30 onward. That's the one that caught me off-guard as you seem to purposefully correct it to make it "wrong" at one point. The code would have crashed in short order, I'm sure, so it was just fixed without fanfare.

[@Mariuspersem]

Ohh very exciting!

[@weirdboyjim]

Hope it doesn't disappoint!

[@0LoneTech]

Regarding that cache sector, it resembles the functionality of the Forth block word set. You have block (load block, except for the return value, since there's just one buffer), with one more flag of status (updated aka dirty) you could add buffer (map without read), update (mark modified) and save-buffers (save modified buffers). You can probably tell why 32x16 characters per screen was used.

[@weirdboyjim]

I actually added a dirty flag to support writing shortly after recording this video.

[@twobob]

fire. Best, as ever

[@weirdboyjim]

Thanks twobob!

[@phill6859]

FAT started on an microsoft disk basic for intel 8080 or z80. MSX also used it, it survived into 32 bit era but i dont associate it with 16 bit era

[@weirdboyjim]

My understanding was that fat was developed for the early pc’s 8086/8088 which are 16 bit. That’s definitely what I associate it with at least.

[@teknifix]

This is fantastic. I've wanted to learn about file systems for a while, and this is a great primer! Thank you! I have a suggestion. Perhaps you could add a RAM chip that uses SPI for doing the caching (or for whatever use). I'm looking at a 23K256-I/SN from Microchip Technology on DigiKey that might work for you.

[@weirdboyjim]

I've actually looked at one of those ram chips as a possibility. In theory I can interface any spi device now.

[@randyt700]

Lol i hope he comes out with a book on computer architecture and operating systems. Most texts are just too detailed and dense but if they explained things simply like he does here, it'd be a great foundation to build on.

[@weirdboyjim]

I'm unlikely to do a book, making videos already takes up far too much time!

[@azyfloof]

This is really coming along :D Will you be able to edit your games in notepad now on a PC then load them into this? Also, I'd love to see a floppy drive implementation :D

[@weirdboyjim]

At the moment the assembler is still only on the pc, so I have to build the executable and then transfer them.

[@TheEmbeddedHobbyist]

good old dos delete, remove the first char in the file name. reminds me of the fun we used to have back in the days before windoz. A TSR linked into the keyboard interrupt was always fun to throw an odd keyboard strike in to the user typing.

[@weirdboyjim]

I used to have writing TSR programs that played pranks on people!

[@edgeeffect]

NICE! I'd be tempted to just stick to the FORTH "filing system" which is basically specified as "Nah! Can't be bothered! Block read/write will do!" ... but there are certain... erm... trade-offs... with that approach. ;) What does `mov abc, X` do? I've forgotten that bit.

[@weirdboyjim]

It should be easy to add byte level file access over what I’ve done here.

[@weirdboyjim]

“mov abc, X” is a macro I added when starting to do the SD card access. It encodes loads for a,b and c to stick a 24 bit number into the 3 registers. My block access code supports 24bit block indices.

[@guygerstel-wd6bs]

when are you going to add sprites to the vga,or did i miss it

[@weirdboyjim]

Sprite hardware is the last bit of functionality that doesn’t exist yet. Will be starting it soon!

[@sparthir]

Great videos. Sadly the thin font on white is hard to see though with my old eyes and less than wonderful yt compression.

[@weirdboyjim]

@@sparthir you mean on the serial console?

[@desertfish74]

Can I find details about the CPU instruction set somewhere? And do you have a simulator for it or are you always running code directly on the real hardware? Thanks!

[@weirdboyjim]

Yes, I have a simulator I've written. You can get it on the #toolchain channel on the discord.

[@miege90]

Hi James, I've noticed a few times now that you still have the entry nop at the beginning of each assembly file. I think at some point this was out of necessity for the pipeline initialization, but didn't you fix this "bug"? Why do you still keep this?

[@weirdboyjim]

It’s very much a legacy thing in most places. The first byte of the loader ROM still has a problem but it’s as much conventional as anything else now.

[@Elixz89]

Would it be possible for you to implement system interrupts? You would be able to make a system timer and process timed events together with processing interrupts from external devices

[@weirdboyjim]

It’s possible but it would be tricky and I have no plans. I will be supporting interrupts in a future build though.

[@Elixz89]

@@weirdboyjim I have thought on implementing it. This is how I would have implemented it: At the end of a instruction, when a interrupt has occurred, a separate ROM starts executing instructions with its own PC. It executes push functions where all registers are pushed to the stack. Afterwards it will call a address where you can implement your interrupt handler. When the RETI command is executed, the interrupt control block triggers and it will pop all registers back into its original state. You can have a special purpose register where it is possible to have all interrupt sources as flags.

[@GodmanchesterGoblin]

@@Elixz89 I'm not sure that you'd need a separate ROM - why not add the functionality in the existing microcode ROMs if space permits? An interrupt can then simply be regarded as a special type of instruction which would also reduce the additional hardware needed. You'd perhaps need to mod the microcode anyway for the RETI functionality. A further step could be to allow the possibility of software interrupts for OS calls and the like, although I know that Jim has said that he's trying to avoid too much feature-creep (something that I'm not good at...)

[@Elixz89]

@@GodmanchesterGoblin Calling an interrupt would be a very long routine would be a very long pipeline of multiple instructions. Keeping track of the original program counter would be challenging. Having a separate instruction rom to start executing pushing everything to the stack and jumping to a special vector address seems like a more sensible thing to do for a NMI controller

[@GodmanchesterGoblin]

@Elixz89 I have no idea where you get that from. Interrupt mechanisms can be quite simple, as on many early minicomputers and microprocessors.

[@miege90]

I forsee a future with lots of game cartridges, erm I mean SD Cards with custom printed labels. 😂 Years after USB Autostart was deprecated, I see james bringing it back to life with SD Cards

[@weirdboyjim]

To be honest I’ll be able to fit every program ever written for this device on a single SF card 🫣

[@miege90]

@@weirdboyjim Yes I know 😂 But SD Cards are cheap and it would be so cool 😂

[@Otakutaru]

the JAMES, Jam Entertainment system

[@weirdboyjim]

Recursive Acronym!

[@rimmersbryggeri]

Did 8 bit computer have a file system or did they jus just utilize storage as part of the memory map?

[@weirdboyjim]

Some did, most of the time it was as an optional extra floppy drive.

[@rimmersbryggeri]

@@weirdboyjim Yeah but some of them had to have a whole extra computer in the floppy drive for it to prepare the data for the main computer. That seems to be the sollution I have most encountered in 8 bit computers. We never used the floppy drives on my grandfathers ABC 80 while it was in his possesion so I have no idea about those. But a c64 must have had a file system that the drive could read and re arrange it so that the computer could recieve it through the tape loading routines?

[@JanBruunAndersen]

CP/M was a very common OS for 8080/8085/z80 with support for many types of floppy and harddisk.

[@JanBruunAndersen]

​@@rimmersbryggeri - that was the solution Commodore chose, along with the idea to connect the computer and the disk driver over a standard industrial bus. I have forgotten its name, something like IEEE-xxxxx.

[@rimmersbryggeri]

@@JanBruunAndersen On z80 to? I am really a little bit too young for these machines to have been anything other than games machines or relics by the time they caught my attention and honestly as games machines out 8 bit nes and master systems seemed more interesting most of the time anyway; after that it was all DOS machines

[@charlesanthony3248]

You need to add the floppy seek sound.

[@weirdboyjim]

Could probably manage a screech of some kind from the audio device.

[@0LoneTech]

The Mega 65 does that using a floppy drive. ;)

[@cj09beira]

🤩

[@weirdboyjim]

🙂

[@guygerstel-wd6bs]

thanx

[@weirdboyjim]

You are welcome.

[@McGewen]

what a hw is it?

[@weirdboyjim]

@@McGewen are you asking what hardware this is running on? This is my homemade cpu/computer system.

[@McGewen]

@@weirdboyjim cpu?

[@weirdboyjim]

@@McGewen the blue pcb’s are a cpu made mostly from basic logic chips. Building that has been the main focus of my channel.

[@McGewen]

@@weirdboyjim you are genius! i will watch your channel with pleasure! thank`s for such a great work!

[@peter.stimpel]

OT: feckin YT is deploying 4 minutes in 3 ads, on an unlisted video. It becomes more and more annoying. James, did you ever consider to allow your Patrons to view your videos outside youtube? On-topic: nice one, as always!

[@weirdboyjim]

Thanks! That level of adverts sounds annoying! I know patron have a facility to host videos now which might be an option. I'll need to think about that, I want to avoid "overhead creep" in video production.

[@peter.stimpel]

@@weirdboyjim Sure. I don't wanna push you into more work. And I don't wanna make you missing income. However, the amount of ads becomes more and more ridiculous.

[@GameBacardi]

...firefox ublock

[@BrainSlugs83]

Nice work, James! It might not be super helpful, since it's assembly for a different architecture, but the x86 assembly source code for MSDOS is available. If you want to study how they do pattern matching, etc. Might give you a few ideas on how to implement similar functionality in your own SDOS (SD-card Operating System?).

[@weirdboyjim]

It's always interesting to around the source to see how things were done. For my project memory is a big constraint, so working out functional solutions that don't consume too much space is always a big challenge.

[@frognik79]

No need to defrag just yet...

[@weirdboyjim]

Far less need to defrag on solid state media. There are some benefits to sequential reads but no cost like moving a physical head.

[@sabriath]

here i thought you were making a file system from scratch.....oh well.

[@weirdboyjim]

Implementing one as part of a series

[@vanhetgoor]

I really dislike the sound of typing on a keyboard, especially the cheap ones. The people way up in the company have a quality keyboard, they incidentally type one letter to select something, then a solid sound or nothing comes from the keyboard. They do not have to be checked if they are working or just sitting and commemorating in silence. If you are locked up in a cubicle with low life low paid workers then you hear this sound, clicking of a cheap keyboard. Sometimes you have to accept a job, to pay the rent, bloody irritating sound.

[@weirdboyjim]

Fortunately all the kids these days type on touch screens that make no noise, You’ll be fine.

[@ChandrashekarCN]

💖💖💖💖

[@weirdboyjim]

❤️