Z80 Retro #18 - How SD Cards Work

[@andrewherron7521]

I just watched this video again abs replicated the steps on my Retro board - 🤣 All working perfectly and another super helpful video John. I’m using the external SD card reader on my current board but my new boards just arrived from JLCPC so I plan to build another board with the surface mount slot this time.

[@JohnsBasement]

Glad to hear you got one going!! 😀👍

[@radarsmutny8462]

yeah, thats what I had in mind, to hack (ommit) filesystem in simple use of sdcard ...thanks for all details!

[@martandrmc]

3:20 regarding the mini-sd cards, i had a small sanyo camcorder that had a normal full size sd card port but for some reason inside of it there was a 2gb mini-sd in an adapter. That's the only instance of one i recall at all, and surprisingly it still works perfectly. Oh and by the way, thanks a ton for those videos, they really help me move forward on my own homebrew computer system. Even though i never lived to experience the 8bit or 16bit era, i still want to look back.

[@JohnsBasement]

You are absolutely welcome! Giving enough detail for someone to make their own custom system is exactly what I was going for in this series! 😀👍

[@ewetoo]

John, just wanted to thank you for your exhaustive treatment in these videos, it's a good mix of just enough hand-holding and hopefully stimulus to further learning for the newbies out there. dd can be a disaster if not handled with respect (like most unix tools!)! I haven't gotten into the guts of CP/M before and I'm wondering if it's recent open-sourcing may be an opportunity to nail down some sort of filesystem format at last (we can only hope).

[@JohnsBasement]

Thank you for saying! It'd be nice to see a standard emerge where the disk parameters incorporated into the filesystem. Given the resource constraints on these 8-bit systems, I am not sure how far we can push things.

[@ganopterygon]

Great job! When I need to decode substantial amount on data like this, what I do is a C program that contains the structures defined (by the file system in this case) and then I just dump the binary into this C program in Linux that then prints out the information, I did this when I added support for Compact Flash and FAT16 in my Z80 computer, then these defined structures can be used in the actual Z80 computer ! :)

[@JohnsBasement]

Thanks! This is good advice. I do the same thing when things get complicated. A simple example is the diag info about the partitions in the final FLASH boot loader. The CP/M filesystem is borderline simple enough that I decided to forego dumping it fancy. I think there is also value in looking at the raw dumps. Having the details of what is adjacent to what in your face is useful when accidently writeing off the end of an array etc.

[@JohnDlugosz]

I use a Perl script with a "decode" command. It can print the list of resulting values without the need for print statements naming each one. I can also put the labels in a list and easily print the labels and values together.

[@JohnDlugosz]

@@JohnsBasement There are dump/disk-explorer programs on Windows that are much fancier. They already know the structure types of common disk formats, and you can put in your own structures. It lets you update the values, too, so you could change your "type" byte and save it back.

[@JohnsBasement]

@@JohnDlugosz There are probably fancier ones for linux too. All I care about here is getting it done with simple tools that are part of a mainstream distribution and thus trivial to install and remain so for the next decade.

[@stamasd8500]

Ahh, /dev/null, the original write-only memory. :)

[@linuxninja]

DOS and GPT Partition numbers have always been 1-based, not 0-based. But it is interesting that the disk drives on the TRS-80 were :0 :1 :2 :3, etc.

[@JohnsBasement]

I waffled on calling the first one zero vs one. I wish everyone would be consistent!! 🙄

[@rwsrwsrwt]

@@JohnsBasement So do I. But even Linux counts the partitions starting from 1 like "/dev/sda1" (and not "/dev/sda0") … while on the other hand it starts counting from 0 for drives such as "/dev/fd0", "/dev/mmcblk0" or "/dev/loop0". At least we don't have to cope with CHS-addressing anymore with cylinders and heads starting with #0 and sectors starting with #1 plus possibly different (fake) geometries used for the BIOS (max. 1024 cyl. / 256 heads / 63 sect.) and for direct I/O with the controller (max. 65536 cyl. / 16 heads / 255 sect.). That's where I got stuck for quite some time when I was younger, trying to write my own operating system (for a 386 though, not a Z80). But where's the fun in it if everything just works right away without anything "weird" to complain about how it is implemented?

[@JohnsBasement]

@@rwsrwsrwt 😂😂😂👍

[@geowar20]

Partitions start a one because the partition maps are considered block zero.

[@JohnsBasement]

@@geowar20 🤔

[@EngineeringVignettes]

A very comprehensive (and thorough) tutorial on SDcard low level operations, nicely done. Some thoughts from me.. maybe you have discussed since this video (I have not checked yet) apologies if you have already touched on them. - [MiB] I think this is pronounced "Mih-Bih-Byte"... KiB would be Kih-Bih-Byte. Something the PC (personal computer) crowd came up with, I think from all of the rage that occurred when HDrive manufacturers claimed their drives were X bytes when in reality they showed a smaller size on the PC... I think that's the story... - [1st partition offset] : That nearly empty 1 MiB space can come in real handy to put in low level boot loaders (in firmware designs they can be referred to as FSBL's). For example, the NXP i.mx6 SoC's can be configured to put the FSBL and SSBL (SSBL is usually U-Boot) into that "raw" region on the SDcard. I would imagine that the CP/M system could do the same.. the ROM bootloader just has to know enough about SDcard I/O to read the FSBL (maybe OS?) out of the space behind the MBR then the FSBL mounts the SDcard volume correctly and uses it for the drives. Simple to add-in using dd. - [Volume type] : Erm, you could set the type to whatever you wanted to even if parted (or other) partitioning program does not support it. Just find the offset for the byte and write it in using dd. The size can be set to 1 for a "1 byte page". - [Saving the initial MBR] : Read it to a file then it's trival to just create a new SDcard by re-writing it back using dd. In the embedded engineering field we use dd to duplicate entire SDcard images. A very handy program. - [sda on RPi] : Looks like the raspberry Pi's use the eMMC peripheral to talk to the SDcard so the card ends up being treated as eMMC media and hence uses /dev/mmblk0 for the media and the partitions use p[0,1,2...] extension, so no worries about messing up a USB SDcard with the native "drive". Other Linux boxen and SBC's are not guaranteed to do the same so yeah, lots of warnings is good. A trick is to do a 'ls -la /dev/sd*' BEFORE and then after plugging in the SDcard to see what changes. One could also monitor the kernel logs (sudo tail -f /var/log/syslog) or 'dmesg') to see how the SDcard enumerated in the device directory. whew. Cheers,

[@JohnsBasement]

Yeah, by the early '90s, the suits ruined the meaning MB and KB. I'm still in the process of converting over. It's a retro series, so going old-school seemed OK. I will start sneaking an 'i' into my GiB, MiB, and KiB suffixes as I go. I went back and forth on the use of partitions and the MBR. Early on I wanted to try to make it generic enough for people to be able to work on Windows, Linux, Mac, or anything else that tries to get in the way of a custom data image on an SD card. So I went with using a primary partition to hold 16 8MiB disk images. Doing so will let us improve the boot loader and add a timed-out prompt to allow booting from alternate partitions, a ROM monitor, etc. Have a look at the Retro's first draft FSBL. I think it does the same as you outline... but that it does it by loading from partition #1 rather than from the raw device. I like the idea of making it partition-aware. Time will tell if doing so is worth it or not. As for the device names, ultimately I figured that since I use a PI for the FLASH board anyway, using it for a dev platform as well provided an a standardized/generic platform for scripts and so on. Anyone with a preferred alternate environment should be able to translate appropriately. Yeah, I sort-a bailed on making the partition type just right. As long as the host does not interfere... I figure we are good. At one point I considered assembling the MBR in with the CP/M image and writing the entire lot onto the SD card with dd into /dev/sda Maybe some day? Yeah, I definitely agree that 'ls'-ing the device names is a very good idea! I put some arnings in the READ.md file around the dd commands to warn about it. I even mention in one of the videos that if you dd into /dev/sdX before you insert your card that you will create an actual file that could get you confused. So always check that it is mounted first! (...and rm the /dev/sd1 file if you create one by mistake.) Are you considering a retro project?

[@EngineeringVignettes]

@@JohnsBasement - RE: Retro project, yeah I have been assembling parts for a vintage "robot pet" project, based on a TAB book that I got back in the 80's when I was young and broke. It's an MCS-85 system (8085) and is pretty tight for memory as it only uses the RAM available in 3 8155 PIA chips. Planning on really "old schooling" it with a switch based programming panel and an optional "cheat mode" using an Atmel board as a monitor with serial link. The MCS-85 system is a really well designed family of ICs that can tightly integrate with minimum of glue logic. I have not seen much discussion on the chipset. I think it would be a good topic for a video series as well (have not done much on my channel for a while so I thought this would be a good new direction, rather than messing with old radios, I had planned on adding different content but life got in the way...). Mulling over at this point if I want a modular system or one big PCB. I like the idea of using card-edge modules but then I have to construct a card cage... still a work in progress. I have already been doing a bit of posting of questions on this project over on Bil Herd's Discord channel, eg. asking about a good assembler and possible C compiler for the 8085, I don't fancy having to make one myself... Other than the above, trying to get back into some FPGA stuff for fun. I work with them for a living though which makes it a bit harder to have fun with FPGA/MPSoC projects.. perhaps some Petalinux tutorials? I have not decided.. Anyways, thanks for posting your progress on your CP/M system. I am grinding through your content backlog on it. Looking good so far! Cheers,

[@JohnsBasement]

Heh... I think I have an Explorer/85 motherboard around here somewhere. The RC2014-card style headers on the board edge don't seem too bad if your not running too fast. Obviously, I went with a single-board for my Retro-project. I figure it would cost less and since it can be assembled with thru-hole only that any pencil-iron-owning hobbyist could build one if they wanted to give it a shot. When I was a kid and went with S-100, it was OK but costly. In the end I could have just used a big perf board. I learned a lot from it all. And I am glad I still have most of the parts. @Bob Perrin commented on the idea of seeing if I can get the original CPU board going again. It just so happens that I found on old code-listing that reveals enough of the design (since I... kind-a never drew up a schematic for it) that I think I might be able to get it to boot again. Luckily, I labeled most of the chip numbers on the sockets on that board, so I know how to re-assemble it. I like tinkering with FPGAs too. But I am drawn to the low-cost LQFP ones that are easy to use... and have simple tool-chains like IceStorm. I have some BGAs before that stock runs out thinking I might want to try to see if I can get one soldered in. And advice on that?? I have been thinking about making an FPGA VGA video card for my Retro board... with old-school sprites and tiles. Luckily, I have a few ICE40HX4Ks around. It looks like it'll be a year before they are in stock again at Digikey etc. Petalinux would be cool. I have some Xilinx Digilent boards around too that I have yet to take seriously. I think one is a Nexys 4 (before the memory upgrade.) If only I had an infinite amount of free time!!!! :-D

[@EngineeringVignettes]

@@JohnsBasement [FPGA Boards] - Matt Hefferman (KZbin) recently got his hands on a quote "$32" FPGA board (Altera Cyclone IV), I'm guessing from China. Going to check out the video this afternoon... It's really too bad that using Intel Quartus makes my teeth ache otherwise I would have more Altera/Intel dev boards around. Randy Rossi (KZbin) has a really neat FPGA Video design series where he makes a VIC2 for a Commodore 64, with some special extra bits. Very nice series. I did a basic Video raster "core" design a few years ago in SystemVerilog, I want to get back to that soon. I have to brush up on my Verilog (and SystemVerilog) skills again... very rusty. [Infinite time] - Yeah I hear you... we just gotta pick our battles. Cheers,

[@JohnDlugosz]

re [MiB] - it's actually an international standard for engineering, not just something for the PC crowd. Back in the day, having 1K be 1024 vs 1000 was not much of a difference. But once people started dealing with MB and GB, things were _not_ the same. The SI gives meaning to the prefixes Mega and Giga etc. So, the suits were correct. But multiples of 512 byte sectors are a pervasive thing for some people, so the "i" forms were introduced. I never liked that choice, and before standardization I would write a subscript 2 or 10. In math, the _base_ is often indicated this way, and the (pre-Unicode) extended character set had symbols to type it! Nobody ever asked what it meant; the meaning is self-documenting. Now how about floppy drive sizes? 1.44Mb is _neither_ meaning! It's actually 1440 Ki bytes, or 1.44 K Ki. What kind of "Mega" is one K of each kind??

[@JohnDlugosz]

BTW, when I took a class, I was taught that the "Copy and Convert" program was named "dd" because "cc" was already taken.

[@JohnsBasement]

I'll buy that! 😀👍

[@PeetHobby]

3:20 mini sd was before Mirco, had a realtive short life, they are obsolete now. My old Nokia N73 used a mini sd card.

[@JohnsBasement]

I don't recall ever having actually seeing a mini SD card. It is unfortunate for the consumer when products are short lived like this. 😕

[@Abrifq]

I used a mini SD card adapter for my Nokia N73, it was really hard to find one though.

[@JohnsBasement]

Yeah...after I did some googling around on it myself, it looks like it disappeared once the MicroSD form-factor came out. Other than an Ebay listing of a 512MB MiniSD for $40.. I can't see anywhere they are even being sold. Buried in the Wikipedia page it notes that: Since 2008, miniSD cards are no longer produced, due to market domination of the even smaller microSD cards.

[@arturslab7102]

47:12 "if you don't redirect stderr to /dev/null, it is going to go into hexdump and make a mess" (paraphrasing). You sure about that? Pipe redirects just the stdout unless you redirect stderr to stdout. So yea, stats from dd will display, but they will not go into hexdump. Also, default size for partitions is 1M. Thus if you use 1 in mkpart, you end up with 1M offset into the drive. According to parted documentation, you can specify different unis, such as Bytes. 1024B would be a legal offset though perhaps not the best choice due to sector alignments. But good stuff. I'm watching your vids and learning a lot. Thanks!

[@JohnsBasement]

The stderr spew will go to the screen and mess up the hexdump output. 🤔

[@jimtekkit]

I just got my Z80-MBC2 board up and running, it's no Z80 Retro but at least it's a starting point when it comes to learning software. One thing I notice is that the MBC2 has problems with SD cards that are too large. From what I understand anything above SDHC doesn't work. But that suggests it relates more to the type of SD card rather than the capacity value itself.

[@JohnsBasement]

I mention somewhere when working on my SD library that the SPI mode is limited to 2T. Because it uses a 32 bit number to express what 512-byrte SD block to read/write. HC and XC therefore should work if you initialize them properly. The spec specifically says that UC cards don't support SPI mode.

[@jimtekkit]

@@JohnsBasement Okay that's good. The idea of being stuck with small capacities that are no longer in production and only available on the used market would be a real pain, so I'm glad that's not the case.

[@tgopaul]

hi John, Great series on the Z80. I have a device running on a breadboard and am using your design to fill in gaps in my knowledge. Is there an order or number to each of your videos. I often get youtube recommendations to your videos but they come without any sense of order. Part way into the video I realize I am missing a detail you covered somewhere else. It looks like some of your videos cover different Z80 projects. Lots of great content. I’ll watch them all if i can figure out the best sequence.

[@JohnsBasement]

Yes there is! It is in a playlist here: kzbin.info/aero/PL3by7evD3F51Cf9QnsAEdgSQ4cz7HQZX5 It is annoying that they don't make the description text as prominent as they used to on these things. On my channel, you likely want to click the all-but-hidden 'more' link that expands the video description where I include related links and stuff that might be interesting. I suppose it would not be a bad idea to go back and edit the titles of all the videos in my various playlists and add numbers to them. Maybe that can be something I can do tomorrow... Actually, the incredibly (over)simplistic tools that KZbin forces us to use don't appear to allow us to label videos in a playlist like this unless the owner of the playlist has created, owns, and can edit each of the videos. I made all but maybe 2 or 3 in this series and can number them. But those included from others for reference can not be labeled in any way by me other than to put then in my playlist. So this can mostly be done. But not completely. You are going to want to find and use the playlist that I assemble with my videos and a few others that might be of some additional help too. Let me know if you have any better ideas.

[@tgopaul]

@@JohnsBasement thanks for the playlist ... numbers on video titles are useful to be sure i don't miss a video .. i watch from multiple devices and locations so I can't reply on the player keeping track of what I watched or missed on a public machine i might browse to your videos and click a video if i can remember which number I last saw.

[@giacomolontra3707]

great videos. all of them.

[@JohnsBasement]

Thanks a ton!!

[@catgirlQueer]

you said the amount of code your board would need to handle something like FAT32 would be bigger than the operating system? I don't know how well 6502 code size and z80 code size correlate (I've never used a z80 before, might try to make my own z80 system at some point as it seems interesting enough) but on my own 6502 machine the FAT32 library (+ LFN handling and other QOL things, it could probably be trimmed down significantly if I wanted to) takes up about 8KiB of ROM and about 5KiB of RAM, again, I don't know how well that correlates but I do think it's decent context for the kind of space 'fancy' FAT32 handling takes

[@JohnsBasement]

They are probably comparable. And yeah, 8k + 5k is definitely bigger than all of cp/m and the bios. BUT at that size and with putting in banked RAM it would be viable. 🤔

[@johncochran8497]

Given that the SDCARDS that handle SPI all have a maximum clock rate of 25MHz (and it's highly unlikely that they'll increase it, since if you want a faster data rate, you'll use the native interface which is much faster). So we have a theoretical limit of 3.125 megabytes per second. Now how fast is that really? 1 GB = 5 minutes 20 seconds. 1 TB = 3 Days, 16 hours, 53 minutes, 20 seconds. 2 TB = 7 Days, 9 hours, 46 minutes, 40 seconds. and since they're talking capacities of up to 128 TB for the SDUC specification and it's already impractical to make full use of a high capacity SDXC spec upper end card (7+ days just to read the thing), it makes sense to abandon the backward compatibility of an SPI interface and instead concentrate on making the native interface as fast as possible. Just a note on the native speed those SDCARDS can run at. The original specification started at 12.5 MB/s (4x faster than the best SPI speed) and the current standard has some cards running at 3938 MB/s (1260x faster than the best SPI speed). So that 2 TB card can be fully read in about 8.5 minutes, while a 128 TB card could in theory be fully read in a bit over 9 hours.

[@JohnsBasement]

😂 And we were all thrilled with the 3.5" floppies when they were taking the market by storm!

[@JohnDlugosz]

A cheap IOT thing or embedded application may save its files as it goes, and does not ever read/write/copy an entire card! It only writes its own files, and the creation rate of the data is low enough that bit-banging SPI is fast enough! Now when you take the card out of this remote monitoring station (for example) after a year, it would be nice to dump the data to your laptop _much_ faster. But the device itself never does that. So, now the users are annoyed that newer cards can't work, and there is really no reason to build such devices any other way.

[@randallrouth9029]

I have not worked with SD cards of rare OSs. However, "fdisk" allows for the creation of CP/M partitions (52). Do you find parted better than fdisk? I am interested in seeing where you are going with this.

[@JohnsBasement]

No particular for parted vs fdisk. Use what ever you think is easier for you. WARNING: You MUST put the partition in the right spot to use the first draft of the BIOS code that I am writing. I discuss this when I am working on the BIOS later on. The only thing that matters about the partition is that it is primary and starts at SD block number 0x800 to be compatible with the first draft of my BIOS. Using a partition type like 52 could mean something that I am not aware of to some host the might try to do us favors that we don't want. Not likely, but possible. Best to mark it as NOT active!

[@michaelhaardt5988]

@@JohnsBasement When I wrote my bootloader and BIOS, I found it easier to let the MBR code look for the first partition of type 52 and copy its start as offset to the BIOS driver. That way any partition start works and you need an offset anyway, so it isn't extra work really.

[@JohnsBasement]

@@michaelhaardt5988 Yeah. Perhaps I should have put more thought into my first draft. I just wanted to get everything running and then go back and clean up the rough edges after I know where I'm going. Even the partition type number bothers me because cp/m provides no way for any utility to know anything about the disk parm block info used to format the drive. Is there any work out there on using a PBR for this with CP/M?

[@michaelhaardt5988]

@@JohnsBasement I am not aware of any and it would require a dynamic allocation bitmap vector. A few Amstrad systems store an extended DPB in the boot sector of a floppy, but that is BIOS specific, not portable. It is probably the largest weakness of the CP/M file system. My implementation uses a fixed format of 8 MB, ignoring the end of the partition, as compromise between ALV size and file system capacity.

[@JohnsBasement]

@@michaelhaardt5988 Why would it need an allocation vector?

[@joeysartain6056]

Is there a reason why loading programs at C000H instead of the standard 0100H?

[@JohnsBasement]

Yes. The O/S is what is loaded from the boot blocks on the SD card. It is not a standard program. The boot loader is designed to load CP/M therefore it reads it into where the CP/M O/S goes, not an application program.

[@joeysartain6056]

@@JohnsBasement I am very interested in this system. I have thought for years that the 64k boundry could be gotten around. (Where there's a will, there's a way). I have learned a great deal already from your videos. I started out on a brand new Kaypro 2x, and there remains a soft spot for the Z80 system today. Keep up the great videos!

[@mikekaffetzakis]

Hello John i have a problem partitioning the card. i used pi 4B and the 16GB card you recommended i setup my pi according to you video everything go smooth until i give the command " (parted) mkpart primary 1 135". I am getting this error : Error: The location 1 is outside of the device /dev/sda. no matter what i try look the internet cannot make it work and i am not a Linux/UNIX guy please help if you can thanks .

[@JohnsBasement]

Are you sure that no other USB devices are connected? Did you zero the SD card out with 'dd' first?

[@mikekaffetzakis]

@@JohnsBasement Yes i did it i follow your steps and to be sour i have no typo i copy and paste from GitHub

[@mtalhakhalid1679]

Try using I2C Sd card modules directly with z80 computer

[@JohnsBasement]

I2C??

[@mtalhakhalid1679]

@@JohnsBasement its a seril communication protocol

[@mtalhakhalid1679]

@@JohnsBasement kzbin.info/www/bejne/p4Wvg4Wda7KYm5I

[@JohnsBasement]

@@mtalhakhalid1679 Sorry for my brevity. What does I2C have to do with SD cards?

[@mtalhakhalid1679]

@@JohnsBasement it uae i2c protocol to communicate bwtween arduino or otger microcontroller

[@geowar20]

I don’t see any value added by using a MBR… I’d just use /dev/sda and access the whole SD card. Put my CP/M boot loader at LBA 0.

[@JohnsBasement]

The benefit on the retro side at the moment is nill. Down the road, I figured I'd want to be able to boot from partition 1, 2, 3, or 4 (maybe boot from the 'active' partition) or some such thing. As I mention somewhere in one of my recordings, the use of partitions at this time is to allow the SD card to be inserted into a desktop without the desktop deciding to become helpful and want to re-format it all the time.

[@dave7f611]

"Embiggenate your readme file." How perfectly cromulent.

[@JohnsBasement]

😁

[@Lucretia9000]

1024 is MB, 1000 is this weird MiB stuff afaik.

[@JohnsBasement]

It surprised me too. But the KiB, MiB,... thing seems to have evolved in the '90s due to confusion (and perhaps annoying marketing departments?) to represent the binary values of 2^10, 2^20, 2^30,.... I'm ignoring it because I'm old and feel like some sort of wanna-be/poser when I use the new terminology. 😖

[@Nikage23]

first view))

[@JohnsBasement]

🎉