Self-Hosted TRUST with your own Certificate Authority!

TRUST. It's what certificates are all about. How do we know that we can trust a server? We verify that the server has a certificate, and that the certificate is signed by someone we trust. That can be a well-known third party like Let's Encrypt, or our own certificate authority. In this video, I'm going to cover the basics of setting up a root private key and signing certificates using OpenSSL, and running a certificate authority server. As a bonus, I'm using a Yubikey to store the certiicate authorities private keys, so they can't be compromised without stealing the physical dongle (they CAN however be used to generate leaf certificates if the certificate authority is compromised). So follow along for a fun journey into the basics of setting up your public key infrastructure!
Link to the blog post with all of the details and commands to follow:
www.apalrd.net/posts/2023/net...
Feel free to chat with me more on my Discord server:
/ discord
If you'd like to support me, feel free to here: ko-fi.com/apalrd
If you want to build your own, here's the hardware I used:
- Yubikey 5 NFC amzn.to/3JcqkmY
- Dell Wyse 3040 Thin Client ebay.us/Ieivdl
Timestamps:
00:00 - Intoduction
00:32 - Certificates
05:16 - Generate Keys
12:09 - Setup Smallstep
21:26 - Caddy Example
23:54 - Demo
#tls #cryptography #publickey

[@br3nd4n]

"Apple - Are - D" So that's how you say it!

[@apalrdsadventures]

yup that's correct

[@jamesr6503]

I was wondering this also

[@blevenzon]

We need to Ansible this :) Awsome video thank you so much as always

[@apalrdsadventures]

First things first, let’s make sure cert renewals are automated

[@smallsteplabs]

Thanks for making this great tutorial, apalrd! 🥰

[@harveyharrison1987]

As always, excellent delivery, nicely paced. Keep up the good work, it is very much appreciated

[@panagiotiskapsiotis2263]

Excellent video! Thank you so much. Cryptography is hard as you mentioned and this is of great help. Looking forward for the next chapters...

[@ehink2716]

Man you are covering stuff I was planning on looking at and setting up but have not had the time. greate work.

[@doublellamadrama]

Hey apalrd, you're doing great work with educating people. Keep it up!

[@bitterrotten]

I love this channel. I've had a handful of tabs open on smallstep for MONTHS. Now I don't have to do any pesky reading. Thank you for helping me learn good easy Apalrd!

[@falazarte]

You are amazing dude! Love your content.

[@apalrdsadventures]

Thanks!

[@frzen]

Thanks I'm looking forward to more episodes about this

[@p0werinside]

Thank you for specifically mentioning that a yubikey can be used to store the certificate key and that smallstep can access it! Great video.

[@LubosMedovarsky]

I'd keep in mind that one specific Yubikey will become a Single Point of Failure. I could talk for a long time how consuming can customers' client certificate replacement be, plus redoing whole PKI from scratch.

[@QuantumKurator]

Good one. Adding to my projects list!

[@Luzer7]

Thanks for the video.

[@random_name-]

Cool !!! great Video.. Will be nice get same solutions to manage the ZFS key for pool severs of truenas..

[@georgH]

I was using my own CA, painfully copying my root certificate to be trusted in all my devices, before let's encrypt was a thing. The main issue for me was that Android keeps a permanent warning message when manually adding a trusted root certificate. Basically, all the ecosystem is designed to trust all keys with the major CAs and doesn't allow you to trust your own. For "safety". Unless you root your phone, of course, which I can't do on my device. Let's encrypt fixed that for me.

[@Subbeh2]

Great stuff! It'd be great if you could also do a video on SSH certificates and how to manage them

[@apalrdsadventures]

SSH keys are a bit of a different beast (since they aren't x.509 based), but they are still using the same underlying crypto (RSA + AES or ChaCha20) and still support certificate authorities and chains of trust. It's on my todo list, but the implementation is quite a bit different than TLS certificates.

[@OscarCarlsson1986]

Yay for smallstep-ca! I have a similar setup at home, using a SBC, a source of randomness & a yubikey. The only thing not working ideally is using the step-cli to renew certificates, if the certificate it's trying to renew is expired it won't to attempt to anything. Using Caddy like in the video works even if the certificate has expired.

[@apalrdsadventures]

I haven't used step-cli for server certificates, just client ones, and I'm still working on a good process for dealing with those.

[@OscarCarlsson1986]

@@apalrdsadventures I'm using step-cli for systems where I don't want to configure and run caddy, so I can run prometheus exporters with a valid certificate (and mTLS at some point). I wrote an ansible playbook to bootstrap all of this but it's a bit hacky. Looking forward to see your experiments in this as well!

[@apalrdsadventures]

I did try other ACME clients with it as well (such as Proxmox's native ACME support), but Caddy was the easiest example. I currently have mTLS running with a cert on my laptop and workstation, but I'm still not happy with the deployment and renewal process yet for end users. It's great when it works though! No more logins (if the backend software supports it)

[@GeoffSeeley]

+1 for the Spaceballs reference!

[@esra_erimez]

Apaird never fails to impress

[@Ebakes94]

Thank You for Making this video. I followed your proxmox LXC with hardware acceleration for Jellyfin as well (using this daily). Your content has been very useful overall and I appreciate it.

[@apalrdsadventures]

Glad it helped!

[@thyme676]

@@apalrdsadventures would this certificate process work well for jellyfin and jellyfin android and/or google tv clients? Thank you!

[@SectorZeroOne]

Awesome content! Subbed!

[@pauletteclark8945]

I just started working on a bachelors in IT I am overwhelmed with the information and allowing myself to think way to hard about things. You really are so much better at helping me understand. I really hope to learn more from you. I wish you could be my tutor. Thanks for the videos…

[@ThePongles]

Hah, finally a video where I actually did the thing before you made a video on it!

[@Parkhill57]

pfSense firewall has a CA and I generate my root and intermediate that way. Then export them to my clients.

[@gordslater]

Respect My Authoritah!

[@AB-yt4hd]

I am wondering if we could use the Mooltipass instead of the yubikey...

[@nkachur]

Flash storage has a shelf life of approximately 1 year without being rewritten

[@haonnoah]

Excellent video! Makes me thing... Maybe make a bootable USB (encrypted; maybe readonly) for the Dell WISE thin client. Easy migration etc. Have some config to write logs (syslog) to a common server, or use Kafka to keep a ledger and records of CRL and CT data.

[@apalrdsadventures]

There are a ton of options for how to better secure the CA, but in general key rotation is the most effective. This is why it issues 24 hour certs over ACME, to minimize the fallout if a private key is compromised. -On a system with TPM2 you can secure boot your way to a validated kernel and initrd (either distro signed, or a specific image if you want, which means software updates will cause secure boot to fail), and from there you can mount the root partition (potentially encrypted via TPM also) to store the CA database (by default in Smallstep it's a BadgerDB key-value store). None of this protects your CA directly since the system is still decrypted while running (other than the Yubikey), but rebooting regularly at least means the kernel and initrd get re-validated. -If you keep an entirely airgapped machine with the root private key, you could re-issue Yubikeys regularly and physically change them out on the intermediate CAs. If you do this monthly, at worst the attacker can issue new certs valid for roughly a month. This would involve physically swapping out the yubikey monthly if you want to keep the root machine airgapped, or only booting it up every month to remotely sign the new intermediate keys. -Client certs (especially hardware-bound client certs) would really prefer a longer lifecycle than server certs since clients can be off-network for some period of time, so this is a consideration when you think of rekeying intervals, especially if clients need a still-valid client cert to renew.

[@WolframWebers]

Awesome video. I juts wonder: Couldn't the root cert be provisioned another way? Like via DNSSEC and and DANE?

[@apalrdsadventures]

I'm not aware of any web browsers that support DANE in 2024, so it's kinda irrelevant now. If you use a Mobile Device Management (MDM) system such as Intune it can provision the root cert to devices for you. Otherwise you have to do it manually.

[@WolframWebers]

@@apalrdsadventures Thanks. Was looking for an alternative to just Intune (Am not a fan of Microsofts stuff). And who has money for an E5 or F5 license for a home lab?

[@apalrdsadventures]

There are other MDMs, but they all cost $.

[@noext7001]

you still have to install root pem to the client

[@pin-pw1234]

I see that you need to tell caddy (a webserver?) to use step-ca as you CA. I assume that you also would need to tell proxmox and all other web services/webservers to have their certs signed by step-ca (e.g., promox cluster ca would need to be configured to have its cert signed by step-ca) is that correct? Does each webservice / webserver need to be manually configured or is there a way for each machine / VM / container to be configured so that all the certs it sends out via ACME get automatically signed by step-ca?

[@apalrdsadventures]

Most ACME clients default to either Let's Encrypt or require you add the ACME server. So you'll need to set your ACME directory in the config of the ACME client.

[@solilladan7440]

Can we adapt your guide to utilize the step-ca docker image option for yubikey ? I have a plan to migrate my CA & other auth services to a single system in near future, but want to setup the CA server early.

[@apalrdsadventures]

I don't use Docker so that adaption is on you.

[@sanjeev2003]

👌

[@matthiasmartin1975]

You did a great job at imitating IT-Crowd-Roy's appearance, that hairstyle and neckbeard is just perfect.

[@zyghom]

you are the only youtuber I watch, although the quality of your video is... crap ;-) It is probably because the content of your channel is amazing ;-)

[@peterwilbrink718]

@apalrdsadventures please look into your autofocus settings

[@kaspersergej]

Help, I’m stuck step CA!

[@andrewjohnston359]

thumbs up just for the spaceballs reference edit: doh! someone else beat me to it ;)

[@onetimeaccount]

If anyone managed to compile step-ca for arm64, I'd appreciate to learn how you did that. My attempts at step-ca cgo builds fail with 'no such instruction' errors. Looks like building this on a RPi is out for the moment.

[@CrazyMineCuber]

The official guide from step-ca is on a pi. I just followed it and compiled without a problem.

[@ziozzot]

Is it possible to use a smartcard instead of a YubiKey ?

[@stanislavtrifan96]

Should be the same (in this scenario yubikey is treated as Smart card anyway)

[@dubz5149]

Why not put the root CA on a YubiKey? What about YubiHSMs, would those be any better/different for this?

[@apalrdsadventures]

With only one Yubikey in my setup, the goal is to keep the root totally offline (it has an extremely long validity) and periodically bring it out of offline storage to re-sign the Yubikey. This works well at this scale. Of course, the root could still be a Yubikey itself that I just plug in every time I need to re-key. The Yubikey itself is adequate to store a small number of certs/keys in hw and do some operations on them. The YubiHSM can store a lot more keys in hw, provide multiple access with different permissions to each key, supports Microsoft AD CA directly, and has built-in audit logging. So it depends on your needs.

[@dubz5149]

So theoretically, having four yubikeys, one could create their root on one, sign three intermediates (or more if needed of course), and then use as needed? I believe the only caveat I found with using a YubiKey as the root storage is the size limitations. It won't go to 4096 RSA. That might be the main blocker for some wanting that route I guess. IIRC, YubiHSM has the same size restrictions, just more slots and control as you mentioned? I honestly may just look to getting some encrypted drives to store the private key, do all the offline isolated magic as you have, and have flexibility to move the key as I see fit. Good example could be flaws found in YubiKeys and replacements offered. Sure, you have the latest firmware, but zero way to truly upgrade to the patched one. Only replace. Thanks!

[@DawidKellerman]

Hey Dude!I am on a similar path at work .. Considered adding SSH signed keys?

[@apalrdsadventures]

Working on a good way to deploy client TLS certs, then I'll work on SSH certs with the CA

[@DawidKellerman]

@@apalrdsadventures Personality doing the key distribute with cfengine - ce. Bit of a B to learn but now I love it!

[@itsishansharma]

Thank you so much for this, I have one request if you can, please can you make tutorial on virtual networking inside proxmox. I have a laptop which i want to use it as my web server. It has one LAN port. I want to use that LAN port to connect to internet, then i want to have a vm (pfsense) for my networking needs (like dns and firewall), then finally i want my other vm's (like website, storage server, jellyfin etc) to connect to the pfsense server and to internet eventually so that all of it can be accessible over internet anywhere. Can you please make a video on it like a casestudy or can you point to a guide or anything. Please Please please.

[@apalrdsadventures]

I have a tutorial on basic Proxmox networking (vlans, bridges, and bonds) and I’m working on new episodes with backend and SDN networking.

[@itsishansharma]

@@apalrdsadventures i saw that and that really helped me understanding those concepts. I was only using vlans prior to that. But what i am not able to understand is how to fit those concepts for my use case. Can you help me?

[@T3hBeowulf]

In the end, you still have to go through every client device and import a custom root ca though, right? If not, I probably misunderstood something and that's fine. I learn more from uncovering misunderstandings. Copying a self-signed root ca is what I used to do too but device management became a nuisance and I ended up going the reverse proxy / let's encrypt route to make things work.

[@apalrdsadventures]

Yes, you still need to import the root cert on clients. However, you only need to do this once, not trust each service separately.

[@shephusted2714]

yubi used to be open source - ideally people want to use their own open source hw dongles

[@AdrianuX1985]

Have you considered an alternative to Yubikey? An intermediate key added as data to a PNG file. PNG file with any graphic (e.g. cat) and any name (e.g. cat.png) saved to usb flash drive. The file browser treats this PNG file as a typical photo. When debian detects the PID:VID of this usb flash drive it extracts the intermediate key from the PNG file. (udev script) I think you understand what I mean?

[@apalrdsadventures]

The downside to this is you can export the private key from a flash drive, so the extracted key file could be compromised. The Yubikey can't export the private key, only sign in hardware using the key. Smallstep also supports PKCS#11 HSMs including the TPM2 standard, which could be a good option on modern systems. But the Yubikey 5 is the cheapest HSM which Smallstep supports as far as I know.

[@AdrianuX1985]

@@apalrdsadventures If the private key was encrypted with a password generated from a checksum generated from another flash drive serving as the password for that encrypted key, would it still be a bad idea?

[@apalrdsadventures]

Ultimately the key needs to be decrypted to be used, so the information necessary to decrypt it is available to an attacker on the system. Would they figure it out? Maybe, maybe not.

[@AdrianuX1985]

@@apalrdsadventures If, in my example, I lose the second flash drive and do not write down its checksum for myself in a safe place (I would use SHA512) then I will not get to the private key. The attacker would have to figure out that the password is the checksum of the second flash drive.

[@apalrdsadventures]

The attacker would probably start by reading the Smallstep config file and finding where the private key is supposed to be stored, then looking for a key file in that location (presumably decrypted on boot into a RAM disk so Smallstep can use it). Ultimately, Smallstep needs to sign things and by reading the config file you can work backward to find the key / the script that loads the key on-demand.

[@eDoc2020]

I would be much more inclined to run my own CA like this if its root certificate had a flag indicating it was only valid for a certain domain. Unless I've missed some new developments in the PKI system this isn't possible. If a bug results in my CA signing a rogue certificate for Google then suddenly I can no longer trust connections to Google.

[@killer2600]

Google pins their certificates and Chrome won't accept imposter certs for Google domains.

[@eDoc2020]

@@killer2600 I was using Google as a standin for any important website.

[@DamjanDimitrioski]

Can we convince Nginx Proxy Manager to use our CA using ACME to issue new certificates?

[@apalrdsadventures]

It looks like they don't have an option to do that easily in their gui

[@DamjanDimitrioski]

@@apalrdsadventures then I will try in the main config, this looks interesting.

[@autohmae]

17:31 pretty certain the comment on the website is wrong ? "not the public keys", I assume you meant: "not the private keys".

[@autohmae]

21:45 might be a better idea to have the CA HTTPS SSL cert issues by Let's Encrypt or other trusted CA. To make sure what you are downloading is more likely to be correct.

[@apalrdsadventures]

Thanks! Fixed it! The command was correct, just the comment was off.

[@autohmae]

Yeah ! Great to hear you are going to do mutual-TLS ! Great stuff ! Some other tips: set up OCSP and/or CRL as well. Short lived keys is already a good start, but my guess is this won't be practical for the clients...?

[@apalrdsadventures]

The hard thing with CRLs/OCSP is most clients don't actually use them. But for mtls, I can at least force Caddy to query it (I think, I haven't tried). Currently for testing I issued 2 month certs for my test clients, but I'm not a huge fan of that. My *hope* was to use hardware-bound keys on clients to avoid the need for short-lived keys, but testing with Yubikeys it's a bit of a pain (Firefox makes you retype the PIN every few minutes) and Apple doesn't allow their hardware-bound keys to be used aside from with mobile device management apps. So now I need a new plan. For 'on-network' clients a few days cert and automation to renew would probably be fine, but more transient devices might still not be on once a week.

[@autohmae]

@@apalrdsadventures what I've done for now, I used it only for checking the status of the client-certificates for authenticating to the servers. So when I need to revoke a client-certificate (not often): I add the revoked client to the CRL-file signed by the CA and update the CRL on each server by hand. Not pretty, but at least it doesn't depend on an OCSP server being online, etc.

[@Deathbybandaid]

Is it possible to have 2 internal certificate authorities?

[@apalrdsadventures]

Yes, but why? A given service can only use certs issued by one authority. Clients can trust any number of authorities. It’s also possible to issue 2 intermediates from the same root if you want multiple locations or redundancy.

[@Deathbybandaid]

@@apalrdsadventures not required, but one of the self-given challenges for my homelab is building unnecessary redundancies where possible

[@apalrdsadventures]

You can use the single offline root key to generate multiple intermediates for redundancy (on multiple yubikeys / ...), and then host multiple CAs using an intermediate key for each. All of them are signed by the root, so there's one hierarchy, but you can issue leaf certs from any intermediate.

[@damiendye6623]

​@@apalrdsadventuresit's possible to have 2 roots, you send the csr to be signed by different authorities, ones that's done the cert is trusted by both due to them having the same public key. This is how let's encrypt recent rolled over to a new root without needing to resign certs

[@apalrdsadventures]

Let's Encrypt signed their intermediate cert with two roots, ACME clients aren't really designed to issue from two CAs at once. You could do the same and sign the intermediate cert from two roots, but if you are looking for redundancy it would make more sense to replicate at the intermediate level (however you choose to do that)

[@ShaunCurrier]

I dont get the point of this. If someone hacks the server, they may not be able to get the key, but they will still get all the important functions of the key because the yubikey secrets are store plainly on the server. So they you wont be able to trust certs signed by that cert. And unless you have a good way of doing certificate revocation lists, you'll in practice need to untrust the root CA anyway. It could be argued this protects against physical key theft but even then its only if the server's yubikey secrets are encrypted and the server is otherwise locked down (no console). Most home servers don't meet that. So in summary this seems to only inject complexity and unnecessary software into the public key infrastructure.

[@apalrdsadventures]

The root cert is stored offline after it's created, so only the intermediate cert is on the CA system at all. So now we have a shorter lifetime (1yr intermediate vs basically infinite) to begin with, which limits the length of an attack. You can reduce this time down further if you are willing to re-key the yubikey more often. Next, the Yubikey can't export it's private key, so while someone could sign their own certs using the Yubikey, they can't steal the private key itself, so they are limited in how fast they can sign new certs. If they are going through step-ca, it will only issue certs with a ~48hr validity, so that's the limit of what an attacker could get. If they manage to hijack the system entirely, then they could issue certs up to the validity of the intermediate cert on the Yubikey. All of this makes the PKI more secure than keeping the root cert on a server somewhere.

[@ShaunCurrier]

@@apalrdsadventures Thanks for responding. I agree with the benefits you mention but not sure I agree that they are valuable in a homelab. But I can imagine some scenarios that would benefit depending on what threats one is worried about.

[@mikepict9011]

Burn the book if you dont like the answers

[@lilrex2015]

NGL, you can skip the "attention grab" at the start, we know what this channel is and I clicked on the video bc the title had something I want to learn.

[@Gagootron]

I just got around to implementing this myself, and I'm wondering why you didn't generate the intermediate key directly on the yubikey? Instead opting to generate it on the computer and importing it?

[@apalrdsadventures]

It doesn't really matter either way. I was working with OpenSSL so I stuck with the OpenSSL cert generation workflow.