No-GPU Proxmox Server Access over SERIAL

[@JoeLerner-tu5oc]

Thanks!

[@apalrdsadventures]

Glad it was helpful!

[@GeoffSeeley]

@4:55 I had this problem with a testing system (consumer board) and I solved it by creating a udev rule to map the onboard NIC to "eno1" by MAC address. Now it's consistent and similar to how my enterprise (Supermicro) servers behave. Maybe an idea for a future video is to dive into the PCI bus/device enumeration and mapping used by the kernel.

[@giannism7157]

This is a great solution if the server boot up but not so great if you want to reinstall or take the server down for maintenance etc. What I do to avoid buying extra hardware and consuming extra power for 3 node's (my opinion is that high availability is overrated for all of as homelab enthusiasts) is having a Proxmox VM instance installed on my main PC through VMware, restoring the backups only of the absolutely necessary CTs or VMs on this instance (Adguard in my case does the DHCP, home assistant, etc) fix the problem (Proxmox needs less than 30 minutes to install and configure) and restore the new backup back to the main Server. In case of hardware failure I will just have my main computer 24/7 on instead of the Server. I hope this will help someone else. Great video as always.

[@amateurwizard]

The second you posted it I watched it, then went down the rabbit hole. Turns out Star Tech sell a 10 Pin IDC to 9 Pin Serial Male which slots right into a PCI bracket. You can then add a flexible converter for this, to ethernet. It ends up looking cleaner and you also have the additional flexibility of using a regular serial cable. You're probably thinking it's more expensive, I can gladly tell you that the IDC to Serial PCI bracket is only $5 which saves a lot on cost compared to making your own.

[@apalrdsadventures]

For me, I was going down a path of wanting RJ45 to start with (since a lot of networking gear has RJ45, using a single console to usb cable for everything makes sense to me). I did look for PCI-bracket RJ45s but wasn't impressed by them. Long term the panel mount RJ45 is going to be in the front of a rack chassis, eventually, when I build the server it goes in, so it will look better than just hanging out.

[@amateurwizard]

@@apalrdsadventures Gotcha, looking forward to seeing it

[@jonathanschober1032]

Looking forward to the TrueNAS/ZFS to Ceph conversion! Looking to do the same thing due to the same types of limitations of single node failures.

[@apalrdsadventures]

I just got all of the hardware in for a new backup server and a little bit of new server hardware, plan is to build the PBS server standalone (ZFS), then one Proxmox rack mount node with local-zfs and a bunch of unused partition space for Ceph, then work up to the full cluster. So probably a mix of more Ceph testing projects and server-building projects before it all comes together.

[@max_uaminecraft1827]

yes! I did the same thing to my opensense router. The motherboard Im using doesnt have a video out for the integrated gpu of the sandybridge cpu, so I connected a serial port and am suing an old laptop with a serial port and cable to have serial access. This is really handy when the network interfaces get messed up and ssh doesnt work.

[@zero7dd]

I really enjoyed your video, and I've been watching a lot of your content because it's incredibly helpful. I have a suggestion for a new video idea that I haven't seen on youtube in detail. It would be great if you could provide a detailed guide on using Proxmox to set up a self-hosted website that can be accessed by the public.

[@dvdragond2]

This is exactly what I love to see! Actually had to do something exactly like this and your video helped so much!

[@Prime_Tecnologia_Automacao]

I love your videos, always looking at situations we face on a daily basis with solutions that are within our reach, but we don't adopt due to simple ignorance. I placed a Serial Port Bracket Cable on the computer connected to a CROSS Female RS232 Cable to invert the RX and TX pinouts, ... finally I connected my USB - Serial cable to it and to the computer. I can't believe it worked. Thanks.

[@qazwsx000xswzaq]

I am not sure if others have asked already but I have been curious about what those 2D barcodes like shapes are behind your desk lol

[@apalrdsadventures]

They are called AprilTags, they are used to identify individual markers reliably in space at long distances, which is why they are so low resolution. They are normally used to locate the position of things - either tags are all over and moving cameras can identify where they are, or tags are on objects and fixed cameras can find the objects.

[@amateurwizard]

Same here. Looks like it could double as cryptic artwork.

[@apalrdsadventures]

The goal for me was to accurately identify tags from across the room with a cheap ethernet security camera, and it works pretty well. I left them up because they look cool.

[@unijabnx2000]

i used ryzen for my backup machine... i installed a 1 slot gpu (super old has dvi) installed and setup fedora server and then removed the gpu and its been running along just fine. So glad to see how this is done. thx

[@VincentSaelzler]

Wow - awesome! This is exactly what I planned on doing. Going to try it out with the new proxmox installer option of "Install Proxmox VE (Terminal UI, Serial Console)"

[@C0LPAN1C]

I have the same Corsair Case (but in Red vs Yellow). Solid budget ATX case.

[@oliveirajmr]

This is a good idea.. pretty sure there must be some premade cables like this somewhere too. The only disadvantage I can see with this is no bios acesss ? Unfortunately no many motherboards these days have com ports anymore

[@apalrdsadventures]

Roughly half of my boards have a com header, at least the full-size ones (ATX and MATX). And you're right, no BIOS access.

Isn't this dependent on the board's BIOS and how it handles the missing graphics card though? So maybe this works with this specific motherboard but not necessarily with another?

[@apalrdsadventures]

In theory the BIOS might get stuck at POST without a graphics card, although this seems uncommon since headless systems are fairly common

[@peterwilbrink718]

It doesn't work on the MSI 970A-G46 motherboard of my home server, that's for sure.

[@TubeSkaterRudy]

I wonder if it is possible to use the GPU as a passthrough display adapter in a VM with Windows

[@apalrdsadventures]

afaik it depends somewhat on the BIOS and the iGPU on if it can be passed through at all, but at least with a serial terminal you still have the Proxmox console

[@xlaape]

Apalrd created own RJ45/DCE-to-header on motherboard adaptor (from "RJ45 Panel Mount Female Extension"), but, is there equivalent for a USB port for computers without serial port or header? (I'm not looking for a RJ45/DTE-to-USB - no shortage of these). So far, I've only found one, "USB 2.0 to RS-485 RS-422 RS485 RS422 RJ45 RJ-45 Serial Adapter Converter FT232" - any more?

[@apalrdsadventures]

So you'd ideally like RJ45 (DCE side female) to USB header, or USB in general? Linux does support USB serial ports for the kernel tty, so it's just a matter of finding the right connectors.

[@xlaape]

I seek a RJ45-to-USB female/port to connect to Proxmox's USB port (not the USB-to-RJ45 male/connector, connected to my laptop's USB port.)

[@MarkConstable]

My strategy with a Terramaster F2-423 NAS is to use a pair of NVMe devices for the Proxmox host (mirrored ZFS) then install Ceph on a pair of SSDs in the two bays. I've currently got 4 nodes, so if I reboot one node, then the Ceph cluster is still "okay" after a short re-balance. Adding the Ceph OSDs and then CephFS is super easy and works well for shared storage.

[@MarkConstable]

FWIW, my next challenge is to use RGW (Rados Gateway) on the nodes (maybe in a VM/CT) so that I can use Rclones S3 compat layer to back up just about anything to my Ceph cluster.

[@apalrdsadventures]

You should be able to set no out on the cluster to keep the OSDs in the CRUSH map, so it doesn't try to rebalance during the reboot / maintenance process

[@Felix-ve9hs]

I've been managing Cisco Catalyst and Brocade ICX switches (as well as a TP-Link...yikes) over serial ports for years, as well as APC NICs and Sophos firewalls. So how did I never think about managing Proxmox VE Ryzen servers over serial? This is literally the reason I went with Intel Core CPUs instead... :P

[@Felix-ve9hs]

Another way of course is managing Proxmox VE servers over SSH, but if there is a networking problem, serial is absolutely superior :)

[@apalrdsadventures]

Yeah, moving around PCIe cards (like I do) can lead to networking renaming itself, which is frustrating when you don't have out of band access. Or any number of other things that break, but the system still boots.

[@clairerovic]

Have my Proxmox up with ZFS NAS from your previous Tutorial. Though looking for a clean way to manage it

[@apalrdsadventures]

Glad you like it! Are you trying to manage the ZFS parts?

[@clairerovic]

@@apalrdsadventures Hi ☺️ Yes breaking up into various shares etc. Playing with Cockpit to see what I can achieve. Must say luv your deeper dive into your subject matter.

[@apalrdsadventures]

You should be able to manage groups, permissions, and shares from Cockpit. You just need to go back to Proxmox to allocate subvolumes in ZFS, but you can use one of those for multiple shares if you want.

[@copper4eva]

Two questions about the upcoming cluster. First, are you going to build with power efficiency in mind? A presumably 24/7 running multi node system can obviously chew up a lot of watts. So while I don’t think one should go crazy on power savings, it should be considered. Secondly, and this is more of a crazy idea, any interest in net booting the cluster? I only recently came across net booting myself, so might be a dumb idea. Obviously you’d need a source to boot from, so I thought having two nodes mirror each other would work.

[@apalrdsadventures]

Yes to power, no to netbooting the cluster. I'm slowly buying up a wide variety of types of drives (high and low end SATA, M.2 NVME, U.2 NVME, spinning drives) and I'm going to try and figure out how to fit my storage needs into a low power consumption setup. Goal is for the 3-node setup to not exceed the power of the existing 3 single devices it replaces (TrueNAS and Proxmox), but with more storage and failover.

[@copper4eva]

@@apalrdsadventures That's awesome to hear. Not a lot of info online on power consumption between different drives. Hard to find out the difference between say a 2.5 inch and 3.5 inch HDD, just for example. And ya, netbooting was a long shot lol. It was just on my mind. I look forward to the upcoming videos.

[@apalrdsadventures]

I'm probably going to have one or two drives 'represent' each class, not a full breakdown of every drive on the market, but general trends between categories.

[@Mr.Leeroy]

@@apalrdsadventures HDDs are all too close in power consumption simply because motor has to spin. Multiply 12V by current in amps stated on the label, it would be a max figure close to average motor consumption. They are 5-8W avg load each. The most "low-power" ones would probably be modern 2.5" 5400rpm notebook drives e.g. WD blues. But they would be slow & low density compared to modern 3.5" 7200 drives like WD DC Series that does 150-170 MiB/s average which is pretty much higher than older drives max speed. On homelab scale, I wouldn't bother with comparing drives and use whatever you may get reliable at a good price and in good condition. CPU idle consumption is the most considerable difference, especially with multiple sockets. My weapon of choice is LGA1151v1 for now, so I hunt for bargains on used Supermicro boards. They idle at 15-30W (measured from the wall with i7-6700k and 1275v5/v6) depending on model. Not sure if early ZEN compares. And good luck with serial OOB when OS fails to boot up to tty init (:

[@JPEaglesandKatz]

I've been looking for some time now for a solution to a similar problem.. I can't have my GPU in my proxmox ryzen 3600 server because it only has 2 PCIEX slots, both taken, on by my LSI, one by my 10gb network card... I'm going to order those parts and see if I can get this working too!!! Thanks a lot! LOl I need to find other cable brands, stuff .. some of the linked ones are 5x more expensive here (NL)... sillyness... But then again, those princes includes shipping from the US which is unncecesarry ofc since I assume these are also available on this side of the pond.

[@LampJustin]

Did you see the bootloader screen or the bios?

[@apalrdsadventures]

No, that isn't shown on the serial console. As soon as the kernel is started, you start getting serial access.

[@JPEaglesandKatz]

@@apalrdsadventures I think there is a way to redirect grub to also output to the serial port

[@Mecrom]

can you still access the serial console when quorum is lost? my switch has been unreliable and losing quorum on a headless server has been pain...

[@apalrdsadventures]

You should be able to get to SSH without quorum too, just not the web UI

[@casperghst42]

How about a MB with IPMI or vPro Advanced - cost more money but give more options.

[@Froggie92]

couldnt you easily snapshot the rpool? and be back up in minutes? maybe even pxe boot?

[@apalrdsadventures]

Most of the issues I have that need physical access are network related, so the system is doing fine but is inaccessible for some reason

[@onkelfabs6408]

I just wonder why this pc even posts. Mine did not, using an Intel Xeon Processor in a consumer mainboard. Does it matter if the serial port is attached or not?

[@apalrdsadventures]

Does the Xeon / consumer mobo combination actually work with a graphics card? Or ix it a Xeon W (workstation) processor?

[@onkelfabs6408]

@@apalrdsadventures It is not supported according to the Intel ARK data. I am running an E3 1231 v3 on an Intel H81(the cheapest chipset of that time)

[@onkelfabs6408]

Yet the OS starts when I attach a graphics card.

[@apalrdsadventures]

Sounds like a BIOS issue then

[@JPEaglesandKatz]

My mortar max b450 with my ryzen 3600 which is my proxmox server does have a bios option to ignore if no GPU is present and it will boot... I now have to get the cables and stuff from this video or another one and see if it also has full com/serial header on the mb (haven't checked yet) but looking promising....

[@kirksteinklauber260]

awesome video!!! I want to use this method for Out of Band Management of my servers!

[@bradleydiggs]

This is AWESOME. This helped me to get Serial-over-LAN working through Intel AMT and MeshCommander on a vPro desktop running ProxMox. Do you have any recommendations on editing config (text) files when working over serial? My beloved nano doesn't really work well, which makes sense, because I think serial always updates new lines one at a time if I'm not mistaken...unlike being able to move the cursor all around the screen like with a GPU based console... (Oh no, is this a use-case for finally getting comfortable with Vi? Ha!) EDIT: Serial works great for all of the above, it was my client/viewer. Even though MeshCommander is awesome, it's Serial-over-LAN terminal is a little buggy, laggy, or however you want to say it on my system. It wasn't without a lot of work, but I was able to find a Linux program called amtterm to test out another "client" and everything, including nano worked just fine. (amtterm was a lot of work because the one available in the Ubuntu repositories is too old to work (authenticate) with my version of AMT, and the newer available ones had to be built from source and/or patched though somebody did make patched .deb package).

[@marcosoliveira8731]

Another lesson! Thank you!

[@Chris_Cable]

Very nice! I'm trying to think of a way one could integrate that with the PiKVM lol. Would be an interesting challenge.

[@apalrdsadventures]

I'm working on a SSH setup where there's a different DNS name corresponding to each serial port on the Pi, so it can act as a serial console server

[@Chris_Cable]

Oh dang, that sounds like a good solution.

[@ericf3688]

why the comm port? wouldn't the serial port on the ryzen work too? (genuine question, I always assumed serial db9, would work the same as you demonstrated?)

[@apalrdsadventures]

The serial header on the board (2x5 IDC) is designed for a ribbon cable to a DE9 port, but I chose RJ45 since it's arguably a more common serial connector these days in the IT space since they are used for the serial console port on networking gear, and RJ45 to USB serial cables are plentiful and cheap.

[@ericf3688]

@@apalrdsadventures ok, but technologically speaking, it would accomplish the same thing, right? (get access to tty). or are there differences.

[@apalrdsadventures]

Yeah, it's the same, just a different connector

[@ericf3688]

@@apalrdsadventures perfect. thx! ps, love your videos, we're on the same journey, you're just always a few months ahead of me, so your vids are always perfectly fitting to my adventure.

[@konstantinvladimirovich7625]

Спасибо! Очень интересно! Ждём следующих выпусков!

[@apalrdsadventures]

Glad you enjoyed it!

[@autohmae]

You have Remove instead of Remote in the title of your blog post

[@flakblas]

TIL what IDC stands for!

[@ikerstges]

w i c k e d ! ! 👌

[@michaczerwonka8720]

UART over Ethernet is the best console

[@shephusted2714]

this is unneeded boards boot fine headless - if they don't boot to a usb and reinstall or restore a working image

[@apalrdsadventures]

It boots, but if you need to do anything with the OS, you'll want some sort of UI to make emergency changes

[@insu_na]

@@apalrdsadventures True. And PiKVM is really expensive, IPMI even more so.

[@apalrdsadventures]

You get more functionality with PiKVM too (you can reinstall the OS remotely), but this is a solution somewhere between pulling it out of the rack and adding a GPU for every little issue to a fully hands off forever solution.

[@shephusted2714]

@@apalrdsadventures if things go south you should probably do a reinstall - this is largely unneeded as are pikvm - run things virtualized (as you are) and you can reinstall remotely bypassing need for pikvm and serial connections - it is more things to go wrong and added layers of complexity and cost - i mean you can do it but should you when it is just gilding the lily? if the machine isn't booting you won't have a serial connection, you can always use ssh/vnc to bypass need for a gpu

[@insu_na]

@@shephusted2714 if you looked very closely, you can see that the machine booted into pve. That means the machine is the hypervisor. I'd like to see you virtualize the hypervisor on... I dunno, maybe the air in the room? ;)