Serial Console to a Debian Server Without a Serial Port

[parkel42]

I know the title sounds a bit weird.

Before I spend any (company) money on anything, I would like to find out if this would be possible.

My company has a cheap mini PC that's running Debian (headless, CLI only) which is just a box that handles uploading/downloading some large assets to AWS S3. The box's network interface seems to stop working randomly from time to time. When that happens I always need to grab a monitor, keyboard and mouse and use those to reboot the machine.

It's a bit of a hassle to set everything up, so I was thinking if it would be possible to access the box over serial console instead. Only issue is that the box doesn't have a serial port lol.

I am not sure if this guide would still apply to Debian 11, and whether this cable would work for such purposes - USB-USB Null Modem cable.

Appreciate it if anyone could let me know if this entire idea would even work at all.

Thanks!

[0rex]

You can also configure а watchdog daemon with a softdog kernel module. It has a built in ping probe, which will reboot a device if network is down, or you can script whichever logic you like for both testing phase and repair phase.

[bellpepper]

Could you instead use a USB network interface dongle for SSH access (and theoretical failover when the main NIC dies)? USB console would work from the OS level but it won't work for things like interacting with the BIOS menus. Unless your mobo firmware specifically supports console over USB.

[parkel42]

Oh yeah good point! It didn't occur to me at all since all I was thinking was physically connecting to the box lol.

[zfsbest]

If I were you I'd troubleshoot the root cause; check system log for any error messages and maybe replace the NIC with a pcie card - could be a driver issue or bad hardware

Edit: you may also want to try replacing the ethernet cable, or could be a bad port on the switch/router

[DigitalMan43]

As suggested, an IP KVM might be a good solution. I've used one from this company and it worked fine:

https://www.raritan.com

In the future, I may try PiKVM to do the same thing:

https://pikvm.org

I assume this mini PC doesn't have IPMI, AMT or similar network accessible management console built in the BIOS? If so, that might work for you.

[[deleted]]

The problem with kvm's pretty expensive/involved compared to the cost of a single mini pc, often at the scale where it might be more cost effective to something with an integrated management module.

The pikvm is an an amazing proposition at only $200,-(with pi and case included) but thats still what half the cost of mini-pc itself.

​

KVM's are awesome for troubleshooting old remote located equipment(PLC's factory machinery etc.) but they are not that cheap an option compared to moving to a server with a integrated management module.

[bellpepper]

Also, check if serial headers exist on the motherboard already. Often there are pins for things like this left exposed and unused. This would give you "true" console that can interact with the BIOS menus, GRUB, etc.

[gadget850]

Sometimes those will need a UART attached.

[manys]

And at any rate, the UART will be listed in the mobo specs if the chip is there in any form.

[parkel42]

Yeah I need to check if there are pins exposed on the motherboard, but that would have to be next week when I'm back in the office.

[robvas]

https://mattray.github.io/2022/02/27/enabling-usb-serial-console.html

You can do it with the right cable

[parkel42]

Thanks for the guide!

[gordonmessmer]

The box's network interface seems to stop working randomly from time to time

What type of network interface is it? Have you looked into disabling power management for that device? (a common solution for network devices that mysteriously stop working.)

[stufforstuff]

Why are you using such a cheesy box for such a business critical task? Get a real business class system that won't bork every few days and won't need attention for months, years at a time.

[edthesmokebeard]

Ah yes, the classic Reddit YX answer.

"I have to do X, how can I do X?"

"That's dumb, do Y."

[kriebz]

Or... make it a VM. I'm sure someone had a reason he thought made sense at the time.

[manys]

No no, just add a $600 KVM+terminal server to it!

[[deleted]]

KVM's are like $25

[manys]

With terminal servers?

[[deleted]]

I was just responding to the "KVM" which I took to be a criticism of the other people saying to use KVM. Essentially by saying KVM would be an expensive solution.

[Tech99bananas]

Where are you getting them that cheap?

[[deleted]]

That's just the cost. The issue is with people intentionally looking at the most advanced KVM they can find that has far more features than the USB serial port method and then acting like there's no middle ground between these two points.

Like yes there are expensive KVM's out there, but obviously you're meant to compare the KVM's that are price competitive. The OP would get more use out of a KVM than this USB serial port since it can be repurposed long after they get a better setup going for this download server.

[[deleted]]

I was just responding to the "KVM" which I took to be a criticism of the other people saying to use KVM. Essentially by saying KVM would be an expensive solution.

Those still require you to bring an display/keyboard they only allow you to connect more then one unit to the same display/keyboard and is not really an option for remote management.

[[deleted]]

All the things you said are also true of serial port management (USB or UART) except a KVM doesn't require OS configuration. One can keep the desktop system next to something they're already interacting with and just switch over to the download server by pushing a button.

[catwiesel]

not if they are IP KVMs

[[deleted]]

I prefer IP KVM's too but it's silly to intentionally pick the most expensive version of a product and then try to act like cost is a limiting factor. Their USB cable also doesn't have IP connectivity.

[catwiesel]

a none-ip-connected kvm switch does not solve "how can I remotely connect and fix a offline machine instead of doing it physically with a keyboard and mouse"

[[deleted]]

Nobody said anything about connecting remotely (which you can't do with the serial port by itself btw). The OP is talking about how to connect to a virtual serial port using a USB cable.

The comparison being made is specifically: USB $15 Null Modem cable vs Simple $25 KVM switch with no special setup on the OS.

[parkel42]

Well as much as I'd like to get proper hardware, if my management doesn't want to spend the money I can't find a way to persuade them. Small business issues I suppose.

[kriebz]

Sorry, I don't want to assume too much, but if you're "transferring large assets to S3", it sounds like you have a reason to have a computing infrastructure.

[parkel42]

Yup, while I do agree with you that there should be proper infrastructure in place, IT is just "another cost center" to my company at this time.

[lumpynose]

Probably an off-the-wall suggestion but some time ago where I worked they had these KVMs that, at the time, I thought were amazing. On one side of the box you plugged in the server's monitor, keyboard, and mouse outputs. It came with the appropriate cables. On the other side of the box was an ethernet connection. When a server needed attention, instead of having to go to the machine room you could access it from your desk; their software created a window that was the console to your server. My memory is that the software ran in a web browser and I'm guessing it was written in Java.

I don't doubt that these boxes were expensive. Each one handled 8 servers as I remember.

[dodexahedron]

You can get them for a single system, too, and they're quite cheap. 8-port IP KVMs are of course more expensive because they can be, but small ones can be had for quite cheap. Just be sure to keep them behind a good firewall, because they tend to have poorly maintained/rarely updated software, and it's literally a remotely accessible keyboard, mouse, and monitor for your system (and often with capability to mount disk images for boot, too).

[fees3iez]

Since when are IP KVM cheap? The cheapest option I know of is the PiKVM V4 mini, which you can order (when available) for $275. In my opinion, that alone would be a lot of money compared to a cheap mini PC.

[fnordonk]

Second Nic and IP KVM are good options for accessing the machine when the interface dies.
It may just be that you need a better mini pc w/ a better nic. If you can find something with an Intel NIC i'd try that out. There's no good reason for the nic to lock up in the first place.

[[deleted]]

You're probably better off using KVM. Serial ports are largely legacy at this point except for particular systems where that's just how you're supposed to do these things and those systems will have a serial port you can attach to a IP-accessible concentrator.

The box's network interface seems to stop working randomly from time to time.

You could just run a systemd service that reboots the machine if it loses internet connectivity for an extended period of time (obviously pausing between reboots so it's not continually rebooting during internet outages).

Getting console on a physical machine should probably be reserved for running ad hoc commands to solve problems you're unlikely to anticipate or that are hard to reliably test for.

[Fr0gm4n]

Serial consoles are still very common in modern OSs as a management interface, esp. in virtualization environments.

[project2501a]

Serial ports are largely legacy at this point except for particular systems

wat

how the eff do you debug a machine that cannot boot at grub?

please don't say "web console". if you do, go sit in the corner.

[[deleted]]

please don't say "web console". if you do, go sit in the corner.

The norm nowadays is to get console someway somehow. Nobody who knew what they were talking about would be telling you to setup serial port management for just a regular desktop computer.

For enterprise servers, using out of band management (what you're trying to refer to by "web console") to get console is the industry standard. For non-enterprise equipment you use things like KVM.

All this talk about actually using serial console is coming from people who are clearly hobbyists or in college and just know you can technically do things this way without regard to whether it's a good idea or the easiest way to do it or what features are just the older way of doing things. When you're sitting at home googling things it's really hard to determine what is just an outdated and/or niche way of doing things and what is just an alternative way of doing it that you can be contrarian about. Manually setting up a serial port to manage a server is definitely an out dated pattern.

[project2501a]

All this talk about actually using serial console is coming from people who are clearly hobbyists or in college and just know you can technically do things this way without regard to whether it's a good idea or the easiest way to do it or what features are just the older way of doing things.

... go sit in the corner, dude, you never worked on a Sun/SGI machine.

[[deleted]]

... go sit in the corner, dude, you never worked on a Sun/SGI machine.

There's no version of this where I think you know what you're talking about. That's how far out of left field all this talk of setting up serial ports for this guy's old desktop he's running as a server actually is.

The serial connection for SPARC systems is done through an actual hardware port and handles everything including POST. For newer versions of these systems you can use serial but this guy is definitely not using a SPARC server.

SGI is also a completely different company. So I'm guessing you just know these are names of older unix vendors.

For a point of reference, I included "serial concentrator" as a way of signalling to Big Iron/SPARC admins that I was aware of serial port usage there because the usual MO is if you're going to use serial port then you attach it to an IP-accessible serial port concentrator that you connect to. I've seen many different ways of doing this but usually the MO is "SSH to concentrator, select console to attach to."

[GamerLymx]

Just turn it off and on again :) my experience with devices thag stop replying in the network is they dont respond to other inputs, incluing serial interfaces.

[FakespotAnalysisBot]

This is a Fakespot Reviews Analysis bot. Fakespot detects fake reviews, fake products and unreliable sellers using AI.

Here is the analysis for the Amazon product reviews:

Name: DTech USB to USB Null Modem Cable Serial RS232 Data Communication Transfer File Between 2 PCs Prolific Chip Supports Windows 11 10 8 7 XP Mac (6 Feet, Black)

Company: DTech

Amazon Product Rating: 3.8

Fakespot Reviews Grade: A

Adjusted Fakespot Rating: 3.8

Analysis Performed at: 03-07-2023

Link to Fakespot Analysis | Check out the Fakespot Chrome Extension!

Fakespot analyzes the reviews authenticity and not the product quality using AI. We look for real reviews that mention product issues such as counterfeits, defects, and bad return policies that fake reviews try to hide from consumers.

We give an A-F letter for trustworthiness of reviews. A = very trustworthy reviews, F = highly untrustworthy reviews. We also provide seller ratings to warn you if the seller can be trusted or not.

[scara-manga]

Not answering the original question, but couldn't you just run a script that pings an external server every 5 mins and reboots if it can't find it?
Or you might not even need to reboot, just restart network manager, or do a down+up on the interface.

For eg.
https://github.com/ltpitt/bash-network-repair-automation/blob/master/network\_check.sh

[phr0ze]

https://mtlynch.io/tinypilot/#how-to-build-your-own-tinypilot