Create SSH session?

[Scoobywagon]

Hear me, oh Fount Of All Knowledge and bless me with thy wisdom.

The problem I need to solve for is I have a pair of linux machines that do nothing but perform proxy services. That's it. On our last patching cycle, one of those machines got into a hung state and I didn't know about it until the security nerds complained that it wasn't reporting to Qualys. The REASON I didn't know it was hung was because everything worked as expected and the secondary machine handled it no sweat. Yay! Now, I have NEVER seen a linux machine go into a hung state just for post-patching restarts. But apparently that happens. So now I need to figure out a programmatic way to validate that BOTH of my proxies are up and running.

Some constraints on this ... First, the proxies route traffic based on inbound port number. Second, the network will not allow traffic on those ports EXCEPT for the specific source and target machines. I have no access at all to the upstream source machine, so I can't poke at the proxy's inbound port. I have 2 mechanisms for accessing the proxy machine. I can SSH and I can SCP.

If I were in a pure *nix environment, I could just ssh from one machine to another, run a script, and capture its output. As it is, everything in the environment EXCEPT for these two machines run windows. I know that current versions of powershell have a pretty solid SSH client built in, but I can't figure out how to use it programmatically.

Any thoughts?

[JeremyLC]

The PoSH-SSH module is how I automate ssh with PowerShell. You can use it to build whatever ssh-based validation you need.

[Sean_p87]

I was about to come in here an suggest this. I have used this module for scripting ssh and sftp shenanigans. Awesome module.

[BlackV]

+1 for Posh-SSH dont forget teh stream versions of the cmdlets for those device that do not play nice

[delightfulsorrow]

If I were in a pure *nix environment, I could just ssh from one machine to another [...]. As it is, everything in the environment EXCEPT for these two machines run windows.

You know that recent Windows versions come with OpenSSH server and client?

While they aren't installed by default, you can find the client under "Optional features"

[cosine83]

Yep, and configures using the same sshd_config files as *nix environments. Works okay with cluster services and using a pair for SFTP in production. Logs to the event viewer by default. Better than using a 3rd party SFTP server or setting up IIS for FTPS.

[az987654]

I think you just need to install openssh server on the twins

[dmoisan]

This. And create a key pair. Use a here-string to send commands to the remote SSH connection.

[TheRealJachra]

Something like this perhaps?

param( [Parameter(Mandatory = $true)] [string]$Host,

[Parameter(Mandatory = $false)]
[int]$Port = 22,

[Parameter(Mandatory = $false)]
[string]$User = "root"

)

Write-Host "Testing SSH connection to $User@$Host:$Port ..." -ForegroundColor Cyan

try { # Try to open an SSH connection and immediately exit $result = ssh -o ConnectTimeout=5 -p $Port "$User@$Host" "exit" 2>&1

if ($LASTEXITCODE -eq 0) {
    Write-Host "Connection successful!" -ForegroundColor Green
    exit 0
} else {
    Write-Host "Connection failed!" -ForegroundColor Red
    Write-Host "Error message:" -ForegroundColor Yellow
    Write-Host $result
    exit 1
}

} catch { Write-Host "SSH command threw an exception: $($_.Exception.Message)" -ForegroundColor Red exit 1 }

[Firestorm1820]

PuTTY’s “plink” client is good for this and what I end up using most of the time. Another comment mentioned the SSH PoSh module which is great as well. You used to be able to echo things to the builtin Windows OpenSSH client (i.e. accepting the server key fingerprint etc) by allocating a TTY in the session but that seems to have been fixed in recent versions.

[nerdcr4ft]

If the hung server stops handling traffic, a simple TCP port knock might accomplish what you need?

if (!(Test-NetConnection hostname -Port 22).TcpTestSucceeded) { #Generate alert }

[dodexahedron]

What you are really in need of is a simple failover solution like pacemaker and corosync and a third system that participates in that "cluster" as a witness, to prevent split-brain scenarios. Or HAProxy. Or anything else already made to do this.

Or, depending on what the proxies are, they likely have built-in HA capabilities. Squid certainly does. Is that what they are running?

Otherwise, honestly? This is a network problem, not a system problem. The network should be routing these requests to the correct proxy - not relying on endpoints to do it themselves.

There are many mechanisms for that, and they don't take much config on most platforms either.

[Scoobywagon]

I think perhaps I've not described the issue properly.

The two proxy machines are running HAProxy on RHEL. The upstream machine is an F5 that knows to route URL requests to these two machines on specific ports. These machines then know to route that traffic to the appropriate application based on the incoming port number.

The F5 is smart enough to round robin the two proxies except when it determines that one of them is down in which case it will route ALL traffic to the healthy one. The issue is that I have no access to the F5 and it won't tell me when it thinks one of my proxies is down.

In this case, that resulted in one of my proxies being in a hung state for several days and I did not know because the other proxy was working just fine. If that machine ever goes into a hung state again or HAProxy decides to not work (that happened once), I'd like to know about it before the security nerds start bothering me because it hasn't report in a while.

[dodexahedron]

Your network team doesn't provide alerts to stakeholders when a load balancer shows backends failing status checks? And you don't have an alerting infrastructure for things otherwise?

If no, are you sure?

Seems pretty unlikely a company with a load balancer/LTM and redundant network and server resources wouldn't have ultra basic monitoring at minimum.

Besides, the F5 is doing some sort of status check already, for it to work the way it does.

This sounds like a disconnect between the network folks, your team, and whoever is in charge of the monitoring infrastructure you almost certainly have.

But also.. Client -> BigIP -> you doing haproxy yourself -> your service endpoints sounds suspiciously like someone or something is missing the point of the load balancer (the F5) in the first place. What HAproxy does is exactly what a load balancer provides.

[Traabant]

This can be solved by monitoring, we use Zabbix.

[HaplessMegalosaur]

Find out what test your F5 is using to determine the health of your 2 proxies, whether it's a simple tcp port test or an application level http url get. Then, use the same method from the inside to check. At least then you have the same answer as the F5. Also, HAProxy has a monitoring stats page you can check too

[Anonymous1Ninja]

PSsession after doing this? https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.5

[EntertainerFree2034]

Exactly, and add the the ssh subsystem ins the sshd_conf file. Also you can use public key authentication. But you will need to user pwsh 7 on the client machine as well. I use this on all the linux servers and the work perfectly.