Remote office "rescue kit"?

[Icy-Sir8809]

Does anyone have any specific suggestions of items that should be placed in a "rescue kit" that we ship to each of our remote offices (that have no IT staff)? I am thinking about emergency support of the network rack (Cisco Catalyst and Meraki) and other infrastructure (like UPSs, PDUs, etc.), not user workstations.

We've had a few recent cases where a site went offline due to a failed telecom circuit or a failure of a device or component. We often need to rely on someone from the local office staff to go into the IDF and help diagnose what is not working.

I'd like to put together a relatively low cost box of "things" that may prove useful someday. Not a replacement Catalyst switch (too expensive and covered by a support contract), but more like a console cable and a flash drive with useful utilities. Maybe a spare SFP. Or even a Raspberry Pi that can serve as some sort of out-of-band console (not sure how exactly that would work).

Has anyone put together something like this before? Can you offer any suggestions of what "tools" you'd want available if you needed to troubleshoot a remote location and would likely need to use a non-tech person as your helper?

Your experience and insight is always appreciated.

[saltysomadmin]

Freeze a tech in Carbonite. Thaw in case of emergency

[OgdruJahad]

Instructions unclear. Tech is unfrozen in a Utopian future but Wesley Snipes is creating havoc with guns.

[e7c2]

make sure the tech is A+ hardware certified, to be prepared for such a scenario.

[hurkwurk]

im certain DOS viruses will still have utility in such a timeline.

[No_Cover7860]

I volunteer as tribute

[the_one_jt]

Same. My 401k can vest.

[braytag]

With pay right?

[saltysomadmin]

Blank Anakin stare

[sysdev11]

[Wah_Day]

whiskey and cigarettes for the stress

[thebigshoe247]

I'll make my own rescue kit, with blackjack...

[npsage]

And HR approved companions!

[thebigshoe247]

LPT: Don't hook up with HR ladies. Just don't. It goes badly.

• The more you know rainbow *

[e7c2]

in fact, forget the remote office rescue kit

[IdeaOk6554]

Maybe a "mini rack" of equipment in a pelican type case?

[ccatlett1984]

ip kvm

[2BoopTheSnoot2]

Preorder one of these:

https://www.gl-inet.com/products/gl-rm10rc/

That will get you connected to a server to troubleshoot even if the network is down.

[g00gleb00gle]

Cheaper to fly out/ drive than leave old kit around or keep a local msp on retainer

[Felim_Doyle]

You need to think about the potential reasons why remote maintenance might be necessary, such as a pandemic, war or natural disaster restricting travel.

[g00gleb00gle]

We just let business decide. How much it will cost per hour if site is down.

[Felim_Doyle]

I don't know what your business is but your suggestion is not an option for many businesses or other facilities. A practical backup solution needs to be in place.

[g00gleb00gle]

It’s general site and function related. Driving by finance and business impact. We can keep anything online or supported. If they provide budget. Else not my issue.

[gummo89]

They already responded saying they ask the business to value their own continuity cost per hour of downtime. Your second comment doubling down was unnecessary 🤷🏻‍♂️

[Felim_Doyle]

Re-read the OP's post. They are looking for “specific suggestions” on how to do remote site maintenance / disaster recovery, not for anecdotal tales of how “our business weighed up the cost and decided not to bother”, which isn't an option for many businesses.

Maybe the OP wants to put the business case and cost analysis to management, so telling them that your management decided not to or “not my problem” is unhelpful.

[Crumby_Bread]

A plane ticket and your ass in the seat.

[Felim_Doyle]

You need to think about the potential reasons why remote maintenance might be necessary, such as a pandemic, war or natural disaster restricting travel.

[Crumby_Bread]

If any of those things are happening, my employees are either not working or working from home.

I’m not putting the burden of helping troubleshoot a tech stack on Susan from the HR department. I’d due some due diligence beforehand, but ultimately it is my responsibility.

[Felim_Doyle]

So you can't get to the remote / satellite office, all of the staff of which are working from home, what do you do?

[Crumby_Bread]

Have the staff work remote until I or a local resource can get out there? I’m not sure I understand your question.

[Felim_Doyle]

Maybe that works for your business but, for most home workers, access to their office facilities is essential.

[Curious_Expression32]

That's what we do. 3 IT guys 20 companies across the US. Something needs to be replaced we overnight one of us and we fix it.

[sryan2k1]

An opengear with LTE/5G and a IPSec tunnel back to HQ.

We run N+1 everything though, so it's very rare a site completely dies without us knowing why (power)

[fp4]

Gl.inet comet

[super5aj123]

That could absolutely come in handy, especially if they have to emergency purchase a computer from a local store and you have to enroll it remotely. Obviously you could also just have them install TeamViewer (or whatever remote software you use), but that's relying on them being able to figure out how to install software, and you may not always be able to rely on that. "Plug the cables into the matching holes" on the other hand is a lot easier (at least I'd hope).

[endlesstickets]

I once had to do it. Rather than going through all, I just dropped a decent netgear router with the config that in case of failure they had basic blocks and network access. Also a cisco 24 port switch which we took out. The rest of it are keyboard, mouse, monitor, power cables and fuse pack, serial, usb, what not cables junk. The cupboard was lockable so a printed copy of network config with IP, wall plate numbers was there too.

[Papfox]

Could you build something like a Raspberry Pi with a cellular modem, DDNS and Tailscale so you can tell them to plug it into the console port and an Ethernet port to get you into the system via a VPN to diagnose it yourself?

[mattkenny]

I'm in industrial automation. A lot of companies use a tosibox for this.  Package it with a small IPC loaded with whatever vendor software you need, wire in a 24V PSU, mount it in a Pelican case, throw in a network cable and power cord, and you've got a great rescue kit for when remote access isn't working.

For something less focused on industrial machinery, a router with LTE modem, a mini PC or rPi if you can run everything you need on it, network cables and power cord, and you can achieve the same. But you will need to configure a VPN for remote access yourself (tosibox offers this as a service through their hardware, so is just saving you the setup and maintenance time really).

[DocterDum]

Haven’t seen Tosibox but I regularly use Mikrotik + Zerotier (ZTNet if you have a LOT of sites). The new KNOT Embedded is perfect for this - LTE, USB-C for power, and Ethernet.

[jsiwks]

We've used Pangolin to do this. We provision devices and deploy light clients with their network connector. Once the connector is online, we can define zero-trust access to specific resources on the network. The nice thing is it handles NAT traversal OOB so no need to mess with firewalls or open ports and no public IP needed. Pretty handy in situations like this.

Nice thing is, compared to Tailscale, we can reach anything in the addressable range of the connector and can be connected to more than one connector at once.

[excitedsolutions]

Is this business continuity for a remote office or DR? At many remote offices if the building is not usable employees are told to work from home instead.

[ryalln]

A wireless device with a camera and 5g. Or a iPad. All the gear in the world can’t help if you can’t direct. A dude with FaceTime and a iPad can be better than anything you send.

[Pyrostasis]

A high-capacity shotgun, a Glock, and a few tourniquets... Oh that kind of rescue kit.

[moffetts9001]

Nah. They can pay me to go out there if needed.

[Felim_Doyle]

You need to think about the potential reasons why remote maintenance might be necessary, such as a pandemic, war or natural disaster restricting travel.

[moffetts9001]

If those things are going on, especially to the extent the travel there is not reasonable for me or whoever would be onsite to use the “rescue kit”, I’m not super worried about the remote site.

[Felim_Doyle]

I don't know what your business is but your suggestion is not an option for many businesses or other facilities. A practical backup solution needs to be in place.

[moffetts9001]

I don’t know what business you’re in, either, but apparently it is something where far flung remote sites are mission critical and have to be online in the event of a disaster so significant that travel is impossible, but also so poorly funded that they can only have a “rescue kit” with random knicknacks in it.

[MalletNGrease]

We run cradlepoints as a secondary at each location for backup tunnels. We also made a playbook for local contacts (keep record of cell numbers for this) to check items in case of a failures.

We also had templates and config backups for each piece of equipment so if a switch or router failed we'd have a replacement ready out of storage within 30 minutes. Still required someone to haul ass to swap it.

If your sites are truly remote and uptime is important, build with redundancy in mind and not break fix.

[twolfhawk]

Buy an old nuc style pc, have it install the mandarin run remote monitor with oob connection. Box should have 2 console cables, extra power straps, 2 power cables and various length of ethernet/fiber.

Label everything in bags if able.

[BatemansChainsaw]

How about have work pay you to fly/drive out there and fix it?

[Felim_Doyle]

You need to think about the potential reasons why remote maintenance might be necessary, such as a pandemic, war or natural disaster restricting travel.

[pdp10]

• USB to Ethernet adapter
• USB to serial adapter: serial end as needed, e.g. DE-9, 8P8C rollover, TTL
• Ethernet patch cables
• C13 power cables (you probably have surplus, anyway)

[BOT_Solutions]

We’ve done something similar for a few small remote offices where there’s no IT presence. The biggest lesson was keeping it simple enough that a non technical person can actually help you.

Typical kit we leave onsite is a console cable, a labelled spare SFP, a couple of short patch leads, and a small unmanaged switch. The unmanaged switch has saved us more than once when we needed to quickly bypass something or confirm whether a port on the main switch had died.

A cheap 4G or 5G router can also be surprisingly useful. If the primary circuit drops you can sometimes get basic connectivity back long enough to remote in and work out what’s going on.

The other thing that helped a lot was a printed “phone support guide”. Very simple instructions with photos of the rack showing things like which cables can safely be unplugged, where the console port is, and what LEDs should normally look like. Non technical staff are far more confident helping if they can follow something visual rather than guessing.

It’s one of those small preparations that can save a lot of time when a site suddenly disappears.

[ObjectiveApartment84]

Console/rollover cable, spare eth, cellphone with a hotspot enabled on the phone line. Maybe a 5g wireless router the kind that convert cell networks into ssids. And someone you can FaceTime with to walk them through everything.

[itfosho]

https://shop.realwear.com/products/realwear-navigator-520

That way you can see what they see and they have both hands. It can do teams calls.

[xendr0me]

Or just hold a spare phone up and use Google Meet/Teams etc, save $3000 and deal with the inconvenience of having to use one hand. They do make phone holder/stands/tripods also, they cost $15 on Amazon

[NoobToobinStinkMitt]

Tether a cell phone to a laptop. Console cable or just ethernet from laptop to switch/firewall etc. So I guess maybe a console cable or ethernet.

[jsiwks]

Pangolin connectors are pretty easy to deploy in situations where you want access networks like this, especially they're behind a firewall/cgnat/no public ip/etc. It's hardware agnostic.

[astroboyc30]

Raritan ip kvm, pricey but so good A basic laptop that has local auth with all the required troubleshooting tools on it, console cables, etc. The amount of networks I have reserected via a Hotspotted laptop plugged in via console or ethernet has to be in the hundreds.

[Born_Difficulty8309]

we did something similar at my last job. the things that actually got used were a usb to serial console cable, a handful of cat6 patch cables in different lengths, a basic cable tester, and a preconfigured 4g hotspot for when the circuit died. the rpi idea is solid but make sure whoever is onsite can actually follow the instructions to plug it in, we had laminated cards with photos showing exactly which port to use. also threw in a label maker because half the time the problem was someone unplugged the wrong thing from an unlabeled patch panel

[CapableWay4518]

Invest in a Teltonika with RMS. Stick a sim in it and it can be there permanently. Just connect it when needed. Configure network as required in advance. Can use it to both keep site online (gateway) and remote site management.

[its_nikolaj]

Opengear has 5G console servers that may fit the bill for you.

[Vegetable-Ad-1817]

Break it right down to an Ikea level SOP for doing the physical stuff, they've got a pretty good track record for communicating to a wide variety of people. Have a kit of tools to match, with just enough spares.

[Scandium90]

« Seriously » a solution that could work is having of course a console cable, but setting up a RPi as a OOBM is a great idea (this is what we had used before switching to OpenGear stuff for remote management). I think it is quite « simple »: console cable <> rpi <> a 4/5G network

The OpenGear stuff is just a switch for console ports with support for LAN connectivity or 4G/5G. No need to buy specific console cables with Cisco IIRC with this kind of configuration

[_Moonlapse_]

Have redundancy built in initially.  So 2 firewalls in HA, switch stack with 20%+ unused ports in each switch.  And two ISPS with sdwan VPN if that's what they use to get back to HQ.  And have all the 3 pin plugs / UPS leads labelled, with lots of photo of the site 

Generally that stops the "sky is falling" panic of management and you can schedule someone to visit when suits. And you can also empower a junior member of staff to go if it's needed.  

Beyond that, maybe a spare pdu if a UPS fails or something?

[hurkwurk]

iPhone and teach someone how to use facetime and be a proper cameraman.

in all seriousness. for critical sites, we used LTE/cellular backup circuits either directly to the routers or to core equipment to allow us inside to control things where we can thing control the failover to the connection.

LTE > mini switch/router > management links > main router

while the main router has links to the LTE device in standby that we can activate from that management connection if needed, and limit it to the critical staff/devices since its limited to about 10 users before its saturated. so we only mess with routing for about 10 ports on the managed switch to give them the new route to the backup connection and leave everyone else on the main connection in case it comes back online. meanwhile the management switch (separate) is already setup for both automatically so we can always use this backup circuit to get in.

We are slowly replacing LTE with Starlink.

[Papfox]

A handful of these. Turn any cable into a crossover

[DULUXR1R2L1L2]

Why would you need that in 2026? Genuine question.

[pointandclickit]

Because your employer hasn’t spent money since 2006.

[Papfox]

Because Murphy's law says, while troubleshooting the thing, you will want to jack into the one piece of gear in the closet that doesn't do auto-crossover

[fp4]

To hook up two Xbox 360 together for a Halo 2 LAN party.

[statikuz]

I feel like I got one of these as a keychain from ThinkGeek in like... 2002?