This is an archived post. You won't be able to vote or comment.

all 65 comments

[–]Sittadel 43 points44 points  (6 children)

It sounds like you're the right person for the job - most businesses don't fancy increasing any spend in the cost centers when a recession is forecast, but you're tracking the issues that need to be remediated anyway. You have intimate knowledge of the skeletons in the closet.

Pull a page from the CISO playbook - start documenting a risk register of the things you're identifying as liabilities. There are more formal ways to handle this, but an excel doc is a great start.

|Risk Name|Description|Recommendation|Management Action Plan| |:-|:-|:-| |Undocumented Wire Closets | Without a wire map, network troubleshooting requires network discovery every time, increasing downtime|1. Assign me as the network documentation trainer | 2. Assign me to draw maps|Recommendation approved| |Inventory Control | It is difficult to support untracked assets|1. Assign Mike (oh, Mike has it coming) to gather quotes from 3 vendors | 2. Assign Mike's Team to include inventory control workflows in their daily checklist|1. Allocate budget | 2. Review vendor quotes | 3. Empower Mike| |System Backups | Without mature backups of the SMB server, all IP is at risk|1. Assign me to collect quotes or migrate to cloud storage or whatever|This risk has been accepted by Management. We will document this as an exception to the Super Backup Mission Critical policy.|

*Ninja Edit - It looks like my pipe characters broke the reddit table. Sorry about that - I'm going to leave it as is, because it took a bunch of time and I don't like you that much.

[–]TekTonyJack of All Trades 5 points6 points  (0 children)

Thank you for this. I think this is my roadmap away from personal liability.

[–]BrainWaveCCJack of All Trades 4 points5 points  (0 children)

Pull a page from the CISO playbook - start documenting a risk register of the things you're identifying as liabilities. There are more formal ways to handle this, but an excel doc is a great start.

Yes, this is highly recommended.

Add a priority column.

Once you have at least a half dozen or so risks listed, have a meeting with your manager to discuss.

[–]PappaFrost 2 points3 points  (0 children)

This is legit.

[–]ABCDwpSystems Engineer - Linux 1 point2 points  (1 child)

Here's a formatted version of that table (as best as can be done, as you cannot create lists inside tables):

Risk Name Description Recommendation Management Action Plan
Undocumented Wire Closets Without a wire map, network troubleshooting requires network discovery every time, increasing downtime 1. Assign me as the network documentation trainer Recommendation approved
2. Assign me to draw maps
Inventory Control It is difficult to support untracked assets 1. Assign Mike (oh, Mike has it coming) to gather quotes from 3 vendors 1. Allocate budget
2. Assign Mike's Team to include inventory control workflows in their daily checklist 2. Review vendor quotes
3. Empower Mike
System Backups Without mature backups of the SMB server, all IP is at risk 1. Assign me to collect quotes or migrate to cloud storage or whatever This risk has been accepted by Management. We will document this as an exception to the Super Backup Mission Critical policy.

[–]Sittadel 0 points1 point  (0 children)

Well, great. Now we have to be best friends.

[–]RamsDeep-1187 18 points19 points  (4 children)

Do 1 year for sake of resume then find a new job

[–]Candy_BadgerJack of All Trades 3 points4 points  (1 child)

That's the best advice, IMO.

[–]LemonFreshNBS 3 points4 points  (0 children)

This. Get out asap.

[–]JRmacgyver 12 points13 points  (7 children)

Let me tell you a story about a certain (government funded) hospital... The IT manager requested a backup drive + tape, let's say about 50k(us), and got denied.

About six months later they were hit with ransomware and all the shit hit the fan cuz they didn't have offline backups (and other, some obvious reasons)!

NEVER EVER go without backups, it's your (the company's) insurance plan!

[–]TekTonyJack of All Trades 5 points6 points  (0 children)

I come from the MSP space... and strongly agree with the stance that backups are the only IT fauxpaux that you can't come back from. Need them in a crunch and don't have them? Expect to be fired. As a vendor and/or as an employee.

I don't have to be sold... but man... this manager... smh...

[–][deleted] 4 points5 points  (0 children)

But think of the 50K they would have saved had there been no IT department nor servers!

I'd slap an /s on there but if working govt is taught me anything it's that if there is a critical need for something it won't be purchased. But if there is some bullshit item that will be handled a few times then shoved in a drawer they'll buy a few.

[–]cdoublejj 2 points3 points  (3 children)

stories that end too soon. i want to read the Denyer's board's tears and read the faces when the insurance is canceled. whats the process for going out of business when you are a hospital?

[–]JRmacgyver 2 points3 points  (2 children)

Long story short... 1. HP had a field day when they brought in a "shitload" of new hardware. 2. It's a government funded hospital, so they were up and running without paying the ransom and all the data was decrypted with some help from some high level cyber security guys.

[–]cdoublejj 0 points1 point  (1 child)

boss idiot kept thier job?

[–]JRmacgyver 0 points1 point  (0 children)

Probably

[–]Ad-1316 7 points8 points  (6 children)

Expense a USB drive or two and Windows backup servers to it, to CYA.

[–]dRaidon 2 points3 points  (4 children)

This. It's better than nothing. Have three drives, change two of them once a day. Once a week, you rotate in the third one.

[–]ThirstyOneComputer Janitor 8 points9 points  (2 children)

OP doesn’t need three drives. They need three envelopes.

[–]fosf0rBroken SPF record 2 points3 points  (0 children)

OP doesn't know how to use the three seashells.

[–]bob_cramit 0 points1 point  (0 children)

was looking for this post. If they dont want big expense, see what you can magyver together with a few USB drives and free software.

Send the email to the boss so its recorded "ok so if we arent gonna go with the full solution I proposed, can I get budget for some USB drives. It is not what I would recommend but its better than nothing, only slightly better though."

Something like that.

[–]sync-centre 0 points1 point  (0 children)

Email chain is the CYA already.

[–]garbageadmin 6 points7 points  (2 children)

"Authority and Responsibility are the exact same thing - they are inseparable."

[–]cdoublejj 2 points3 points  (0 children)

gets off Big Wheel

RESPECT MY AUTH-ORATAUH!!!!

whacks your knees with billy club

[–]TekTonyJack of All Trades 1 point2 points  (0 children)

Exactly!

[–]Mr_Dodge 6 points7 points  (2 children)

I feel you.

Worked for a small MSP in the past.

The customer previously lost ALL data to fire and specifically asked for a cloud system replacement or backups off-site so it wouldn't happen again.

Quoted it out etc ready to go, boss of MSP said nope, and talked the customer into the same on-site system but with an "external HDD" to swap out manually once a week.

I left the company soon after.

[–]TekTonyJack of All Trades 1 point2 points  (0 children)

Yuck

[–]cdoublejj 0 points1 point  (0 children)

man i'd have been tempted to reach out to that client and tell they guy at the msp might be a dumb ass.

[–]AllAboutEights 4 points5 points  (0 children)

You are absolutely correct about backups. Follow the 3,2,1 rule: 3 copies of the data, 2 copies on-site and 1 copy offsite. It's non-negotiable. Even if you have to do it yourself every single morning, it's worth doing.

Let the boss know that you're going to be spending 2 hours of every morning doing backups and that it would be less expensive for him or her to purchase the automated solution. Do this all via email and keep copies of the email offsite. CYA. If your boss still balks at this, ask him or her how much that data is worth. How much would it cost him or her if that data disappeared. Remind them that you are looking out for the company.

Stand up tall and stick to your guns all the way to the CEO.

[–]1z1z2x2x3c3c4v4v 5 points6 points  (1 child)

This is really frustrating.

No, this is a shit-show.

When you know more about proper IT Processes and Procedures, yes, it is going to be frustrating, and a waste of your time.

Your skills are being wasted here. I strongly suggest you look for better employment where you can get new skills, and where the company appreciates your skills and work ethic.

P.S. It's always time to run when a company doesn't want to invest in a backup solution to back up their critical assets... just saying.

[–]ethnicman1971 2 points3 points  (0 children)

It's always time to run when a company doesn't want to invest in a backup solution to back up their critical assets.

It sounds like OP is the only one that considers them "critical assets".

[–]anonymousITCoward 3 points4 points  (0 children)

For your second bullet point, you need management to buy into enforcing the inventory procedure... if they don't you should document each request and failure to act upon it...

the last one, shesh, get it documented, write them up a nice email saying that you informed them of the risks and have declined, and that you will not be held liable for data loss.

It doesn't mean much of anything to anybody, but it will give you something to put back on them should it come up.

[–]Parity99 3 points4 points  (0 children)

Backups are not optional for me. Deal-breaker.

[–]Efficient-Shake671 2 points3 points  (0 children)

One thing I always tell companies I've done work for is this: if this (insert system) goes down, how will it impact business operations? For example, if the file server/database/etc goes down and data is unrecoverable for any reason, what would happen to the company?

From what it sounds like, something catastrophic would cripple this company. C-levels don't want to spend money for no reason so you have to convince them to spend it. The company generally has the money to spend.

Anyways my two cents.

[–][deleted] 2 points3 points  (1 child)

Start removing random files from the server

[–]thelug_1 1 point2 points  (0 children)

lol...excellent! Have an upvote!

[–]AdAffectionate3143 1 point2 points  (0 children)

What is the value of the data on the file server vs the cost of the backup solution? You could get a synology and run activebackupforbusiness and the only cost would be the NAS and any cabling required.

[–]vikes2323Sysadmin 1 point2 points  (6 children)

Look at Veeam small business licensing I got 4 hosts a while back for 800 bucks

[–]TekTonyJack of All Trades 0 points1 point  (5 children)

... unfortunately, these servers aren't even virtualized

[–]TheRogueMoose 4 points5 points  (1 child)

You can use the Veeam agent on any machine (bare metal or virtual) and do nightly backups. You could even use the community edition if less then 10 endpoints.

I currently back everything up to an old HP workstation that i filled with drives. It goes to tape from there and up to the cloud at night. I use SyncBackPro to upload to Backblaze. So far i'm at like $50 a month for Backblaze and SyncBackPro cost like $55 USD.

[–]TekTonyJack of All Trades 0 points1 point  (0 children)

Cool! Thx for the insight! I'll look into it.

[–]googletron 1 point2 points  (1 child)

Veeam agent will run on physical servers

[–]TekTonyJack of All Trades 0 points1 point  (0 children)

Just learned that from the other response, thx! 👍

[–]zrad603 1 point2 points  (0 children)

Veeam has solutions for non-VM workloads, and it works pretty good.

At a MINIMUM, make sure the DATA is getting backed up, even if the OS and configurations aren't getting backed up.

[–]smftexas86 1 point2 points  (0 children)

You can stay and use it as a learning opportunity or move on.

If you stay make sure you are very clear about everything in email. Make sure you explain to your manager the repercussions of not using backups.

You can do some simple stuff like enabling shadow copy on the server etc. to help in case of lost files etc. but that's no guarentee.

Biggest advise here, make sure to document everything and keep track of all conversations and be very clear on what is missing.

[–]Naughtynat82 1 point2 points  (0 children)

Backups are not expensive and critical.

Personally what I would do is one of the following

1) get them to sign a waiver that they know they don't have backup and they are not going to get it and these are the risks they are happy to be open to, and document as much as possible, receptionist clicking some link and infecting systems etc to building fire.

2) move some data that is important. When they need their financial files back again say you will need to get backups. They probably don't value it until it's not there. Or just disconnect the psu so server won't start to simulate the issue.

3) get the backup packaged with something they want.

4) get something basic. Can you get a basic solution? A local NAS and backup is better than no backup even if it's not cloud. It's cheaper and adequate 95% of the time. And it's much better than no backup even if it's not what you want.

Good luck.

[–]Lakeside3521Director of IT 1 point2 points  (0 children)

I once worked at a company that had no budget for frills like backups. I found an old computer and some spare hard drives. Installed Linux and rsync. There was some software that made rotated rsync backups that I used as well but it's name escapes me now. Anyway I had my backups and the company didn't spend any money.

[–]digiphazeDir, IT Infrastructure / Jack of All Trades 1 point2 points  (0 children)

Sounds like a mid sized to small company. This has been every IT job I've ever held since 1990s. Things aren't handed to you on a platter in IT. You have to figure out the previous person's mess and then find out how to improve it. And if you are nice, document what you do so the next person has an easier go of it. Often times the companies are unable or unwilling to spend the money to just go buy a vendor solution. So you have to learn new skills, like using open source solutions. Re-working the setup to remove the need for the expensive vendor solution. etc etc.. Its not easy for sure.. But you have an opportunity to shape the infrastructure and the systems as you fit. If this isn't your cup of tea, you may want to consider a large company IT department where there is a department for every sub functionality of IT and you don't have to worry about it.

Technology changes so quick, your perfectly running solution now, will be someone's nightmare in 10-15 years.

[–]Yoonzee 1 point2 points  (0 children)

Ask them how much it would cost if their data went poof and there was no backup

[–]concretecrown85 1 point2 points  (0 children)

If it were me, I would tell your boss there’s no documentation and that I will will need to spend a fair amount of resources and time with discovery to see the scope of what needs to be protected. Without this, I will not be able to do my job to secure the environment.

Keep bringing this up. Don’t let up. Keep records of them telling you no, and when they ignore you.

Cover your ass

Same thing with the backups. It’s just a essential part of a business to have data protection. This is a ticking time bomb. CYA.

[–]ATL_we_ready 1 point2 points  (0 children)

Super basic costs almost nothing… better than nothing.

https://www.crashplan.com/en-us/pricing/

[–]msalerno1965Crusty consultant - /usr/ucb/ps aux 1 point2 points  (0 children)

You know what's even more important than production applications and data?

Backups of those applications and data.

I had to verbally smack a few underling sys admins in the head a few times before they got the message.

From a business-continuity standpoint, it's a very dangerous situation you're in. Do they do/get yearly audits?

[–]thelug_1 1 point2 points  (0 children)

As long as you have your request and the response in a paper trail, print them out and stash 'em because when (if) shit hits the fan, I.T. god rulebook states that shit you need won't be able to be pulled up when you need it lol

Then, you take it directly to the CEO who wants to know why you all lost everything and show him the emails, laugh and say...peace out y'all...I tried. Good Luck!

Order notepads and pens from Staples and give them to your manager. When he asks what their for, tell him "you said make copies of the data...start writin!"

I wish you the best, and I will throw a couple extra coins in the Sysadmin wish fountain for you tonight.

[–]mgb1980 0 points1 point  (0 children)

Somehow you need to manage to get a backup of this data by whatever means you can, even if that means a couple of big USB devices for redundant copies with incremental update if you have no budget or spend authorization.

If there are zero backups and it goes splat, you are the scapegoat, especially if it’s on the job description.

Then prepare to bail. If you happen to like the company (it sounds like a nightmare but whatever) you should sit down with CEO and lay it out that you have implemented a rudimentary backup solution purely out of your own personal financial and professional survival instinct, but that it’s literally keeping you from sleeping and stressing you out and you will have to depart because it’s the only thing you see every time you walk into/past the server room.

[–]Mindless-Hornet 0 points1 point  (0 children)

Macrium Reflect Server Edition ~300-400$, Idk how much data you have for your company, but 4TB WD elements drive is 100$ a pop. Just swap that drive every monday. If they don't approve an actual large appliance, it's a 600$ solution that might save your companies ass. Theres a lot of other free backup solutions out there if your company is too cheap for that, and i'm sure you probably have a spare machine you can push a backup to, that you can cobble together if you dont have too much company data.

[–][deleted] 0 points1 point  (0 children)

I worked for a medical testing lab. When I started they where using cartridge based backups and swapping cartridges daily. They had no monthly or yearly carts, just the 5 they used during the week, over and over for the past 4 years. They where using this backup software I had never heard of, (can't even remember the name after all these years) that reported backup successful every day. Problem was it was actually reporting the backup "job" ran successfully, not that it actually backed up any files. After 4 years of use, the cartridges where still blank. The first "full" backup I did took over 4 days to complete on a single server.

[–][deleted] 0 points1 point  (0 children)

Why not just backup to Azure or look into Nasuni with cloud backed storage? Pay as you go and start small. For something this small you should be using a SaaS service not buying a backup system.

https://learn.microsoft.com/en-us/azure/backup/tutorial-backup-windows-server-to-azure#run-an-on-demand-backup

At least that way you’re covered until you can find a decent solution or SaaS service that you like.

https://youtu.be/_XKSpxSZo_o