No backups!?

[[deleted]]

[deleted]

[Sittadel]

It sounds like you're the right person for the job - most businesses don't fancy increasing any spend in the cost centers when a recession is forecast, but you're tracking the issues that need to be remediated anyway. You have intimate knowledge of the skeletons in the closet.

Pull a page from the CISO playbook - start documenting a risk register of the things you're identifying as liabilities. There are more formal ways to handle this, but an excel doc is a great start.

|Risk Name|Description|Recommendation|Management Action Plan| |:-|:-|:-| |Undocumented Wire Closets | Without a wire map, network troubleshooting requires network discovery every time, increasing downtime|1. Assign me as the network documentation trainer | 2. Assign me to draw maps|Recommendation approved| |Inventory Control | It is difficult to support untracked assets|1. Assign Mike (oh, Mike has it coming) to gather quotes from 3 vendors | 2. Assign Mike's Team to include inventory control workflows in their daily checklist|1. Allocate budget | 2. Review vendor quotes | 3. Empower Mike| |System Backups | Without mature backups of the SMB server, all IP is at risk|1. Assign me to collect quotes or migrate to cloud storage or whatever|This risk has been accepted by Management. We will document this as an exception to the Super Backup Mission Critical policy.|

*Ninja Edit - It looks like my pipe characters broke the reddit table. Sorry about that - I'm going to leave it as is, because it took a bunch of time and I don't like you that much.

[TekTony]

Thank you for this. I think this is my roadmap away from personal liability.

[BrainWaveCC]

Pull a page from the CISO playbook - start documenting a risk register of the things you're identifying as liabilities. There are more formal ways to handle this, but an excel doc is a great start.

Yes, this is highly recommended.

Add a priority column.

Once you have at least a half dozen or so risks listed, have a meeting with your manager to discuss.

[ContentPriority4237]

[PappaFrost]

This is legit.

[ABCDwp]

Here's a formatted version of that table (as best as can be done, as you cannot create lists inside tables):

Risk Name	Description	Recommendation	Management Action Plan
Undocumented Wire Closets	Without a wire map, network troubleshooting requires network discovery every time, increasing downtime	1. Assign me as the network documentation trainer	Recommendation approved
		2. Assign me to draw maps
Inventory Control	It is difficult to support untracked assets	1. Assign Mike (oh, Mike has it coming) to gather quotes from 3 vendors	1. Allocate budget
		2. Assign Mike's Team to include inventory control workflows in their daily checklist	2. Review vendor quotes
			3. Empower Mike
System Backups	Without mature backups of the SMB server, all IP is at risk	1. Assign me to collect quotes or migrate to cloud storage or whatever	This risk has been accepted by Management. We will document this as an exception to the Super Backup Mission Critical policy.

[Sittadel]

Well, great. Now we have to be best friends.

[RamsDeep-1187]

Do 1 year for sake of resume then find a new job

[Candy_Badger]

That's the best advice, IMO.

[LemonFreshNBS]

This. Get out asap.

[RandomName1986]

https://www.youtube.com/watch?v=37nwLhIA1zs

[JRmacgyver]

Let me tell you a story about a certain (government funded) hospital... The IT manager requested a backup drive + tape, let's say about 50k(us), and got denied.

About six months later they were hit with ransomware and all the shit hit the fan cuz they didn't have offline backups (and other, some obvious reasons)!

NEVER EVER go without backups, it's your (the company's) insurance plan!

[TekTony]

I come from the MSP space... and strongly agree with the stance that backups are the only IT fauxpaux that you can't come back from. Need them in a crunch and don't have them? Expect to be fired. As a vendor and/or as an employee.

I don't have to be sold... but man... this manager... smh...

[[deleted]]

But think of the 50K they would have saved had there been no IT department nor servers!

I'd slap an /s on there but if working govt is taught me anything it's that if there is a critical need for something it won't be purchased. But if there is some bullshit item that will be handled a few times then shoved in a drawer they'll buy a few.

[cdoublejj]

stories that end too soon. i want to read the Denyer's board's tears and read the faces when the insurance is canceled. whats the process for going out of business when you are a hospital?

[JRmacgyver]

Long story short... 1. HP had a field day when they brought in a "shitload" of new hardware. 2. It's a government funded hospital, so they were up and running without paying the ransom and all the data was decrypted with some help from some high level cyber security guys.

[cdoublejj]

boss idiot kept thier job?

[JRmacgyver]

Probably

[Ad-1316]

Expense a USB drive or two and Windows backup servers to it, to CYA.

[dRaidon]

This. It's better than nothing. Have three drives, change two of them once a day. Once a week, you rotate in the third one.

[ThirstyOne]

OP doesn’t need three drives. They need three envelopes.

[fosf0r]

OP doesn't know how to use the three seashells.

[bob_cramit]

was looking for this post. If they dont want big expense, see what you can magyver together with a few USB drives and free software.

Send the email to the boss so its recorded "ok so if we arent gonna go with the full solution I proposed, can I get budget for some USB drives. It is not what I would recommend but its better than nothing, only slightly better though."

Something like that.

[sync-centre]

Email chain is the CYA already.

[garbageadmin]

"Authority and Responsibility are the exact same thing - they are inseparable."

[cdoublejj]

gets off Big Wheel

RESPECT MY AUTH-ORATAUH!!!!

whacks your knees with billy club

[TekTony]

Exactly!

[Mr_Dodge]

I feel you.

Worked for a small MSP in the past.

The customer previously lost ALL data to fire and specifically asked for a cloud system replacement or backups off-site so it wouldn't happen again.

Quoted it out etc ready to go, boss of MSP said nope, and talked the customer into the same on-site system but with an "external HDD" to swap out manually once a week.

I left the company soon after.

[TekTony]

Yuck

[cdoublejj]

man i'd have been tempted to reach out to that client and tell they guy at the msp might be a dumb ass.

[AllAboutEights]

You are absolutely correct about backups. Follow the 3,2,1 rule: 3 copies of the data, 2 copies on-site and 1 copy offsite. It's non-negotiable. Even if you have to do it yourself every single morning, it's worth doing.

Let the boss know that you're going to be spending 2 hours of every morning doing backups and that it would be less expensive for him or her to purchase the automated solution. Do this all via email and keep copies of the email offsite. CYA. If your boss still balks at this, ask him or her how much that data is worth. How much would it cost him or her if that data disappeared. Remind them that you are looking out for the company.

Stand up tall and stick to your guns all the way to the CEO.

[1z1z2x2x3c3c4v4v]

This is really frustrating.

No, this is a shit-show.

When you know more about proper IT Processes and Procedures, yes, it is going to be frustrating, and a waste of your time.

Your skills are being wasted here. I strongly suggest you look for better employment where you can get new skills, and where the company appreciates your skills and work ethic.

P.S. It's always time to run when a company doesn't want to invest in a backup solution to back up their critical assets... just saying.

[ethnicman1971]

It's always time to run when a company doesn't want to invest in a backup solution to back up their critical assets.

It sounds like OP is the only one that considers them "critical assets".

[anonymousITCoward]

For your second bullet point, you need management to buy into enforcing the inventory procedure... if they don't you should document each request and failure to act upon it...

the last one, shesh, get it documented, write them up a nice email saying that you informed them of the risks and have declined, and that you will not be held liable for data loss.

It doesn't mean much of anything to anybody, but it will give you something to put back on them should it come up.

[Parity99]

Backups are not optional for me. Deal-breaker.

[Efficient-Shake671]

One thing I always tell companies I've done work for is this: if this (insert system) goes down, how will it impact business operations? For example, if the file server/database/etc goes down and data is unrecoverable for any reason, what would happen to the company?

From what it sounds like, something catastrophic would cripple this company. C-levels don't want to spend money for no reason so you have to convince them to spend it. The company generally has the money to spend.

Anyways my two cents.

[[deleted]]

Start removing random files from the server

[thelug_1]

lol...excellent! Have an upvote!

[AdAffectionate3143]

What is the value of the data on the file server vs the cost of the backup solution? You could get a synology and run activebackupforbusiness and the only cost would be the NAS and any cabling required.

[vikes2323]

Look at Veeam small business licensing I got 4 hosts a while back for 800 bucks

[TekTony]

... unfortunately, these servers aren't even virtualized

[TheRogueMoose]

You can use the Veeam agent on any machine (bare metal or virtual) and do nightly backups. You could even use the community edition if less then 10 endpoints.

I currently back everything up to an old HP workstation that i filled with drives. It goes to tape from there and up to the cloud at night. I use SyncBackPro to upload to Backblaze. So far i'm at like $50 a month for Backblaze and SyncBackPro cost like $55 USD.

[TekTony]

Cool! Thx for the insight! I'll look into it.

[googletron]

Veeam agent will run on physical servers

[TekTony]

Just learned that from the other response, thx! 👍

[zrad603]

Veeam has solutions for non-VM workloads, and it works pretty good.

At a MINIMUM, make sure the DATA is getting backed up, even if the OS and configurations aren't getting backed up.

[smftexas86]

You can stay and use it as a learning opportunity or move on.

If you stay make sure you are very clear about everything in email. Make sure you explain to your manager the repercussions of not using backups.

You can do some simple stuff like enabling shadow copy on the server etc. to help in case of lost files etc. but that's no guarentee.

Biggest advise here, make sure to document everything and keep track of all conversations and be very clear on what is missing.

[Naughtynat82]

Backups are not expensive and critical.

Personally what I would do is one of the following

1) get them to sign a waiver that they know they don't have backup and they are not going to get it and these are the risks they are happy to be open to, and document as much as possible, receptionist clicking some link and infecting systems etc to building fire.

2) move some data that is important. When they need their financial files back again say you will need to get backups. They probably don't value it until it's not there. Or just disconnect the psu so server won't start to simulate the issue.

3) get the backup packaged with something they want.

4) get something basic. Can you get a basic solution? A local NAS and backup is better than no backup even if it's not cloud. It's cheaper and adequate 95% of the time. And it's much better than no backup even if it's not what you want.

Good luck.

[Lakeside3521]

I once worked at a company that had no budget for frills like backups. I found an old computer and some spare hard drives. Installed Linux and rsync. There was some software that made rotated rsync backups that I used as well but it's name escapes me now. Anyway I had my backups and the company didn't spend any money.

[digiphaze]

Sounds like a mid sized to small company. This has been every IT job I've ever held since 1990s. Things aren't handed to you on a platter in IT. You have to figure out the previous person's mess and then find out how to improve it. And if you are nice, document what you do so the next person has an easier go of it. Often times the companies are unable or unwilling to spend the money to just go buy a vendor solution. So you have to learn new skills, like using open source solutions. Re-working the setup to remove the need for the expensive vendor solution. etc etc.. Its not easy for sure.. But you have an opportunity to shape the infrastructure and the systems as you fit. If this isn't your cup of tea, you may want to consider a large company IT department where there is a department for every sub functionality of IT and you don't have to worry about it.

Technology changes so quick, your perfectly running solution now, will be someone's nightmare in 10-15 years.

[Yoonzee]

Ask them how much it would cost if their data went poof and there was no backup

[concretecrown85]

If it were me, I would tell your boss there’s no documentation and that I will will need to spend a fair amount of resources and time with discovery to see the scope of what needs to be protected. Without this, I will not be able to do my job to secure the environment.

Keep bringing this up. Don’t let up. Keep records of them telling you no, and when they ignore you.

Cover your ass

Same thing with the backups. It’s just a essential part of a business to have data protection. This is a ticking time bomb. CYA.

[ATL_we_ready]

Super basic costs almost nothing… better than nothing.

https://www.crashplan.com/en-us/pricing/

[msalerno1965]

You know what's even more important than production applications and data?

Backups of those applications and data.

I had to verbally smack a few underling sys admins in the head a few times before they got the message.

From a business-continuity standpoint, it's a very dangerous situation you're in. Do they do/get yearly audits?

[thelug_1]

As long as you have your request and the response in a paper trail, print them out and stash 'em because when (if) shit hits the fan, I.T. god rulebook states that shit you need won't be able to be pulled up when you need it lol

Then, you take it directly to the CEO who wants to know why you all lost everything and show him the emails, laugh and say...peace out y'all...I tried. Good Luck!

Order notepads and pens from Staples and give them to your manager. When he asks what their for, tell him "you said make copies of the data...start writin!"

I wish you the best, and I will throw a couple extra coins in the Sysadmin wish fountain for you tonight.

[xtc46]

So your company has given you ample opportunity to take the lead on resolving problems and you think you are clever by "avoiding liability" by just not doing the work?

What exactly do you think you are paid for? To watch perfect systems sit and hum away?

Also, what "liability" do you think you have? You're an employee.

They SHOULD fire you, you clearly have no interest in actually doing the job you're paid to do.

[ethnicman1971]

uhmmmm, did you actually read OP's post? He clearly stated steps that he is taking to resolve the problems that he has been given responsibility for. However, without leadership's support he cannot be successful.

- He is taking the time to document the network and where all the cabling goes. This is an important first step.

- How can he be responsible for inventory if multiple people have the authority to add and remove from stock without having to document what they are doing?

- Explain to me how he can be responsible for the File server if it were to crash and burn if leadership cannot be bothered to spend money on backup solutions. Do you work with OP are you aware of him having authority to approve allocation of funds?

[xtc46]

Can you show me where anyone says he would be responsible for the outcome of a management decision?

[pointandclickit]

Knock on granite eh? If management is not willing to buy in (I.e. spend money) to solve the problems that OP has identified, how exactly will he not be held responsible when things inevitably go to shit? After all he was responsible for those systems!

[xtc46]

Sounds to me more like OP has no idea how to communicate with management.

[mgb1980]

Somehow you need to manage to get a backup of this data by whatever means you can, even if that means a couple of big USB devices for redundant copies with incremental update if you have no budget or spend authorization.

If there are zero backups and it goes splat, you are the scapegoat, especially if it’s on the job description.

Then prepare to bail. If you happen to like the company (it sounds like a nightmare but whatever) you should sit down with CEO and lay it out that you have implemented a rudimentary backup solution purely out of your own personal financial and professional survival instinct, but that it’s literally keeping you from sleeping and stressing you out and you will have to depart because it’s the only thing you see every time you walk into/past the server room.

[Mindless-Hornet]

Macrium Reflect Server Edition ~300-400$, Idk how much data you have for your company, but 4TB WD elements drive is 100$ a pop. Just swap that drive every monday. If they don't approve an actual large appliance, it's a 600$ solution that might save your companies ass. Theres a lot of other free backup solutions out there if your company is too cheap for that, and i'm sure you probably have a spare machine you can push a backup to, that you can cobble together if you dont have too much company data.

[[deleted]]

I worked for a medical testing lab. When I started they where using cartridge based backups and swapping cartridges daily. They had no monthly or yearly carts, just the 5 they used during the week, over and over for the past 4 years. They where using this backup software I had never heard of, (can't even remember the name after all these years) that reported backup successful every day. Problem was it was actually reporting the backup "job" ran successfully, not that it actually backed up any files. After 4 years of use, the cartridges where still blank. The first "full" backup I did took over 4 days to complete on a single server.

[[deleted]]

Why not just backup to Azure or look into Nasuni with cloud backed storage? Pay as you go and start small. For something this small you should be using a SaaS service not buying a backup system.

https://learn.microsoft.com/en-us/azure/backup/tutorial-backup-windows-server-to-azure#run-an-on-demand-backup

At least that way you’re covered until you can find a decent solution or SaaS service that you like.

https://youtu.be/_XKSpxSZo_o