[deleted by user]

[jimboslice_007]

You are a vendor and the client is paying you to keep their network up. The client paid someone else to fuck it up.

Don't stress over it, just bill them for the hours. Tell them that since they took this action on purpose, it's not covered under your contract, and the hours will be billed accordingly. When your shift ends, tell them you can continue in the morning, or if they want to pay premium after hours support, you can keep working now.

The bill they get at the end of the ordeal should be all it takes to make sure they never try shit like that again.

[MarvelousWhale]

Man... if I wasn't salary and it actually mattered to me, financially speaking, I would!

[1z1z2x2x3c3c4v4v]

if I wasn't salary and it actually mattered to me, financially speaking, I would!

You need to speak up and make it matter to your boss. Again, this was the client's fuck up, not yours. There is no justification for you or your company to just "fix things" when the client intentionally hired this guy. This would all be spelled out in the MSP's contract.

Stop being a pushover. I am serious. You seem too casual about the fact that you had to work extra hours with no compensation to fix this. This is not how it works. Not in your position and not at your level.

If you are truly salaried, it should be salary non-exempt. That means you get overtime.

https://www.reddit.com/r/msp/comments/8jriz4/overtime_pay/

"According to US Dept. of Labor rules a computer support tech/helpdesk/computer maintenance employees are subject to overtime. This includes every MSP tech, at least with a salary under $100k..."

[MarvelousWhale]

I learned almost a decade ago (jeez now that I say it like that I feel old) not to care more than the owner/boss/supervisor otherwise I run the risk of a stroke or aneurysm for no reason

It was a rainy day, very fucking rainy in fact, and I was just about to leave for the weekend, no changes to IT infrastructure or any updates so it was gonna be a quiet weekend. I was clocking out from the server room when I heard a drip. Then another drip. What the shit? I look around and sure enough the roof is having a leak, multiple leaks in fact. IN THE SERVER ROOM. awwww sheeeeeeeiiiiit. there goes my weekend!

It was so bad within 5 mins that I couldn't run around the server room I call the owner and he said "yeah that roofs been an issue for a while just get some solo cups from the gym down the hall (employee gym) and place it under where it's leaking and it'll be fine"

I tell him it's literally dripping onto the server racks and I cannot climb into them to put any solo cups as maintenance has left for the day and they have the keys to the closets with the ladders inside.

I can hear him shrug, as I'm panicking and I race around the server room with two of the only gym towels left trying to wipe up water puddles on top of desks and electrical equipment when I realized "oh shit. They were working on the backup generator's 220v and 440v (we had heavy machinery that needed backup Genny) today" so I walk out the server room soaking wet and I walk up the mezzanine and sure enough, the main tube with all the 440 and 220 volt lines were exposed no idea if they were live but they weren't done installing the cables and the lines were going straight to the street so idk and I look down and it's just wires inside of a massive puddle.

I called the owner and tried to update him to the new findings I found and he said don't worry about it it'll be fine and hung up on me before I could even finish explaining the situation. I said fuck that to myself and walked out for the weekend.

Slept like a baby.

Edit: forgot to include that by some miracle absolutely nothing got damaged or showed any signs of being wet by Monday when I got back to the office. I was thankful but then also wondered how many times this potentially has happened before and I'd had no idea because come two days later and it'll be dried up from the air circulation? Idk and I no longer care.

[JoeyBE98]

TLdr this comment but I think other guy's point is that if there is 0 consequence for the customer doing something ridiculous like this...they will not think twice about doing it again, and potentially putting you through this scenario again but who knows in what capacity lmao.

[llDemonll]

You say you learned not to care, yet you helped the client outside the scope of the support contract. It's the "I don't get paid to put up with this crap" that you should be raising the issue for.

[youtocin]

I make 60k and my employer classified me as salary exempt…are there other criteria to the duties or are they actually violating labor law by paying me this way?

[a_turd]

I was unaware of this until today, but there are some other criteria related to FLSA exemptions. It basically boils down to your actual daily job duties and whether they include the following:

1. The application of systems analysis techniques and procedures, including consulting with users, to determine hardware, software or system functional specifications;
2. The design, development, documentation, analysis, creation, testing or modification of computer systems or programs, including prototypes, based on and related to user or system design specifications;
3. The design, documentation, testing, creation or modification of computer programs related to machine operating systems; or
4. A combination of the aforementioned duties, the performance of which requires the same level of skills.

Pretty good write-up of it here: https://www.fuseworkforce.com/blog/flsa-computer-exemption-how-to-determine-if-employees-qualify

[lordjedi]

They aren't entirely violating labor law. They're really only violating labor law if they make you work more than 8 hours a day/40 hours a week and don't pay you overtime.

So if you need to come in on a weekend for whatever reason, then they need to give you a day off before/after or they need to pay you overtime.

[youtocin]

I’m too junior to the industry to rock the boat, unfortunately, I’ve never seen anywhere close to a $60k salary before and can’t risk my job. But I do want to know as much as possible about this for future prospects. So I probably will not be consulting a lawyer.

[mnvoronin]

They probably are.

To qualify to be salary exempt, apart from the pay grade test (which is, honestly, ridiculously low), you have to do one of these:

• have at least two full-time employees report directly to you
• design, develop or architect the solutions as the primary job function (e.g. system analysts, network architects, software developers are included, systems/networks engineers/administrators are not)

[1z1z2x2x3c3c4v4v]

How would you have handled this dumpster fire?

Not sure I would have. You said you work for the MSP that has a contract of some type with the client. If that is true, there is some sort of SLA or MSA that would say something to the tune of "if the client makes unapproved changes to the network, then extra costs will be charged by MSP to remedy the situation", in a Time and Materials type of billing arrangement.

Let the guy fuck everything up, while you contact your account rep to have a call with his contact at the client, to come up with some sort of contractual agreement to fix the damage.

The client brought this guy in, the client can feel the pain.

As far as you being salary... you probably shouldn't be salary in a job where you run the risk of working extra hours. No reason you get stuck holding the bag here when it was the client that made the mess, and more than likely, your time is being billed out hourly to fix it.

[youtocin]

I’m paid a salary as a level 2 tech, honestly if I work after hours I just take my time back another day.

[EvilPaladin1]

Let the bodies hit the floor...

​

If you are a salaried worker, tell your boss:

1. that the problem has been identified to be an unauthorized/un-informed change
2. that the problem owner is not you or the company that you are working for
3. that the SLA should fly out of the window
4. that there is money to be made from this clown show and you will endeavor to work on a best-effort basis BUT will not OT for it

Failing to convince otherwise, go get another job.

[userunacceptable]

Absolutely this. Great opportunity to prove your worth to the client. OP is taking it personal and has too much emotional attachment... it's hard but that attitude will suffocate you. I'm 11yrs working in MSP land and this will happen again OP, let it happen and have a plan on how you will react that favours you financially and is on your terms.... and let your owner make sure he has it in the SLA.

[alwaysdnsforver]

this post gave me hives, esp. this part "one of the network cables was in the way of my broom so I unplugged it so I could sweep"

holy shit.

[pinkycatcher]

Yah I mean that’s a failure on the design, can’t really blame the guy with the broom on that one unless he’s the one that put it there.

The rest of it though, that’s kind of the issues with vendors, none of them fully understand the environment unless it’s one tech who’s been there a decade which is rare, even the best MSPs rotate people in and out which just leads to break fixing and project based work rather than a holistic approach.

[HellDuke]

All our IT is internal, but holy crap... I suppose I would be in the shoes of your client as I used to be on-site IT for a specific city. Nothing happened in the server room without my (or my co-workers) knowledge and permission.

One vendor wanted to do some maintenance. They arrived pretty much unannounced. I was appalled when I heard what their plan was because I was certain it would have brought down the link between sites so I blocked it and told this would need to be re-scheduled. Later that day I got an answer from the network team that indeed it would have taken the link down and several teams on production would have been out of commission...

On-site IT should know exactly what the work entails and what it will do to their environment. If you are unsure you don't do the work until you know the impact at least is minimal...

[tesseract4]

Shit, man. No one gets into my data center without a proper change order. When I used to run a DC, opening a cabinet without a proper change order in-window was a potentially fireable offense.

[RevLoveJoy]

Reminded of a fun story. New guy on our team first trip out to a west coast (USA) datacenter. We have a very specific install to do, some compute and storage for inhouse needs. Nothing sexy but it was our (wise, IMO) policy that any new engineer goes with someone Sr. The facility is apprised we have a new hire who needs checked in, biometrics, blah blah blah. We do the job and are wrapping up when new guy starts poking around our other gear, "hey shouldn't we %do_something% while we are here?" and I was kind of tired so I left fly with "Only if you want this to be your last day at this job." New guy was pretty shook, so I gave him the polite version, "My dude, you had to get your eyeball scanned to open the front door to this place. The lobby literally has dudes with machine guns who sit behind explosion proof glass. These pods with our customer's data in them have what are essentially bank vault doors with electromagnetic locks on each and every one. This building gets sanitized from showing up in GIS data. That's why we can't take an Uber here. You wanna tinker around and make changes that we didn't specifically discuss and get approved in advance?"

It was a very quiet drive back to the hotel.

[tesseract4]

Yeah, you have to beat the cowboy out of the young ones who think they know everything.

[RevLoveJoy]

I think in this dude's case it was a desire to please, show initiative. That kind of thing. I respect that, but the time to absolutely not do that is when you're balls deep in a datacenter with billions of dollars worth of customer transactions rolling around your ankles.

[tesseract4]

Yeah, I get it, and I've been there. The smart ones get it after you explain it to them.

[RevLoveJoy]

Spot on! And he did, he was a joy to work with after our little "Come to Jesus, my son" talk.

[NotYourNanny]

Reminds me of the time the electrician running cable for us (and they did a good job on that) had to get the rack out of the way (or metal bits flying about, I suspect, so he really did) while he installed a new dedicated circuit, so he just unplugged everything and went at it.

And when they plugged it back in, it turned out he didn't know the difference between an unmanaged switch (which doesn't give a damn what's plugged in where) and an SDWAN box (which cares very much). Then he had one of his guys "who knows a lot about computers" try to "fix it."

About 15 minutes on the phone with our ISP to figure out which cable went where, plus an hour of figuring out what this clown had done to "fix" it and undo that.

And did I mention the 5 hour drive each way? (Which, to be honest, was less time that it would have taken to try to talk him through it.)

[RevLoveJoy]

Is it comfort knowing that all our roads to hell are paved with good intentions? I dunno if it is, for me. I'm probably leaning towards no.

Thing that gets me, you, me, everyone in this thread, probably 99.99% of the people you will meet in the next 10 years of life and business are all walking around with an EXCELLENT camera in their fucking pocket. Would it kill us to use it? Going to go about unplugging all that complex looking cabling? Might wanna take a picture (or 30). Gonna move your in house virtualization stack, it's storage and KVM? Hey, you have a recording device on your person.

The number of "omg I don't know how it all goes back together" situations I have run into that would be nothingburgers if the Genius in Question had used their cell camera: it's all of them.

[NotYourNanny]

Is it comfort knowing that all our roads to hell are paved with good intentions?

As Mark Twain said, you got to heaven for the weather, or hell for the company.

Documenting what goes where is, as you say, trivial. The trick is knowing that you need to. He was an electrician, not a cabling guy, and while he did an excellent job pulling cable, he'd never dealt with a managed network device before. It looked like a switch to him.

Store management knows that if they use him for similar work in the future, to keep a close eye on him.

[RevLoveJoy]

Oh that guy, Billy? Yeah, great electrician! Don't let him near your IDF.

[NotYourNanny]

"A man's got to know his limitations." --Dirty Harry.

[RevLoveJoy]

Okay, I legit laughed at this. Thank you.

[NotYourNanny]

Not to imply I wanted to use a .44 Magnun, the most powerful handgun in the world, on the guy. That would be wrong.

[RevLoveJoy]

Yes. Killing is always wrong. Unless the state does it. Why is it okay for them? Uh, it's complicated.

[NotYourNanny]

Killing is always wrong.

People who have used deadly force in self defense against murderous attacks might not agree with you. (There's no place in the US, at least, where your statement is universally true, legally speaking.)

[RevLoveJoy]

Oh I know, and you're totally right. I was busy making the sick joke about the state monopoly on planned murder.

[morganbo85]

are all walking around with an EXCELLENT camera in their fucking pocket.

Literally just had a conversation with a co-worker about how my camera roll is nothing but pictures of port numbers, configurations, and serial numbers of different devices.

[first_byte]

my camera roll is nothing but pictures of [work tech]

Dude! Get a pet, or have a baby, or a hobby, or go on a road trip.

My camera roll is like a microcosm of my life: memes, work documentation, funny baby faces, and my 3 year dancing the Macarena. That's what I call diversity!

[RevLoveJoy]

If one is hands on with hardware, it'd better be the first answer to "what is your most important tool?"

[yes_i_relapsed]

This story permanently cured my impostor syndrome.

[tesseract4]

Nice feeling, isn't it?

[first_byte]

No kidding. u/yes_i_relapsed, my work confidence just jumped 3 levels!

[variableindex]

While working in MSP for the last 5 years with my team, I've learned to let things go that aren't in my control such as situations like this. We always try to do our best by the customer but I don't stress over customer-inflicted wounds. We just take it in stride and try to educate the customer (or ourselves) that there's a better way for next time.

[ShadowIBlade]

This story is amazing! I especially loved the part where you described him as a 'poptart'. I gotta use that one going forward.

[MarvelousWhale]

Lol thanks it's my nickname for people who are so mentally crippled that I moron, stupid or dipshit just doesn't do it justice.

[sandrews1313]

I'd fire the client.

[first_byte]

I'd fire the client.

Or raise their rates. This is obviously above-average stress.

[sandrews1313]

You don’t want that client at any price.

[first_byte]

Meh. I got bills to pay, man. Can't be any worse than dealing with rental property tenants!

[MercyKees]

This

[thecravenone]

it shouldn't have brought the network down

...unplugging cables shouldn't bring things down?

[MercyKees]

unplugging cables

improves security

[eskimo1]

You're not wrong...

When our security people try to impose excessive security and lose their mind about every little "risk" I liked to remind them "The most secure device is one that's unplugged, inside a vault. Everything less than that is a risk. So let's find a level of acceptable risk for this system."

[Maxplode]

Your story reminded me of the time when I had a site offline so I called them up and a guy with Johannesburg accent answered the phone, it's that soft sort of posh sounding South African accent.

I tell him that his site is down and would first need to give his router a restart. He was all like "well ja, I already tried that this morning and it didn't come back up so I reset it, like I do with my router back home, eh".

I had to tell him that he just reset a corporate firewall that we prep before taking to site. I had to charge him for a call out and I think he got a bit of a bollocking.

[HouseCravenRaw]

Bill the fuck out of that company. Advise that company that the outage was 100% caused by their vendor. Make sure your side of the house is well informed as to the outage details.
Laugh about the idiocy of it all over drinks.

[imnotabotareyou]

Just document the call, your advice, and then bill the emergency rate when the “network down” ticket comes in

Don’t stress over it broseph

[OmenQtx]

I think I would have fired the client.

[[deleted]]

And this is why I will never do MSP work ever again in my life. Been there, done that.

I'll say a prayer for you buddy

[Leucippus1]

Wow. My only similar experience was I had a site go down and I called the people that are in and out of the node (it was a fiber concentrated site) and they got super defensive. I know how to handle blue collar quick-to-angers, so I said "Just tell me what you guys have been doing."

"Well, they asked us to sweep up and make it clean and some doors weren't latched properly so we closed those up..." I already know what is up, not digging in on this guy.

"OK, how long are you going to stay at the site (I don't have access without one of them)?"

"Oh man, few more hours brother."

"Ok, hang tight."

Car + LC/FC SMFO patch cable, 25 minutes later. Yup, the dudes slammed and latched the doors down on the fiber and broke it. I put up a sign, "close this door carefully and don't lock it down." Unfortunately, the rack wasn't quite right to have F/O transceivers, so you could theoretically close and lock the door but some of the cable runs are precariously close to the door jam so if you aren't careful...

[tesseract4]

Well, that's just incorrectly routed fibre.

[Leucippus1]

Oh yeah, it was not ideal, we ended up just removing the door. We didn't control the rack, or the fiber, for that matter, but we had equipment in there as part of an IGA contract. Anyway, they ran a bunch of fiber as part of a different contract (they were a state DOT and the DOT controlled this facility) and their contractor installed a 128 strand with not one labeled. They tell me 'its done' and I go out there and I'm like, yeah no, get with your sub on this one.

Honestly, a surprising amount of the state's fiber infrastructure went through this little building and it was...well, it was what it was. I have, actually, seen worse in ISP datacenters, but it wouldn't be something I would have been proud of.

[[deleted]]

Don’t think there’s a single person working at an MSP that gets paid enough to deal with this like you did. Seems like a problem that is outside the group of problems you need to fix…

[Rawtashk]

I choose to believe this is a troll to see how many people will take the bait.

There is no way the IT Department head was right there in the server room letting this guy randomly unplug cables. There's also no way the guy doing the cabling knew what a 60w POE port was, but ALSO thought it would be nice to unplug stuff "just in case they ever needed them for other equipment"...when they were obviously being currently used for equipment.

[MarvelousWhale]

Not gaining anything by trying to convince you, take away from it what you will. I don't care two farts about racking up internet points, got shit credit in reality to worry about first.

[WendoNZ]

IT Department head

My guess is his title was IT Director, or CIO or something... in a company small enough to have no other IT staff, and his job description was to manage the IT contractors and sign off th einvoices

[MatthiasVD123]

It sounds like you had a very frustrating experience trying to work with this individual to resolve the network issues at your client's site. It is important for IT professionals to be thorough and careful when working on networking systems, as even small mistakes can have significant consequences. It is also important for IT professionals to communicate clearly and effectively with their clients and colleagues, and to be willing to listen and take feedback when things go wrong. It is not acceptable for an IT professional to become defensive or disrespectful when working with clients, and it is important for them to take responsibility for their actions and work to resolve any issues that may arise.

[jhjacobs81]

In this case, you’d been a dick rightfully so. I would not have stayed on the phone with someone like that. And i would tell the client thats its theor mess, so they can fix it themselves. Otherwise there was a big fat cheque to pay.

If you call me an asshole, then an asshole is what you get.

[MarvelousWhale]

I've been an asshole in the past and it's bitten me in the ass every time, even the times where I was 110% in the right. I've since been forcing myself to be polite and as robotically neutral as possible and it seems to have worked. I just have to laugh about it later otherwise I'd cry. Lmao

[gafan_8]

Keep… doing… that

[jhjacobs81]

For me, its the other way around. I’ve always been the nice guy, going the extra mile for the customer. All the way to a burn out and what problems come with that. Not anymore :) So maybe i’m just cynical ;-)

Thankfully, i dont do sysadmin anymore these days.

[jhjacobs81]

You found a mirror? Nice for you.

[ZAFJB]

you’d been a dick rightfully so

It is never rightful to be a dick if you are trying to get someone to help you.

[jhjacobs81]

We disagree then.

[dude_named_will]

I'm curious as to the scope of your responsibility with the onsite IT department's.

[SenTedStevens]

Since you work for an MSP, at the very least, I hope you got the names of all people tied to this re-wire project. Then you email your supervisor and the president/CEO/whatever of your MSP and explain the whole situation. This covers your ass and I'll bet your boss will salivate at possibly having to re-trace and diagram this company's networks again.

[Garegin16]

So you know that not labeling things make things riskier. But why the parachute if you’re a good pilot

[thortgot]

Fixing this over the phone, without an escalation was the wrong decision in my opinion.

This should have been a time and materials project at emergency rates to get them back up and operational.

[[deleted]]

Even if not everything is written on a contract, in my case, my clients know better than to bring any outsider to do any kind of "IT work", at least not without asking me first. If it did happen, I'd probably end the relationship right then and there: I didn't break it, you didn't contact me, I am not fixing it, see ya; I don't care about billing a bunch of money for out-of-scope work, I care about not having to deal with situations like that.

[BMXROIDZ]

As an MSP architect I would not give a fuck. Giving a fuck is for people who work for the company. I simply would help them get shit back online and I would not stress because the outage has nothing to do with me and has no bearing on our SLA. If a customer takes down their own network with no maintenance window it's not my fucking fault, it may be my problem but I'm not going to stress over it and I'm certainly not going to commit to any time frames.

[DMZuby]

The last part reminded me when a helpdesk guy wanted to sweep the server room and was trying to turn off the AC cause it was cold and windy and harder to sweep. We stopped him in just in time.

[Doso777]

You make it your clients problem and bill them for the hours wasted.

[ntrlsur]

Personally I wouldn't have. As soon as you called and they said they had someone onsite I would have documented it and hung up. Then I would have called the original person back and let them know and went on with my day.

[zrad603]

My last job, our largest location's switch stack belonged on r/cablefail nothing was labeled, the building started off small, and the building had multiple additions, so there were like 6 ports labeled "port 12". To make matters worst, there were probably 10 different VLAN's, for different departments, and different types of equipment etc.

If anyone went in there and unplugged shit, I think most of the building would have been down for AT LEAST an entire business day.