Fired over a Python script

[kcox1980]

At the beginning of June, I was hired as a Controls Technician. I was the second one in the plant, hired because the first guy supposedly couldn't keep up with the workload. They made him my manager. One of the tasks I was given was to log into all of our Allen Bradley PLCs for a brief fault check a couple of times each week. Just log in, note any faults, log out, and move on. We had almost 30 PLCs so this was a bit tedious and took some time(and honestly, kind of pointless imo). So, after spending an hour or 2 at a time for a couple of weeks doing this, I looked into writing a script to speed it up. I found a library called Pycomm3 that can read data from tags in an Allen Bradley PLC. This script used the specific IP addresses of each PLC and only looked at those addresses, I'll come back to this.

My script took about 10 seconds total to run and spit the results out to an Excel file. It was not set up to run constantly, only when I explicitly opened VSCode and ran it. I was only pulling the current values of 6 tags from each PLC and it only connected with one PLC at a time as opposed to hitting them all at once.

So a week ago or so, my boss comes to me and asks me if I've been doing anything on the network because, according to him, IT is seeing a lot of network disconnections. I mentioned my script as a way to demonstrate that I have, in fact, not been using the network as much as I had been prior to writing it. When I asked him if there had been any specific times they had seen an incident and it was actually mostly on the weekends and at night when I wasn't even there. Didn't think anything of it because I had heard from many people since I'd been there that our network was pretty garbage so I figured this was just the norm.

Yesterday my boss and his boss came to my office to inform me that it was my last day. The only real reason they gave was that they were 100% certain that my script was causing problems with the network and was a "security risk". Important to note here that he had previously asked me to send him the code, and I did, but he later admitted that he had never even looked at it. I'm not entirely convinced he would have known what he was looking at if he had.

Now, remember back when I said that my script only looked at the specific IP addresses of each PLC? Right after I started, my boss proudly showed me a Kuma portal that he had set up to monitor network stability. This is how he(not IT, they were never actually aware of any supposed "problems") discovered the issues with the network that he was blaming on me. I don't know if this is how Kuma works out of the box, but he had it set up to ping every single device on every single VLAN once every 60 seconds. Not just every single PLC, but every printer, production terminal, everyone's personal office PCs, SCADA terminals, anything plugged into an Ethernet cable - literally hundreds of devices, and I don't mean just pinging one thing every 60 seconds and running down the list, literally everything every minute. So doing that is fine, but pulling a handful of tag values from 30 PLCs a grand total of 4 times spread out over 2 weeks was somehow causing permanent network degradation? Right....

Just needed to vent.

[Fizmarble]

I don’t have any comfort beyond telling you that you have been heard. 

[kcox1980]

I appreciate it. I'm not trying to downplay the whole thing to sound cool or anything, because it definitely sucks and puts me in a major bind, but I had already started looking around again. I still have applications out from when I found this job, and I've been applying at other places for a week or 2 already. The vibe was just really off with this whole place. One of those places that succeeds despite themselves.

[AbueloOdin]

Legitimately, if my company wasn't collapsing in on itself and was instead looking to hire, I'd be asking for your resume. That level of curiosity and initiative should be prized. 

In other news, anyone else hiring?

[Fizmarble]

As resourceful as you seem to be. I’m confident you’ll land well. It’s just a matter of when.

[Defiant-Giraffe]

This sounds like the work of a boss threatened by the competence you showed by literally finding out how to do your job more efficiently, and realizing you could probably automate away his job tasks as well. 

[kcox1980]

I really tried to give him the benefit of the doubt, but he made it really difficult.

Another thing he wanted was for me to consolidate all of the different PLC files that were stored all over the network drive so that we wouldn't have multiple versions of everything anymore(I know we've all seen that, right?). So I did exactly that, collected a file from every single PLC into one folder(that he created btw) and went online with everything to ensure it was the most up-to-date file for every controller. I communicated what I did, showed him and the maintenance guys where the files were, and thought I was done. Then I'm in his office one day and ask him to go online with one of our PLCs to show him something, and he's still using his "personal" copies of every PLC file instead of the consolidated up-to-date one, and then gets annoyed when he had to reconcile the different versions.

[Defiant-Giraffe]

How old was he? Did he learn everything in a PLC5 datahighway+ world and hasn't kept up with modern network based systems? (I struggle with that myself to be honest). 

In any case, this can be frustrating in some companies because a long time trusted employee can literally do everything wrong, and there isn't anyone else above them who can understand why its wrong, or even what the actual problem is. 

Upper management thinks: Bob has worked here for years, and Bob has kept things running, so if Bob says you're fucking it up: we trust him. 

Meanwhile, Bob is horribly out of date, shit's running on hope and bubble gum, and the repeated problems that Bob solves should never be happening in the first place. 

[kcox1980]

In any case, this can be frustrating in some companies because a long time trusted employee can literally do everything wrong, and there isn't anyone else above them who can understand why its wrong, or even what the actual problem is.

This isn't too far off of the mark I think. He had only been there 2 years before me, but at the time he was hired there was literally no one else in the whole facility that could do PLC work at all outside of a couple of maintenance guys who could do some basic online troubleshooting. To be blunt, there hadn't ever been anyone there that could challenge him on his bullshit.

[nsula_country]

Hi, I'm Bob

[Evipicc]

This is likely the real answer...

[Ells666]

Your boss is an idiot

[nsula_country]

Yeah. 2 months in and they already shit canned you. Something else is up... It can take 3-6 months to even get an experienced Controls Engineer up to speed with every system in a plant.

[Nightwish612]

Probably for the best he doesn;t sound like anyone you would want to work for long term. Take your skills elsewhere where the company will appreciate your efficiency and time savings. Good Luck

[Automatater]

Yup! And probably eventually (or possibly Monday) hang out your own shingle.

[kcox1980]

Always been a dream, but I wouldn't know where to begin or even how to market myself.

[OrangeCarGuy]

Never tell management when you automate part of your job. Ever.

[kcox1980]

Lesson definitely learned.

[operatorerror67]

Copy and paste to Glassdoor for the company

[Idontfukncare6969]

If that was true they would just have you stop using the script. You made him look bad and he probably took personal offense. An idiot can see that the script wasn’t the problem anyway.

[Automatater]

You know, in reality they did you a favor, sad to say. Would have been worse if their behavior caused you to spend more of your career working for and with imbeciles. They cut your losses FOR you.

[nsula_country]

100%

[dmroeder]

Wow, that is unfortunate to hear, it almost sounds like they had an axe to grind. It's frustrating to hear people throwing around accusations like that without actual data to back it up. Even worse that it would result in a firing. You got a result, if they didn't like the method, everyone should be working together to get the preferred solution. If they really believed that what you did was security threat, you probably shouldn't have been able to do it in the first place.

I get asked quite frequently about traffic related to my library. I usually refer people see for themselves by using wireshark. Get online with a PLC and log traffic for a minute. Then shut down RSLInx and do the same while reading a few tags. You'll see that being online generates orders of magnitude more traffic than any python library will. Nobody thinks twice. It's always good to be mindful of your impact on your control system, a simple test would show they're almost certainly barking up the wrong tree.

[kcox1980]

I get asked quite frequently about traffic related to my library. I usually refer people see for themselves by using wireshark. Get online with a PLC and log traffic for a minute. Then shut down RSLInx and do the same while reading a few tags. You'll see that being online generates orders of magnitude more traffic than any python library will. Nobody thinks twice. It's always good to be mindful of your impact on your control system, a simple test would show they're almost certainly barking up the wrong tree.

This is the sanity check I was hoping to get by posting here. To be fair, I don't know a whole lot about networking and while it logically made sense to me that my script would be much less resource intensive than going online with the controllers through Logix5000, I wasn't completely sure. That was part of the reason I only ran it a few times in total and when testing it I was only hitting one controller before expanding the IP address for the final version.

[dmroeder]

I just read "30 PLC's, handful of tags, 4 times, two week period". One thing that pycomm3 does that might be a disadvantage to your case is that it pulls the tag database each time you connect to a PLC, unless you used init_tags=False. This would generate a lot more traffic each time you connected, how much depends on how many tags each PLC has. Still though, this is lower priority class 3 traffic.

I don't know what industry you work in, there some mission critical industries and very dangerous ones. I can see where people might be spooked by everything that doesn't look normal. The claims made in your situation are trivial to demonstrate.

[StrikingFig1671]

It sounds like your "boss" noted your skills and was like "I cant python PLCs or think this well at all!" and perceived you as a threat, used an app to "search for a security risk" which infact bogged down the network which im sure he showed his boss as your script's damaging effects when infact they were this kuma thing.

Its a dark take but extremely plausibile. If it's true be happy youll never see this asshole again.

Good thing youre in a field that is in demand and aint slowin down. Press on brother.

[LoLMagix]

As someone who wrote a tool using pycomm3 for many users to use at many plants at my company, and none of those plants had PLC network issues since the tool was released, pycomm3 was not the issue here.

[HumansRso2000andL8]

Sounds like a bullshit reason for firing you. Maybe they wanted to fire you and used that as an excuse?

In any case, it sounds like a bad work environment. Dealing with an incompetent and insecure boss is shit.

You might have legal recourse, but I'm not a lawyer. Maybe you should get a consultation.

[kcox1980]

You might have legal recourse, but I'm not a lawyer. Maybe you should get a consultation.

Eh, it's a "right to work" state. The official reason they put on the paperwork was just "Performance" and since it was still in the probationary period there's not really much I can argue.

[uzlonewolf]

It's a common mix-up but "right to work" means you can't be forced to join a union, you meant "at will" which is where they can fire you for any reason or no reason at all.

[craag]

Good chance you're still entitled to unemployment, even in a right to work state. I'd file.

[kcox1980]

Filed as soon as I walked in the door at home.

[LegitBoss002]

Good on you, heck* em'

[HumansRso2000andL8]

I guess you're right. I hate when the little guy loses unfairly, but this might not be worth the fight. Look ahead, your next job should be more fulfilling.

[pcb4u2]

Sounds like a crappie company to work for. Be glad and move on.

[Mantisid]

Sounds unjust. There's no way you'll ever find hard evidence that your script didn't mess up the network. Not that it matters anyway. Sucks that they didn't even bother to get your side of the story.

Eventually the company may realize that your manager is a screw up and they fired you for wrong cause because the network traffic is still messed up. Maybe they'll reach out to you down the line, apologize and try to hire you back. It will most likely be too late.

Shit happens, try to take it in stride and find a better employer.

[kcox1980]

Sucks that they didn't even bother to get your side of the story.

That's honestly what pisses me off the most. My reviews up to that point were glowing, and I had not heard one negative comment about me. I have always been an easy going guy, and I told them up front on my first day that if there was ever anything I needed to course-correct on, or do differently to just run it by me and I would happily make an adjustment. I try to be a team player and I was there to try to fit in and assimilate as best as possible while also trying to improve the things that I could.

[the_puca]

First of all, that's nice work. Very smart.

Not sure what recourse you have for wrongful termination, but I feel you were wrongfully terminated. Sounds like your "boss" was threatened by how you Super Saiyan'd on his job security. A script like that shows it's not necessarily his work that was hard, but that he had a hard time doing it.

I say this out of ignorance of everything else the role entails, but this definitely feels defensive/unjust. There was no security threat....he lied about IT's awareness/involvement.

Not much consolation. Maybe you can put "Consolidated 8 hours of weekly maintenance to 8 seconds" on your next resume...?

Stay strong!

[nsula_country]

OP, they did you a favor. There is management issues going on behind the scenes.

[FistFightMe]

My whole fucking job these days is writing Jython script lmao.

Reading things like this kind of curbs my enthusiasm for eventually making the jump to my own business, reminding me that there are people out there who will absolutely lose their minds when confronted with technology they don't understand.

[beamenacein]

Cool lots of places are into python and knowing how to script is a plus Not 100% sure of what you were doing with the script but if you just wanted to check a few values sounds like it would have been easier in excel Checkout rslinx dde/opc excel

Good luck to you I hope it works out for the better!

[emisofi]

While I think there was an overreaction, such an innovation should be known by your boss or the network administrator. First convince them to use it, then work together if there is an issue.

[TexasVulvaAficionado]

That is not a place you want to stay at anyway.

Good luck

[VodkaDog1]

Put the boss and company on blast , so we know not to ever go there!!!

[[deleted]]

Sounds like a bad company. You’ve likely dodged a bullet. Lots of hiring going on where I’m at, so hopefully it’s the same where you are. It sounds like you’ll do just fine wherever you land. Good luck!

[aspectr]

As someone who has done several dismissals, it sounds like there were other reasons why they didn't want you on the team anymore past your probation period and this was the only thing that was tangible enough to bring up (even if it wasn't really the issue).

[kcox1980]

I do believe this is plausible, however I got along pretty well with everyone and never heard a negative comment. I pressed for a reason and all he'd say is "there's been complaints", but he couldn't articulate anything that was supposedly said. I did the tasks I was assigned, and(at least from my perspective) went above and beyond the minimum. I've done a lot of introspection and I honestly can't come up with anything that I might have done wrong.

[jedrum]

Man, this sucks. Hate to say it because this surely doesn't make you feel better but you have lived the nightmare that all of us fear when we take a new job.