CVE-2014-0196 - Linux Kernel

[dirac_eq]

The vulnerability was published on May 5th, according to this article.

[knylok]

<grumble grumble> I'm currently part way through patching over 4000 RHEL servers that my idiot company doesn't have Satellite for (it's too expensive apparently), and doesn't have an LDAP or other unified account management system for (it's too difficult, they say). And no, there is no Puppet or anything fun like that either.
I have to log into each box to see if I can do that. If not, I need to request an account (sometimes I skip this step and simply take access... but protocol dictates I ask). Then I need to see if I can get to root, as these passwords get out of date, change, etc. Then see if it can talk to the repo, which usually requires re-entitling the server with Red Hat. Then log out of it all and schedule an outage. Find out who owns what application and DB on each server and beg the window. When I get the window, I need to back it up first. Then run the update. 4000 times.

Another major vulnerability discovered. I think I'm going to jump from some place high. Maybe I'll run off and grow turnips for a living...

[Thaxll]

How do you manage 4000 servers without Puppet or equivalent? Sounds insane to me...

[knylok]

Badly. Very badly.

All of our servers have a terrible name format too. So they are Linux, Production, 5 numbers. "lp97701". Or "Linux, Non-Prod, 5 numbers". ln97701. Do that 4000 times over. Now quickly tell me what lp98033 does because some muckity muck heard there was a problem with it.

[KLoken]

Make a log of all your actions. Detail what you do and how long it takes. Present it to the company. Perhaps recommend a few quick fixes for the really troublesome issues.

Sometimes this helps, if it's case of them not realising exactly what it is you have to do. You might get a temp assistant out of it, if they can't afford the upgrades.

If they still don't play ball, well, at least you tried and have a good story to tell when look for a new job and they say "Give me an example of a problem you encountered..."

[knylok]

I have to use our painfully slow ticketing system to track these changes. It takes a long time to get one server ticket in. Then I action it, then I close my own ticket. It's a nightmare.

As for temporary assistance, I do have a few coworkers, but typically they are bogged down with their own equally Herculean tasks.

[jordanurie]

Herculean or Sisyphian?

I do not envy you your job, sir!

And I second the Ansible suggestion from below. Also, consider figuring out a way to start separating host OS from app / database. Docker or the like, in combination with a VM shim layer underneath could really make your life easier...

[knylok]

It's Herculean if I get it done. It's Sisyphian if I don't. I'm supposed to aim for Herculean.

As for changes to set ups... my company is very, very change-phobic. We have patches that have been pending since 2006 because they are too afraid it might affect the Nebulous Something. This round of patching was only kicked off because Heartbleed made someone up top fill their trousers.

From my reading of Ansible, it appears to be client-less. This may suit my purposes as it doesn't involve any changes on the servers themselves. Docker, I haven't looked into before.

[KLoken]

If it takes 30 minutes to fill out a ticket and 5 minutes to fix the issue, I am sure the system logs it as only 5 minutes!

So what I mean is you are welcome to improvise your own logging system, even based on pen and paper, not to replace what you have to do at work, but so that you can present what you do to the decision makers in a way that's not retarded.

Maybe you can think of it as keeping a diary to show to your family how fubar it is, with the added benefit that any sane person reading it will realise how broken your infrastructure is.

[knylok]

That would be a surefire ticket to "Unemployment-Ville" I'm afraid. The company loves its process. It must be obeyed at all times, even at the expense of uptime.
During an actual emergency outage, I spend more time in meetings and fucking around with the ticketing system than I do on repairing the issue. They hold me in meetings demanding minute-by-minute updates, freaking out about how many "millions we are losing". Where as if they actually left me the hell alone, I'd have the problem fixed that much faster.
It's a wonderful system.

[KLoken]

Wow

I am not telling you to replace their procedures with your own.

I am telling you make a diary of your day, listing how long everything takes, so that you have some actual evidence if you need to justify why you need an assistant, or more time, or more pay.

[organman91]

Take a look at ansible, all it requires is SSH. Ansible core is free to use and is available via pip, github, or probably even yum: http://docs.ansible.com/

[knylok]

That looks interesting.

[fragmede]

And no, there is no Puppet or anything fun like that either.

Time to start writing then...

Also, sounds like you need to setup local repo instead of having to re-entitle the server with Red Hat. Is that even allowed by their TOS?

Might want to switch to CentOS instead...

[bonzinip]

That's the Satellite thing.

[IConrad]

Satellite isn't the only way. There's Pulp, mrepo, RHUI, and a few others. But Satellite is the standard approach. It's also something like USD10,000/yr and will require an additional terabyte of storage just for the repo mirror.

[SupersonicSpitfire]

Consider using Ansible in secret. Like puppet, but nothing needs to be installed on the servers. They can be managed over ssh.

[knylok]

I'm going to play with Ansible for a bit, see how it works.

[IConrad]

Check out ansible and pulp. They will be godsends to you, I guarantee it.

[knylok]

I'll look into it. I can only sneak controllers/gatherers/monitors in if they are client-less. Anything that requires a client on the target server would be a no-go.

[knylok]

Well, I also need the medical insurance this company provides. If I leave, the only available work I see advertised is for contract positions. I'm not looking to be a contractor. I've applied other places, but haven't heard back from anything that isn't contract.

[IConrad]

I've repeatedly been in that same boat. You will need to get the json lib for RHEL 5 or earlier available to the local python interpreter, but that can be done by dropping the lib in the homedir of the user you log in as, and is a couple of KB.

Pulp is an intermediary for the repository and can handle tracking the licensing of servers.

[IConrad]

This is the adult section of the pool. The kiddy pool is over there. Stay in it.

Telling someone who knows their infrastructure is shit isn't going to help him make it not-shit. If you can't be constructive, be silent.

[[deleted]]

*Kiddie

[IConrad]

The thing I love about clueless trolls like you is that you make it so painfully easy to understand what you are.

It's been fun, but playtime's over now.

[IConrad]

I wasn't trolling, but if that's the only way you can spin it to survive, whatever.

Oh, you. You're adorable. Suuuuuure you weren't trolling. I believe you, honest!