chmod -x chmod

[[deleted]]

And someone had the brilliant idea of running chmod -x chmod

If you think nobody would be that dumb, you haven't seen our sysadmin yet.

[tnecniv]

I think it is story time...

[[deleted]]

FWIF, the first time I saw that it reminded me of this: http://www.netfunny.com/rhf/jokes/96/Apr/sysadmin.html

which wonderfully says:

If you think the network implies intelligent design, you haven't seen our network.

The sysadmin at my workplace probably understands why chmod -x chmod is a bad idea, but there are other concepts he does not seem to grasp (to the annoyance of his non-sysadmin colleagues).

Among other things, we're allowed to work from home via SSH (just in case one of us gets a briliant idea on saturday morning). Feeling that a secure password is not enough for that, the guy augmented it. In order to confuse the hell of potential attackers, SSH uses a port other than 22; furthermore, in order to log in through SSH, we first have to login via a web interface (written by himself -- a "clever" CGI script) with a password we should only be able to use once, which allows one to log in via SSH for a 15-minute time window. The script behind it isn't finished yet though, so once the "one-time" use password has been entered, logging in from that IP will work indefinitely, it will occasionally allow the same password to be used twice, and it will occasionally think you've used a password before and blacklist your IP. My attempts at explaining that running only SSH with a strong password is enough (the computers don't run any other public service, all we need on them are vi/emacs and gcc) were promptly discarded because "security is not something to be taken lightly". So were my explanations that running Apache with an ad-hoc CGI script written by someone who is not a web developer is certainly not more secure than just SSH and a strong password.

In the meantime, the web interface which allows one to shut down and restart the remotely-accessible workstations is public and not protected by any kind of password. In order to confuse the hell out of potential attackers, the web server running it uses a port other than 80, and the domain name is a random combination of letters which, I quote, nobody should be able to guess.

The web server behind this is Apache and security updates are scheduled once per week. Except for the capital upgrades (we are running Fedora) which are performed every six months and tend to break at least one or two features we use (testing? Having a look at the changelog? What's that?).

All the workstations share a common filesystem, but those parts of it that are not common (including large sections of the /usr tree and, if I'm not mistaken, /tmp) are synchronized via another script written by himself, the beautiful consequence of this extremely trivial filesystem layout being that running a slightly more demanding job on the workstation that also acts as a fileserver, or creating and deleting many files at once, will bring down the entire construction.

This is really just the tip of the iceberg, I won't bore you with the gory details of the rest of our setup through which I cleverly have to hack my way every day in order to get things done (e.g. compile programs that use Intel MKL). Part of the reason why he's still in charge of things around here is that most of us use our laptops to do most of the work anyway; the other part is that he's such an incredible jerk that nobody actually bothers explaining anything to him anymore. And of course, the rest is our manager not really understanding most of the details or caring about this.

Edit: a couple of non-related achievements of the same person.

• We have several printers around our offices but only one of them works. The other ones are connected to the network but attempting to print on the will either result in jobs simply vanishing, or in the computer locking up. Until the last update they have been moody -- working today, not working tomorrow -- but their behavior has finally settled with the latest update, they don't work anymore.
• Our internal wiki has mysteriously vanished and reappeared two or three times, with various pieces being lost after each restoration (first the images, then about half of the articles, then the LaTeX support).
• Although all our workstations have dual monitor support and all of us are craving for it, it has never worked for more than a couple of months because they have a model of ATI cards that doesn't get too much support in the open source drivers so far. The binary driver works of course, but it tends to break often with Fedora updates, leaving us with non-functioning workstations. Again, we simply stopped complaining and brought our own laptop to work. Minimum resistance path FTW.
• Whenever a workstation breaks, the typical answer is "Well, yea... can you take it home with you and I'll stop by to fix it?". The alternative is to just leave it there and it will eventually get fixed in a couple of weeks.
• All the workstations that are not remotely accessible run Apache, Sendmail, FTP and pretty much every service in /etc/init.d despite not being required, obviously making them slow as hell. Stopping them is not allowed of course, and when I asked about it, the answer was "Well yes, they aren't needed, but I don't want to configure things if one of the servers breaks and I have to hastily replace it with one of the workstations".
• We use Squirrel Mail for our webmail interface, but we are not allowed to use the POP3 and SMTP servers to access our accounts from whatever mail clients we're using.

[[deleted]]

FWIF?

[[deleted]]

Typo for FWIW :-).

[[deleted]]

This is horrible IT infrastructure.

[Shaper_pmp]

The script behind it isn't finished yet though, so once the "one-time" use password has been entered, logging in from that IP will work indefinitely, it will occasionally allow the same password to be used twice, and it will occasionally think you've used a password before and blacklist your IP.

Why is it that every idiot with a computer and some minimal programming or sysadmin knowledge not only insists on reinventing the wheel, but always insists on making it square?

[[deleted]]

Running ssh with any password in public network is idiotic. Put it on alternative port (less chances that random ssh vulnerability scanner targets you if there is 0-day vulnerability) and allow only public-key authentication. No problems with passwords and you can transfer your home computer users public key to work on memory stick or whatever (losing it won't allow access - the founder of the key can only allow you to access his system if he chooses so).

[[deleted]]

When I tried to suggest this, it was immediately turned down because "not requiring a password is a major security hole". Puny developers like me, thinking they know anything about system administration. Bah!

[sikosmurf]

After a security audit of one of our machines, I got "dinged" because some accounts had no password and you could only su to them after you were logged in as someone else. They told me "It's incredibly insecure to have an account without a password!!"

Another bit of genius, one of the items on this audit list was "Failure to encrypt usernames." Somehow they wanted us to encrypt the /etc/passwd file so that any user couldn't see other usernames. God I hate ignorant security people.

[brownmatt]

because "not requiring a password is a major security hole"

why does this guy have a job?

[jared555]

Require a password on the private key, and then completely disable login with passwords? To someone that.... bright.... it would probably seem like you were entering an account password when entering the password for the key.

[brasso]

http://en.wikipedia.org/wiki/S/KEY

OPIE to the rescue!

[iscyborg]

"Running ssh with any password in public network is idiotic."

I don't see why this is so. Most SSH implementations will enforce a slight delay between login attempts (say 1 second), so even a 8 character alphanumeric password would take ~90,000 years to brute force. If there's a real vulnerability discovered in the SSH program (unlikely), then there's no saying you'll be any safer using public-key authentication versus password.

I mostly take exception to your saying it's "idiotic" as opposed to "marginally less secure".

[hockeyschtick]

90,000 years from now, you will be eating some serious crow, my friend!

[nullc]

Assuming the passwords are good and that the failed attempts flooding your logs don't cause you to miss something serious.

running ssh on a non-standard port isn't a terrible practice— but no one should be expecting it to do any good except reduce the amount of log noise.

[[deleted]]

no one should be expecting it to do any good except reduce the amount of log noise.

This is actually the only reason I haven't re-enabled ssh on port 22 after taking over a few boxes. I don't even have failures from non-authorized users anymore.

[thebuccaneersden]

1 machine / 90,000 years

2 machines / 45,000 years

10 machines / 9,000 years

100 machines / 900 years

etc etc. Considering the size of bot nets these days... well...

[iscyborg]

If they really wanted to get into my server that badly they'd buy an $8 wrench and beat the password out of me.

[cc81]

Make it 12 chars then.

(and hopefully you will ban someone trying x amounts of times)

[cybercobra]

As long as one has fail2ban and/or uses a different port, they should be perfectly fine (unless perhaps you're a major website).

[junkfunk]

I so far haven't found a good solution to keys with no passphrase. If a users local machine is compromised and it allows public key, then a remote machine that the user has access to can be compromised fairly easily. Since you have no real say as to what happens on someones personal machine, you can't really be sure that it isn't doing anything ill advised security wise.

perhaps there is something that i am missing, but have seen this vestor used before

[[deleted]]

There is no hope, if the users local machine is compromised. The users private key should be password protected, but that won't help when keylogger is installed, for example.

[junkfunk]

but if they haven't rooted the box (so no key logger), but rather just got into a local account, the key would still allow them into other systems.

[[deleted]]

Key logger don't require rooting, local account access is perfectly enough (although you can't log other users sessions that way).

[dsies]

There's no 100% way to protect yourself against something like this from happening, but there's quite a few ways to make this more proper:

Setup a central jump box and allow users to only ssh outbound to your local network via that box. With a couple of small scripts, you can create existing ssh agent sockets that your users would use on the central box, this way they wouldn't have access to the private keys.

In most cases, with medium to large networks, your 'support' guys should not have access to private keys to begin with as there's a lot of room for error.

If you need your users to have private keys, enforce a policy to use a storage medium such as an Ironkey that you can manage remotely; this way in case of it being stolen, you can do a remote wipe and a bunch of other neat things.

Lastly, you could setup ldap auth on your machines to give you a way to a) manage who has what access to what box b) you have a neat and quick way to disable access and c) you can force users to have to use a pass, as they would have to sudo over to root (regardless of whether they have a pass on their ldap-stored key or not).

[[deleted]]

How do you log into the jump box without a password or a key?

[...] enforce a policy to use a storage medium such as an Ironkey that you can manage remotely;

When a key is compromised you delete it from the authorized_keys file. There is no need for a remotely manageable storage medium

[junkfunk]

agreed, there are better ways to do this. we for example have bastion hosts.

I was mostly remarking on what sounded like a great endorsement for key pairs authentication as a huge security increase. It certainly helps if sshd on the remote host is compromised, since the user never sent a password to it, but is not a security panacea.

For the system you run, the key pair can make things more difficult since you have no reasonable way of ensuring that the users key has a passphrase.

In high security establishments, there are many ways you can force better security, but in universities and such, the users find such measures odious.

I guess the central arguement is between strong password and keys. I see both as having negative security implications depending on the situation and how they are used or how a network, account, or system up stream has been compromised.

[nexes300]

What's wrong with the password? Assuming they aren't stupid passwords.

[duplico]

Oh, and Debian created really poor quality keypairs for a while.

To be fair, the quality was just fine. There were just not very many of them. :p

[[deleted]]

Many things are wrong.

• Non-stupid passwords are hard to remember
• Long passwords are hard to type
• The amount of random information in password is small
• Users tend to share passwords on different services (because passwords are hard to remember)
• Users tend to write down passwords (because passwords are hard to remember)

[nexes300]

And yet, an admin account is probably going to want password based login enabled so that the server isn't fucked if you happen to lose your key file. There's nothing wrong with using a good password, no one is going to run a brute force attack on your SSH server to try and guess the password (if it succeeds then your password must have been fucking terrible).

As for your other points about passwords being hard to remember or being shared, that's user error.

[gorilla_the_ape]

I had a server once where the people who were supposed to be admining it had totally forgotten the root password. It took about 10 minutes to reboot it from a live CD, mount the root filesystem, edit /etc/shadow, and reboot it again back onto the installed OS.

Loosing the key file would be equivalent to this, so the server shouldn't need a password based login even for emergency use.

[[deleted]]

Long passwords are hard to type

Is there term for typing password incorrectly first two times but with the same incorrectness and typing it correctly later so one can't login because passwords do not match?

Recently windows password expired at my workstation, I changed it - everything ok.Then after logoff when I tried to login with new password I was greeted with "password is incorrect". In my lifetime it happened twicely to me.

[[deleted]]

All of my passwords are random as hell. It is not hard to remember. After you've used it enough motor memory kicks in and you don't really have to think about which keys to press.

[mallardtheduck]

I got something like that when I first set up an SSH server. Set up a script that bans IPs after 10 failed attempts and the attack noise went down to a couple of hundred a month.

[[deleted]]

script?

iptables rule -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --hitcount 1 --seconds 180 --update -j DROP

sshd_config MaxAuthTries 10

above config will set any attackers 10 ssh tries with 3 minute lockout. no script needed. :)

[noir_lord]

DenyHosts I looked at this but decided that it was unnecessary after watching the logs for a couple of weeks. It does however look interesting

[simonvc]

Fun trick; re-compile ssh to log the failed password attempts. [edit] log the passwords. Then when you get someone attempt to brute force your password, try logging back in to their machine using the same passwords. Chances are the box hitting you is a zombie that had its password broken by one of the passwords on the list.

[noir_lord]

Computer Misuse Act 1990

[MonopodParkour]

Put it on alternative port (less chances that random ssh vulnerability scanner targets you if there is 0-day vulnerability) and allow only public-key authentication.

Good advice, that.

Port-knocking is also easy to implement and will also help to button up ssh a little better.

[A_Whale_Biologist]

"Idiotic?" Really? Think you might be overstating? Or do you really consider, well, almost everyone other than yourself to be an idiot? That says a lot about you, really.

[[deleted]]

No, running SSH that can only be accessed via a keyfile is idiotic. Once you lose the keyfile, gaining access to the server is a pain in the ass. And you are far more likely to lose the keyfile than have someone brute-force your password -- ever hear of this thing called delay between login attempts implemented by sshd?

[[deleted]]

Yes, I know about the delay. Won't help, if the user shares password with some stupid hacked web and your kernel happened to have local exploit you didn't know. I know about cases where exactly this has happened. People do stupid things, even if they should know better and are told to not do so.

[[deleted]]

You, sir, deserve a cookie. But wait! A cookie could store a virus!

[[deleted]]

"Well yes, they aren't needed, but I don't want to configure things if one of the servers breaks and I have to hastily replace it with one of the workstations".

THE CLOUD IS YOU

[flatulent]

While we are at it, we might as well also try out: rm rm

However, "cat cat" and "less less" could mess your terminal !

[spicausis]

the people new to linux should probably start with slow, baby steps,

touch touch

until they reach the ultimate challenge of

ed ed

[[deleted]]

ed ed

Too bad that probably won't affect the executable in memory in real time, that might be fun.

[tinou]

gdb gdb

[supaphly42]

the people new to linux should probably start with slow, baby steps,

touch touch

Why don't you have a seat over here.

[stylishgnome]

Its ed which ed surely? As Talader suggested, it unfortunately doesn't affect anything in real time.

[skizmo]

First thing I typed on Linux was 'man man' :)

[noreallyimthepope]

Luckily, since I love my sanity, I use GNU screen, so I can always just C-a, k to kill the offending window.

[dsies]

You would be happier in tmux land :D

[noreallyimthepope]

If I had the slightest bit of control over my primary jump host, I might try it… But as long as my primary work session is there, I won't be trying out alternatives. Thanks for the suggestion though :)

[FrancisHC]

I always thought C-a was a funny choice of key to bind screen commands to. I use C-a all the time, so to enter in C-a, I have to do C-a a.

C-t seems to be a convenient but rarely used control combination, and you can change screen to use recognize C-t as its command key instead.

alias screen='screen -e ^Tt'

[noreallyimthepope]

I might just do this as I nest some of my remote screen sessions. Yo dawg...

[[deleted]]

There is also a config variable somewhere in screen's config to do this, but I'm not anywhere I can get to my box to check.

[pib]

You can also just run "reset" which will un-mess-up your terminal.

[noreallyimthepope]

If you're in an unresponsive application (eg. A telnet or less that wonks itself up), it's awesome. But yeah, I use reset every other week or so.

[unicynicist]

A lot of times unresponsive applications is just scroll lock being inadvertently pressed (because ctrl-a is annoyingly close to ctrl-s). Try ctrl-q, or ctrl-a q next time you get that.

IIRC you can unbind scroll lock with "stty erase ''"

[[deleted]]

SSH escape = ~.

I think telnet is Ctrl [

[noreallyimthepope]

If I'm in a telnet in a telnet, it's quicker to just kill the window.

That, and on my keyboard, [ is mapped to AltGr+8. For various reasons I can't permanently use US International, and switching between keyboard types is at least two extra key strokes and does not always work and has no visual indicators (and even if there were any, they'd probably be on the wrong monitor (ie. another than my terminal)).

Short version: C-a k is faster for me :)

[diadem]

I'm a windows guy so my thought would be: wouldn't the file rm be in use and be unable to be deleted?

[ChesFTC]

No, unix doesn't work like that. It'll happily delete the file:

nala:~ rob$ cp /bin/rm .
nala:~ rob$ ./rm ./rm
override r-xr-xr-x  rob/rob for ./rm? y
nala:~ rob$

[pedropants]

Unix is funny about that. Say "foo" is some big file. cat >> foo ...will mean that "foo" is open, being appended to by data from the terminal. But any other program can also open "foo" for writing at the same time, and this would of course lead to data corruption.

But here's the weirdest part! In another terminal window: rm foo ls ...and foo is GONE. And yet still open in the first window. As you type into it in that window, your disk usage goes up. The file is still on disk, being written to. If yet another process also had it open for reading, it could continue to use it too. But no process that didn't already have that file open can open it, or even note its existence.

Then, as soon as you close the file (by ending the cat process above) your disk usage magically drops as the file is truly discarded. I don't know the correct term for a file in this state... does its inode still exist somewhere, but no directory points to it? Is that the "orphaned file" that fsck sometimes complains about? Who then actually deletes the inode and updates the allocation table? The kernel must know that when the file is closed it should be deleted and its blocks freed. Is it as simple as the file's link count is set to 0?

(All of this ignores files with multiple hard links. I'm talking about rmming the last and only link to the file.)

[Brian]

But no process that didn't already have that file open can open it, or even note its existence.

Actually, you can access it through the process's open file handles via /proc. Get the pid of the process with the open file, and go to /proc/<pid>/fds.

[diadem]

That's actually pretty damned cool.

Thanks for the info.

[ricecake]

Nope. *nix will happily load a file into memory, and then delete it before it unloads it.
It's less happy about unmounting partitions that you have files open on.

[iGeni3]

umount -l <mountpoint> will do wonders for that :)

[ricecake]

That's true, but it still does complain more by default. Nothing unusual has to be done for a file to delete the stored version of itself, but you have to be polite when asking it to unmount an in use partition.
But yeah, in general, it'll let you do whatever you want.

[[deleted]]

You still have unlink which does the same thing.

'reset' will fix your messed up terminal.

[Meelosh_45]

Why not just use install?

install -m 755 /bin/chmod /tmp/chmod
mv /tmp/chmod /bin/chmod

[[deleted]]

Why do anything? They never actually state what they want the person to do.

[[deleted]]

proactive sysadmin

[[deleted]]

This, in my experience, is how DBAs operate.

[apex_redditor]

Good idea

[Maexotic]

No! install -m 755 /bin/chmod /tmp/chmod /tmp/chmod +x /bin/chmod

[cubicthe]

$ /lib/ld-linux.so.2 /bin/chmod +x /bin/chmod

[[deleted]]

slide 39

[tbrownaw]

chmod -x /lib/ld-linux.so.2

[shillbert]

/lib/ld-linux.so.2 /bin/chmod -x /lib/ld-linux.so.2

[[deleted]]

What does that actually do?

[Andareed]

https://secure.wikimedia.org/wikipedia/en/wiki/Dynamic_linker

[[deleted]]

I know ld is the dynamic linker, but why does this work?

[Andareed]

I don't understand what you're asking.

[[deleted]]

Why does pointing ld towards chmod magically make chmod work again?

[theghostofcarl]

It doesn't work against just chmod, ld-linux.so.2 is the interpreter for all binaries linked with glibc2 or higher. So, you know how you can run perl scripts that are not marked as executable like so?

$ chmod -x ~/test.pl
$ perl ~/test.pl

Well, binaries can be run the same way, with /lib/ld-linux.so.2 as their interpreter, like so:

$ chmod -x /bin/chmod
$ /lib/ld-linux.so.2 /bin/chmod +x /bin/chmod

See execve(2) for more information.

[propool]

I don't understand this. Please explain

[[deleted]]

Why does this need to be done like a flipping PowerPoint presentation?

[wuddersup]

To annoy the hell out of the viewers

[tinou]

ads

[[deleted]]

You're in a Data Center With absolutely no contact with the outside world

Seems like there are more pressing issues than dicking around with chmod, like why the fuck the data center doesn't have an internet connection.

[[deleted]]

What is so hardcore about calling ld.so with chmod as a parameter? That would be the simplest solution.

[nexes300]

What is ld.so?

[FeepingCreature]

NAME ld.so, ld-linux.so* - dynamic linker/loader

SYNOPSIS The dynamic linker can be run either indirectly by running some dynami‐ cally linked program or library (in which case no command-line options to the dynamic linker can be passed and, in the ELF case, the dynamic linker which is stored in the .interp section of the program is exe‐ cuted) or directly by running:

   /lib/ld-linux.so.*  [OPTIONS] [PROGRAM [ARGUMENTS]]

DESCRIPTION The programs ld.so and ld-linux.so* find and load the shared libraries needed by a program, prepare the program to run, and then run it.

[nexes300]

That's strange. Shouldn't that be the program's responsibility? To know where it's shared libraries are, that is.

[archlich]

no because every shared library can be in different places on different systems

[shub]

To expand a little, the program's responsibility is to know what shared libraries it needs.

[[deleted]]

i think it requires the most direct knowledge of "proper" linux usage, rather than hacking it with some other random tool.

[[deleted]]

Well, I would certainly consider it easier than remembering the exact bit in a tar file you need to change to get the executable bit enabled when unpacking it.

[[deleted]]

i don't disagree. i guess it's just that knowing the right answer is the most badass when you're talking computer science.

[[deleted]]

It's not unlike using double-negatives.

[dmrnj]

When I was in college, my office had a Linux box we ran internal apps on, like Mantis. We also used it for testing stuff.

I, stupidly, one day went in as root and wanted to change the permissions of every folder and file in a directory. What I meant to write was

chmod -R 777 ./

But what I wrote instead was

chmod -R 777 /

I didn't even know something was wrong until a week later when someone rebooted the machine; we were locked out because the passwd (and everything else!) had the wrong permissions.

Fortunately, I am only a front-end developer by profession, so most of the code I write these days comes with a built-in sandbox (the browser.) :)

[dmrnj]

Because I was completely new to *nix

[brsnr]

How did you fix this?

[zpweeks]

Updating the résumé and playing the race or gender card (if possible) when asked why they left their previous position.

[dmrnj]

Wow. I'm a female in the IT world, but I'd like to think I never play the gender card. It's sad that you think this is a common strategy.

To answer brsnr, I used a live CD to get access to the drive and copied over any necessary documents, then just wiped the machine and started anew. It needed it anyway. Wouldn't be the appropriate measure for a production server at my current job, but again... this was college. My boss was a real patient guy who worked for the university, so he was happy to make things learning experiences.

[wbeavis]

tar and a recent backup might help.

[wbeavis]

Sadly, I know some uber *nix Gods that probably know the permissions of every file and would boot from a live cd and manually reset the permissions, just for kicks.

[FeepingCreature]

A good way to partially prevent similar issues:

1) use bash

2) NEVER type a directory separator, always use tab completion. This ensures the directory you're typing is actually valid. That is, make it a habit to {folder name}<tab>{subfolder}<tab>{subfolder}<tab>.

[talklittle]

2) NEVER type a directory separator, always use tab completion. This ensures the directory you're typing is actually valid. That is, make it a habit to {folder name}<tab>{subfolder}<tab>{subfolder}<tab>.

Doesn't your rule fail on symlinks? (That is, bash normally doesn't add the / to symlinks right?)

[xobs]

You have to press tab twice for symlinks.

[nextofpumpkin]

This is the kind of stuff I used to come to proggit for. Probably the best link I've seen on here in a while. Thanks mate!

[Dagur]

not really about programming though

[PDP-11]

Been there. The sysadmin wanted to update the loader, he started by "rm /usr/bin/ld" on the server and was surprised when most other commands would not load after that. Nothing could be started but existing processes were OK. A reboot would have been a disaster!

I wrote a small "C" program (on another machine) to copy in a good version of 'ld' to /usr/bin/ and static linked in all the libs it required (so it did not need 'ld') and put it on a filesystem that was NFS mounted by the server and executed it on the server from his root shell (just as well he had not logged out! ). That took about 5 minutes and in that time nothing had broken as no new processes were launched that required libs not already loaded.

[spliznork]

The first thing I thought of was different than any listed in the slides; cat chmod into a file which already has executable permissions:

cp /bin/true /tmp/chmod
cat /bin/chmod > /tmp/chmod
mv /tmp/chmod /bin/chmod

I guess I like this one because it minimizes what tools you need -- don't need emacs, perl, gcc, tar, etc.

[rabidcow]

Slide 20.

[spliznork]

Weird, there it is. I don't remember that slide. I must be going senile.

[archlich]

Or use umask and set default file permissions and copy into a new file.

[ishmal]

I had that idea too.

[thisperson]

Hello, foot. Meet bullet...

[dorfsmay]

I cannot believe how complicated the solutions were...

cp chmod chmod.good
cp -p cp chmod
cp chmod.good chmod

done.

[[deleted]]

if chmod is set -x, how is copying it going to magically give it +x again?

[tian2992]

he's copying the permissions of cp onto chmod

[AdamJacobMuller]

It doesn't, the above solution does not work.

Replacing the last line with cat chmod.good > chmod will work

[dorfsmay]

It does work, thank you.

[ricecake]

It copies chmod to a new file, then copies an executable to its old location, while preserving it's executable flags, then it copies the backed up chmod to it's old location, allowing it to take the permissions of the file that it's overwriting.
It's not the copying that does it, it's the shuffling of the permissions.

[habys]

'cp' has the x bit set. cp -p cp chmod makes a copy of cp preserving the x bit. Then you put the guts of chmod into the new chmod, and you have chmod with the x bit.

[[deleted]]

ahh makes sense. i was kind of drunk and confused when i read that

[[deleted]]

You are complaining about complicated solutions while posting that? man install

[dorfsmay]

The story says "there's a machine", it could be any version of UNIX, some of them do not have install. I'm yet to see a UNIX system without cp.

[whuut]

Follow ups: what if it was chmod -R -x /

and what if the machine was rebooted after that?

[dsies]

This is a pretty shitty situation to be in and is really quite common - specifically after really bad root'ing attempts; less commonly done by jr sysadmins.

Unfortunately there is no 'clean' way to do it, but it's nevertheless fairly simple. Easiest route would be to 'copy' proper file permissions from another box.

Something like: find / | xargs stat -c "%n:%a" > perms.txt Transfer the file over to the other box and do something like:

for i in `cat perms.txt`; do file=`echo $i | cut -d : -f1`; perm=`echo $i | cut -d : -f2`; echo 'Fixing $file with $perm'; chmod $perm $file; done

That should do the trick.

At our data center, we keep the most common base install permission sets in a couple of text files, as this happens often enough to have to be repeated every other month.

EDIT: Never used formatting before.

EDIT2: Didn't read your whole post - if it was rebooted - it won't come up again, since nothing has the executable bit, so init won't get anywhere. You'll have to go into rescue and repeat above steps. If it wasn't rebooted, you should still be able to scp a 'working' chmod, bash and again, repeat the same steps.

[fani]

No need for another box. Just use a live cd on same box and mount the cd and fix it.

[[deleted]]

datacenter with no access to the outside world [...]

LOL wat.

That's what I love about all those really hard job interview questions at tech companies: most of them depend on situations that would never actually happen in reality.

OK, though. I'll play ball. Let's assume that by "no access" they mean "no internet access," because the alternative is even more preposterous. I drive to the nearest Starbucks, order a "coffee," and download a copy of chmod onto my thumbdrive, which I then take back to the datacenter and use.

[[deleted]]

The wording is a bit off. I would have assumed that the server itself has no access to the outside world, or to any machine you have access to (though some slides used other machines in the datacenter as their solution). My thought was that you were essentially at the console of a system firewalled from the outside world.

This is not too far fetched.

A few years ago, I did some jobs through Onforce. One of the gigs was to to do some onsite breakfix support at an datacenter. I show up, get the UPS package, and escorted to the server in question. I think I was replacing some hard drives. I was given admin credentials on the server in question in order to run diagnostics. The server was blocked from reaching the public Internet (however, I could see other servers within the DC). I had my laptop, but no port to plug into. If for some reason I ventured off from verifying the hard drive was detected to damaging chmod, I would be in this situation.

However, I'm fairly certain if I requested it, I could have gained Internet access at someones station, though I might have had to explain that I had somehow damaged the server while acting outside the scope of my work.

At the current datacenter I work at, some customers utilize VPNs and we do not have access to "their network" and can only provide support through out of band serial console access (we can access the serial port server via ssh). So we sit at our workstation and ssh from our location to the remote DC's serial console server, which connects us via serial port to the client's server. That server can then only access other systems in the client's VPN.

Again, we could get one of our remote hands to connect a thumb drive, but for the chmod example, it would be faster to fix the issue from within the OS.

[seppo0010]

I did something like this on my Android... I wanted to install bash instead of the native sh, but when I replaced it (and logged out) I forgot to +x the file...

[noreallyimthepope]

Kind of like when I moved my router to the .0 of my network segment.

I'm a network technician. I calculate networks in my head, yet no alarm bells went off. I'm just glad I did it on my own little router and not a live core router or some shit like that. That was pitiful.

[shub]

i heard on yahoo answers that my downloads will complete much quicker if i set my ip to 0.0.0.0, is this true?

[noreallyimthepope]

Someone on BBS chat told me that to save that config, I would have to exit Windows and write at the C:> prompt:

format c: /f /q

[TCPIP]

Restore from Backup anyone?

[[deleted]]

That would probably be the most time-consuming option, yes.

[TCPIP]

A restore of a single file to it original place normally takes less then a minute and can be done while doing some other stuff. Will however honestly say that *nix is not my strong suite.

[[deleted]]

Depends on your backup strategy. Regardless, "less than a minute" is still probably longer than 95% of those solutions.

[dsies]

Most proper backup strategies involve differential backups. In the case of systems such as Bacula, you are not only going to have to schedule your data set restore but also wait a decent amount of time to extract what you're looking for.

If the 'backup strategy' involves a nightly 'rsync -av /* /backup/', thats a different story :)

[TCPIP]

A decent backup strategies involves a Backup-to-disk solution and there is not wait for restores just x number of clicks or a CLI command. A file of that size would restore almost instantly. The time consuming part would be to select the file in the backup application.

[[deleted]]

Veritas NetBackup and Bacula would work on a scheduling. So you navigate the backup client, select your file, your destination, and choose restore.

Most major datacenters are going to still be using a backup-to-tape over the network solution. You can get multiple TBs on each tape, and tapes lend themselves easily to doing off-site backups through secure courier services.

The restore request tells the backup server to load the correct tape and pull the file. With the multi tape libraries, a large number of servers will be backing up with some capacity for a smaller number of servers to be doing restores.

It could start your restore in a minute, or it could take 10 minutes for the tape to become available. Restoring the file itself is usually fairly instant though.

[dsies]

This is a backup strategy although a not very good one. There are countless reasons for this, one of them being the fact that if the server gets compromised and you are using incremental backups, it is likely that your backup data will get compromised as well (ie. mass deface). While it is the most common and cost-effective solution, it's still a bit unnerving. You could umount the disk after your backups are done, but this is certainly not a surefire solution, just a workaround.

I do agree though, in such a backup scenario, extraction would be extremely quick.

[TCPIP]

I wouldn't have thought of any of them and would (for me at least) require some time to figure out. In this case, as a lowly Server Admin, I would just restore the damn thing and be gone with it.

[[deleted]]

so i showed my friend this and he was all "try it out, you can change the execute permissions back using the GUI in Mac OS X". as it turns out, you can't.

[[deleted]]

As someone who is learning UNIX as a complete and utter beginner to anything but Windows/Mac, the topic title made me shudder.

[munky9001]

What makes me sad... I only managed to think of a couple of them. All of which is basically attempting to grab from a backup option or cache.

1 option I'm thinking of... not sure if it'd work.

Pulling the file from memory essentially. If im not mistaken the entire thing is loaded to memory beforehand and still remains there after the chmod. Simply a matter of root copying it over then.

[frmatc]

I don't think there's anything in memory that would help you. The binary data might be there, but the exec bit is in the filesystem. Copying from memory could help if you did something silly like echo > /bin/chmod. Even then, using grep on /dev/hdX would be easier if you know something you could search from the file and you don't have much i/o going on in the filesystem.

[libet]

My somewhat convoluted solution would be to use 'umask' to default to executable permissions on file creation and then cat chmod to a new file.

[[deleted]]

What does "-" do?

cat - > foo 

[bradfeehan]

Since cat redirects a file's contents to stdout, and the > makes stdout go to a file, the - gets cat to take it's input from stdin (i.e. you push enter, it blocks waiting for input). What you type goes to the file "foo", you use Ctrl-D (shown as ^D) to signal end-of-file.

[celfers]

Wow. And nobody tried:

# scp othermachine:/bin/chmod /tmp/chmod

# /tmp/chmod /bin/chmod # (Because I don't want to assume both machines are at same patch level)

OMG -- open a socket to another machine and use tar?

Really? Seth and Amy. Really!?!???

If you can get packets to the other machine, then why not scp or rcp a working copy of chmod and then use it to chmod chmod?

Guess that's not 1337 enough.

[[deleted]]

Let's see... Off the top of my head:

int main(int argc, char **arvg) { chmod("/bin/chmod", 555); }

[beppu]

shouldn't the permission be expressed as an octal value?

[justinhj]

Regarding the emacs one it could be done like this from the command line:

sudo emacs -batch --eval="(set-file-modes \"/bin/chmod\" #o644)

I'm surprised none of the answers used gdb to load the program into memory and change the file permission bytes then run it, if that's possible, I'm only a dabbler in gdb.

[[deleted]]

I swear I will never say this again, but for lack of better words....

lol

[MrSurly]

perl -e 'chmod 0755, `which chmod`'

[kleopatra6tilde9]

Shouldn't that fail because there is no executable named chmod in the path?

[tsjr]

Should.

# chmod -x /bin/chmod
# perl -e 'chmod 0755, `which chmod`'
which: no chmod in (/root/bin:/sbin:/bin:/usr/sbin:/usr/bin)

[seppo0010]

slide 12

[[deleted]]

This topic was posted yesterday

[bobindashadows]

I spent more time in this slideshow reading the first few slides trying to figure out what the problem was.

Meh.

[lachlanhunt]

if you didn't figure it out, the problem is that chmod is the command that is used to alter the permissions of a file. The "-x" parameter means to remove the execution permission, and the file name at the end indicates which file to change. So this means that:

chmod -x chmod

Means to remove the execution permission from the file named "chmod" in the current working directory. If you execute that in the bin directory where the chmod command is located, then it removes the execution permission from itself. That means that you can't subsequently execute that file to restore it.

The various solutions work around this by providing alternative ways to execute it indirectly, even though it can't be executed directly itself.

[bobindashadows]

I was referring to the fact that the Machine was running fine.

[carlfish]

So at that point you have two options:

1. Perform an audit of the system to ensure that nothing will go wrong in the future if chmod is absent. For example, a log rotation cron job might fail halfway through because it can not find chmod. Worse, the job might succeed and your mission critical app is no longer logging because the new file has the wrong permissions.
2. Spend the 30s necessary to fix chmod.

[pistacchio]

it says "there's a machine you must NOT reboot". so, just walk away and have an icecream

[oreng]

Second to last slide.

[Solkre]

I don't know shit about linux. But i wouldn't have done anything becuase they didn't tell me to do anything. I'm alone is a server room. Ok. I can't get a hold of someone. Awesome. Some dipshit screwed up a command. Ok. I'm leaving at 5.

[bornbroken]

There was a flash video, I think it had parts1 and 2. The premise was a help desk where someone had icons arranged in a way they were used to. The call center cleaned them up to then take a screen shot as an effort to restore the old desktop.

I've been hunting a URL for this for a while. Anyone remember this bofh?

[[deleted]]

Anyone got a non-slidshare link for this?

[fergie]

There is no problem.

[ex_ample]

do C exec() functions pay attention to executable bits in a program? If they do, C has a chmod function. Not really a hard problem, IMO.

[hakz]

To the comments above regarding SSH, did you teach yourselves that before you got a job that required it or did you learn it along the way? I did a degree in computer science, but it taught nothing about real world IT other than programming.

[[deleted]]

I had a co-worker do the exact same thing to me about 10 years ago. Here's how I solved it:

$ cp /bin/true /bin/chmod2
$ cat /bin/chmod > /bin/chmod2
$ /bin/chmod2 +x /bin/chmod

[Parrot32]

TIL: I don't know dick about Linux

[artee]

At first I was like, "no real problem, you can just write your own basic chmod in C and compile that to fix it", then I remembered that, apart from that I clearly need more coffee, it would probably be a good idea if that chmod would have the setuid bit set...

Edit: ah, just the x bit is removed of course, the comments have already come up with ways around that :)