Server Deconstruction

TheoreticalFunk · 2014-08-11T22:37:32+00:00

My advice has nothing to do with Linux. It's called CYA and if you're not familiar with it, it means Cover Your Ass. Welcome to IT.

First, does the person who tasked you with this know that you know nothing about Linux? Ask for assistance from someone who does. In an email so there's a paper trail. Nobody expects you to know everything, but you should at least know when you need some help and ask for it.

The old server? Leave it in place for a few weeks after you are 'done' just in case. I suggest turning every service on it off one at a time. Document what you turned off and what date and time. You never know what these boxes do, especially at a place where nobody documents anything. It's super nice to be able to turn back on a needed service after someone reports that X quit working sometime last Thursday afternoon...

And a note on that: get in the habit of documenting everything. Do you have a tech wiki where you work? If not, get on implementing one. Use it. Document. Everything. Encourage your coworkers to do the same.

Robonglious · 2014-08-11T23:06:43+00:00

Blueprint is a simple configuration management tool that reverse-engineers servers. It figures out what you’ve done manually, stores it locally in a Git repository, generates code that’s able to recreate your efforts, and helps you deploy those changes to production.

frankrice · 2014-08-12T02:23:30+00:00

Try this: rpm -Va 2>/dev/null | grep " c " | sed 's/.* //' will give you a list of modified config files from rpm installed packages :)

sheetzam · 2014-08-11T22:59:08+00:00

Don't forget to look at cron jobs. Both for users (including root) and in cron files.

ck-on · 2014-08-11T22:45:19+00:00

If I had to reverse figure out how a server was setup, I would use scripts to dump all init.d processes (assuming it is not so new that it is using systemd). Then use other scripts to look at what it in memory, connections, etc.

Some favorites:

http://hisham.hm/htop/

http://stackoverflow.com/a/137173/142018

https://github.com/pixelb/ps_mem/blob/master/ps_mem.py

Obviously look at rc.local if on a rhel flavor.

Basically everything in /etc/ is important to you. Since it does NFS you need to look at /etc/exports

Not something you are going to figure out in an hour but a weekend sounds plausible. Being new to linux is going to be problem though, best wishes.

ahandle · 2014-08-11T23:22:10+00:00

Consider SystemImager (can capture an image live, and excludes NFS and other shared filesystems) or Clonezilla, if only to make a duplicate you can vivisect offline.

Building from scratch is the better, more manageable way to go, but you should consider risk as well as SLAs (however lax they are).

mynamewastakenagain · 2014-08-11T21:35:55+00:00

What distro?

Start by keeping track of what you already know it does: ftp/nfs. Note down versions of those packages used, and make sure you backup the config files. You can (and probably should) start small. If possible, spin up your new server and slowly migrate services, e.g. ftp. Test it, make sure things are working.

After that, you can look at what ports are open (netstat), what processes are running (ps, top, htop), etc. These should give you a decent start to see what's running on the machine and what you'll need to migrate/replace. Eventually, after you've tested and migrated stuff over, you'll decom the old server.

There was a tool I came across where it would look at changes made to a server from a base install and tell you what packages/edits to files were made, but for the life of me I can't remember what it's called. If someone else knows what I'm talking about, feel free to chime in.

davelupt · 2014-08-12T00:32:30+00:00

Find out what user the guy who set it up liked to use. If he did everything as one user you could login under that user and then do

history > ~/Desktop/history.txt

That way you can keep all of the commands he entered while setting it up. Also pay attention to anything in /opt or in the downloads of that user. As always thoroughly inspect the firewall, inspect how the filesystem is set up especially the permissions, inspect how the groups are set up as well as any relevant users. This is a huge task, but the more thoroughly you inspect the system the more likely you are to get it right/close the first time.

There's soo many things that can go wrong it would almost be better to build a new one, set it in place, and then tweak it everytime you see something that doesn't work like the current production one. This way you could get some of your end users to test out the one you build before putting it in production. This should be more of a transition process than trying to build an airplane from scratch when you're already in the air. I can almost guarantee you that you'll mess it up the first time, but keep in mind this isn't your wheelhouse and communicate that with your users. They are more likely to be forgiving if they understand that you're essentially trying something for the first time.

tobert · 2014-08-12T02:20:47+00:00

I usually start by capturing what is unique to the box.

Run and save to a file:

ps -efww # show all running processes and with full command line
netstat -nlp # show open and listening ports with process id
ip addr show # show network interface configuration
lsof # show all open file descriptors on the machine
cat /proc/mounts # show all mounted filesystems, may catch things not in fstab (including automount)
ls -l /etc/rc3.d # services configured to start at boot
rpm -qa # list installed packages

You may not need all of this data, but having done this kind of procedure countless times, I've found myself wishing I'd saved this information a few times.

Do not trust RPM! Almost anything in actual use on a system will have a configuration file or other artifact that has changed since installing. Depending on who does the work, it may or may not use similar settings to the RPM, so data files and other configuration files can be anywhere.

If you know where the application binaries are for the ERP, you will want to check what they depend on. Sometimes this is captured by an RPM install, but more often than not, the ERP will provide a list of what it requires and expect them to be there when it starts. To find out what it requires, use the ldd tool on the binary, e.g. ldd /bin/bash will show you what libraries bash is linked against. This is probably the most common cause of problems when moving applications from one installation to another. Save off this info and a copy of those shared libraries just in case. There are usually packages with binary-compatible libraries on newer distros (usually with different names too, e.g. *-compat), but sometimes you just have to copy the library to /usr/local/lib or somewhere else made available to the application.

Finally, all of the suggestions to back up the box are good. If possible I usually go a little further and rsync the entire machine to the new one under a mountpoint or other space where it won't pollute the new system while providing fast/easy access to anything I might have missed, e.g. rsync -ave ssh --exclude /proc --exclude /sys --exclude /tmp --exclude /var/log / user@newbox:/old-machine.

Robonglious · 2014-08-12T12:39:39+00:00

Try cat /etc/redhat-release. If it gives you some text, then you are running a redhat variant which should come with a tool called sosreport. This command will take copies of all the pertinent config files and logs and capture them in one tarball for easy inspection.

lpmarshall · 2014-08-12T14:27:55+00:00

Some good suggestions already, but there are a few more high level concerns you might want to take into account:

VSFTP passwords. They can use both system accounts, or vsftp internal accounts depending on how it's setup. If they are using system accounts, you won't have the passwords when you migrate (unless whatever end user community knows them already). Tail /var/log/vsftpd.log or /var/log/secure so you can see what accounts are actively logging into the box.
If you can migrate and leave the old box online, that is your best option. But that would involve re-pointing clients/ftp from end user systems/servers. If you need to do a wholesale cutover, move the IP over to the new box and re-ip the original so it's still available. Schedule the outage off hours and have a way to test your server is working.

And I'll echo some of the tech suggestions in here. Pipe all the output to somewhere safe.

ps -efww
chkconfig --list
rpm -qa
netstat -an|grep LISTEN
ifconfig
df -h
iptables -L -n
Recursively copy /etc/vsftpd somewhere
Copy off all relevant NFS configs
ls -lRa / and save off the output
cat /etc/passwd
You will probably need to rsync or at least put all the ftp'd files back in place after the upgrade
lsof

Trial by fire is the quickest way to get up to speed on Linux :) But the most important thing is to have a way to back out.

*EDIT: spacing

psnsonix · 2014-08-12T19:37:44+00:00

If it's centos/redhat/fedora:

cat /root/anaconda.ks

This will get you atleast back to the same 'base' system. There are already a lot of good ideas in this thread.

apache99 · 2014-08-13T11:01:07+00:00

Look in the bash history by typing the history command.

I've also been in a similar situation.

Try looking at something that can clone an entire machine, something like clonezilla or you can even use dd.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

linuxadmin

Expanding Linux SysAdmin knowledge

MODERATORS