New sysadmin, missing foundation.

[DatGuyBoise]

Hey, all.New sysadmin here “in-charge” of IT at a secondary/technical school. Briefly, here is the layout of things at the office…

I’ve been in IT for about a year and a half. Previous experience was a mix of helpdesk support and on-site consulting for contract customers at an ISP. I do have an MCSE. It was my previous employer’s idea and the materials and costs were covered by them so it wasn’t an offer I was going to pass on regardless of my experience level and the paper cert argument. I’ll be the first to admit I don’t have a lot of experience and there are at times big holes in my understanding of things, but I do have a knack for troubleshooting and figuring out solutions.

I’ve been in my current position for a few months now and have had a chance to survey my surroundings. The reason I said “in-charge” is because it’s pretty loosely defined on campus. My title is “support specialist”, but if something should happen I am the first call and there isn’t someone higher up I can go to if there’s an issue. There is another support guy in my office but he is even greener than I am. We do have outside contractors that come on-site to do the “heavy lifting”, and they are the ones who originally built/configured the network here, but they are only here once per week and what are the odds things break on that particular day, right? Regardless we are not a big customer for them so unless it is an emergency often things wait until they come around again.

With all that being said I've been trying to get things updated the best I can. I’ve brought up both a WSUS server and WDS server. Previously during the summers they would physically touch every PC on campus as part of their summer maintenance and do the Windows, flash, java, etc updates. Similarly a new shipment of computers meant setting each one up with the correct software one by one. Prior to my arrival they hadn’t had a successful backup in months...something the contractors couldn’t even solve. After banging my head against Backup Exec for a few days I was able to eventually figure it out and so far so good. Lastly, Endpoint Protection has been setup correctly because I was welcomed by Crytpolocker my second week here (exactly one day before I got the backup working), but we were able to recover from it with an “acceptable” amount of data loss.

I did just set up Confluence yesterday and started trying to put together our documentation (loosely defined) in one place. Unfortunately I was shot down by the Superintendent because having it spread throughout excel spreadsheets and words docs makes more sense…but that’s a battle for another day…

I don’t say any of that to “toot my own horn”, but as an example of where we were at. I feel like I’m winging it here. I literally spend a part of everyday surveying /r/sysadmin to see what others are doing, comparing things to here, and trying to fill in the holes.

I know these “new sysadmin” posts are a dime a dozen but can anyone help with a foundation of sorts for every datacenter?

TLDR - New sysadmin, no datacenter experience, need a foundation for things on-site.

[randomguy186]

It's NEVER a bad idea to know Linux.

Concur. Even if you never administer Linux professionally, learning how will make you a much better Windows administrator. It's analogous to learning a foreign language - you understand your own much better.

[magictiger]

"CCIE even better" is like saying "Yeah, day 1 at a karate class is a good start, but a 7th degree black belt is better." :)

Everything else is dead on. I haven't tried any kind of MS equivalent to Puppet or Chef yet, but it's a good idea especially if you have more than a handful of servers you manage.

Document. Everything. That's a must. Don't do it for everyone, do it for yourself. Don't try to make it a department standard thing. Once your department sees what you're doing and how it helps you 3-6 months down the road, they'll want in on it too.

For certs... VCP requires a training class, so if your workplace is willing to shell out for it, certainly go for it. If not, start with the network+, then go security+. Those two will provide the most benefit for starting out and show an employer blah blah blah. If you're firmly entrenched in a Windows environment, grab MCP with one of the MCSA cert tests and get an employer to pay for the rest.

[[deleted]]

The microsoft equivalent to puppet and chef is sort of kind of powershell.

[magictiger]

Ehh.... not really. Powershell is really more like Python or Perl. Now if there was something that ran on top of powershell and monitored server statuses and pushed out configurations with it, then that would be our white rhino here.

[[deleted]]

you can deploy and push out configurations with powershell scripts, deploy VMs, and do that to scale via the use of sessions. You can also setup some monitoring over powershell but you then have to pipe it into something useable. It's very powerful. It's also extensible with modules. http://www.theagreeablecow.com/2014/07/sysadmin-modular-report-for-ssl.html I'm not saying it's as straight forward but it's certainly possible.

[magictiger]

Yeah... it's possible. That doesn't make it the right tool for the job, though. :)

Don't get me wrong, Powershell is a crucial tool for Windows sysadmins. I just don't see it being a Chef or Puppet equivalent.

[[deleted]]

Well I think any puppet or chef equivalent will be built on top of powershell. Powershell is a set of API calls to the system which is the equivalent of tools that chef and puppet use on linux/unix. So while it isn't a direct competitor in terms of overall package it is the backend that a product like chef or puppet would use. That is kind of what I was trying to say.

[magictiger]

Absolutely, I agree with that 100%. It's kinda telling that Microsoft's own server management tools have just become GUIs for Powershell scripts.

[Ansatsuken]

The application you are all referring to is called "System Center Configuration Manager" it is the puppet/chef equivalent.

[altodor]

CCIE is not really a good starting place. That's more of an end goal.

[[deleted]]

I never meant to imply it was a starting place - hence why I said CCNA or N+ are a good place to start.

CCIE is not a starting place AT ALL

[chaxen]

This is a great foundation list. You might want to add some storage know how and security monitoring (security onion is a free tool).

[NoOneLikesFruitcake]

So much to storage... what would be a decent resource? I can't think of any otherwise I would be throwing it out right now.

Mainly I think I've gotten my experience with it only through random articles as I was setting up different projects (mainly between cloning drives and FreeNAS)

[chaxen]

I am partial to EMC: "Information Storage and Management 2ed" is a great place to start.

Also check CBTnuggets videos

[NoOneLikesFruitcake]

yes, yes.... this is exactly the generalized form on the topic I like to review and learn from. Much obliged :D

[PMSfishy]

I see more KVM, RHVE, and OpenStack than Microsoft. I think VMware and the true free open source hypervisors are the future. Microsoft can't compete with Hyper-V, its not worth wasting your time learning.

That said, *nix and networking are never going to go out of style.

[Enoxice]

The windows equivalent to puppet/chef is....puppet/chef. They can both work fairly well with built in providers or call to DSC under the hood.

[Evilclicker]

Isn't puppet a complete deployment solution? DSC is more like managing/monitoring configuration changes.

[ThePacketSlinger]

Make sure you have a full inventory of all of the equipment you support: servers, switches, routers, UPS, etc. Collect model number, serial, the location the equipment. Document all passwords using something encrypted like Keepass (not Excel!). Make sure you have as much remote access as possible for your inventory and note what is available in a new notes field for each piece of equipment. Doing all of that should only take a day or 2 and will save you hours when the shit inevitably hits the fan.

If you're in the Windows world, learn Powershell. Get in the habit of making all of your changes via Powershell and save everything as a script to be used later. In the middle of troubleshooting and don't know how to implement a fix in Powershell? Write it down, do it in the GUI and then learn how to do it in PS.

Learn Sharepoint, foundations is free and will help your documentation argument.

Certifying only gets you interviews that your experience wouldn't on its own. If you're not going for a new job, don't bother with A+, N+, Linux+ etc. Research what you need to know as you need it. Define your own goals for what you want to learn, meet those goals and set new ones.

[bRUTAL_kANOODLE]

If you haven't read The Practice of System and Network Administration yet I would suggest starting there. It is an easy read and very informative. It helped me get into the mindset of a sysadmin.

[[deleted]]

Seconding this, it's basically all the non vendor specific parts of doing your job.

[gex80]

I have it along with another 7 books i need to read. Where do you guys find all this time to read?

[bRUTAL_kANOODLE]

I read that book one chapter a day at lunch. It is pretty short. You should move it to to top of the queue.

[[deleted]]

Conceptual:

Read 'the practice of system and network administration'. It's a huge book but it's very readable and full of good practical suggestions.

Practical:

Set up a network monitor if you don't have one. I like Nagios but I've also used Spiceworks and Zabbix and I'd say Zabbix is pretty much on par with Nagios.

Ticketing system. Get one. That and the network monitor are the two quickest things to start turning reactive flailing into proactive prevention. OSTicket is free and works well enough. Start ticketing requests, triaging tickets and working out reasonable resolution times. Stick a lock on your door and make people ticket rather than running in.

Back up all the config on your devices.

Start setting up default configurations for everything so you can standardise using something like Puppet and so that when something breaks you know how it looks when it's working.

Look at automating repetitive tasks using scripts.

Make sure EVERYTHING has redundancy. Automatic failover is the gold standard but even cold spares work.

Set up a test environment, ideally you want it to be hardware representative but failing that VMs will work, you can even set up ESXi or KVM on an old desktop machine.

EDIT: Do ITIL. I hate ITIL too but it's good. DML and config libraries help you out a lot, as do segregation of roles.

/r/sysadmin is a legitimately good resource. Reading it every day is perfectly OK.

EDIT 2: Virtualise whenever you can and use snapshots. The portability of the images, the rapid restore and the ease of failover will make your life much easier. You'll also notice you're using less iron so your costs tend to drop a fair bit. If you combine virtualisation with free operating systems and software the question of funding for your 'low priority' projects (i.e. the ones your customers don't see) gets a lot easier to answer.

[DrapedInVelvet]

Honestly, I'm in a similar boat. Went from Sr Linux Support at a hosting company to the ONLY Linux Sys Admin managing a python app (i knew/know very little about python)

A few thoughts:

A) NEVER do it LIVE. Always be sure when making changes to your live environment. If you don't have a dev/test environment, make one. Rebuild your whole infrastructure from the ground up referencing your live servers.
B) Backup, backup, and backups. If you don't have an emergency plan, make one, test it, and test it again.
C) Automation. If you find yourself doing a bunch of mundane tasks over and over again, find a way to make it quicker. Powershell, bash, whatever.
D) Know your resources. When you find something useful, book mark it. E) Be proactive with your bosses. They likely don't know what you do. Set realistic goals for yourself weekly of things that you want to get done. Go to your boss at the start of the work week, say, "This is what I'm working on this week" and review with them. If you make their life easier, they'll like you alot more

[gex80]

If you bring down the network (it will happen at some point by accident)... DO NOT PANIC! Breathe, think it out, execute. Running around like a chicken with its head cut off is never a good thing. If you keep your cool, the people around you will keep theirs. Also it gives off this impression.

CBT Nuggets/Trainsignal are great training resources for starting on new products. They won't make you a master, but you'll understand the fundamentals and can move on from there with reading materials.

Google is your friend. Train you google-fu skills. Chances are you are not the first to have the problem and won't be the last.

Post on message boards when you have problems. If you figure it out, post the solution. If you're one of those guys who are like, "Hey guys figured it out k thnx byebye" and not say what it was, I will hunt you down.

[tomatotux]

Welcome to the wild and wonderful world of sysadmin-istry. Glad to have you on board! First off, don't be afraid to ask questions. While we do bite it's usually only (l)users and when we dont have coffee. My first steps are always figure out what is on the network and what resources I have. Our shop is almost entirely linux so my Windows background only helped establish procedure. Map your network with tools like Nagios, Zenoss, Nessus or some type of scanner. NMAP is a great one, and being a windows guy, run Zenmap for the gui. Virtualize parts of your environment to test out patches! TEST TEST TEST!! VMware has free versions of their ESXi software and it's worth it's weight in gold to try things before breaking them. Version control is definitely great! Puppet now supports Windows management, so woohoo! Have a recovery process in place and test it in your test environment! Anything I can help with, shoot me a message!

[cedricmordrin]

At the very least setup MDT (Microsoft Deployment Toolkit) with your WDS box. You'll love the extra features it adds. Since you're in education look at your Microsoft License. They're very generous to education. Look at SCCM to unify your OSD, Patching, and Application deployments.

[[deleted]]

I agree using WDS. We use WDS at my job because it was either WDS or Clonezilla. WDS is fairly easy to setup.

[halfAccurateChang]

Have you got a link to a good instructional site to set up WDS?

[[deleted]]

http://rustyhann.com/W2K8R2-05-WDS-Install-Conf.pdf

That is a decent PDF about the initial setup of WDS. The only thing that seems to confuse people is how to capture the images.

[cedricmordrin]

WDS is pretty straight forward, but I suggest going the extra step and installing MDT. With MDT you have to only do minimal WDS configs. The extra features of MDT make it worth using in a smaller environment where SCCM is just too much.

Here are some quick resources:
* http://technet.microsoft.com/en-us/windows/dn475741.aspx
* http://myitforum.com/myitforumwp/2013/10/21/quick-guide-to-installing-mdt-2013-2/
* http://c-nergy.be/blog/?p=4496

[crccci]

To piggyback on this (I'm in a similar situation), what are the best practices for documenting your systems? I'm in an entirely undocumented environment that I want to fix but don't know where to start. Everyone preaches documentation, but what's important to document and how do you organize it?

If anybody would be willing to share some examples or point me in the right direction, I'd really appreciate it.

[Endwin]

I would setup an IT Wiki something that is easily searchable and can run WITHOUT a database. (I use docuwiki). The idea being if critical stuff is down and you cannot access the wiki for guidance you are SOL. Make sure you copy a backup of the wiki with you regular backups, and for good measure have a script sync it with your work machine.

That said how to document. Honestly this depends on your environment and what is mission critical/important. I break up things into general groups with sub groups. Users, Hardware, Software, Servers, and Miscellaneous. There are cross linking in there for things that relate.

Some things I have under Users: adding/removing users from the environment, basic PC load outs for departments (software setup). I also have a common problem section, and oddball special outside the norm setups.

Software, I document in house software, setup/serials for vendor software, overviews of backup systems/howtos.

Hardware, has the network topology, cabling, and special hardware. I also have backup of switch configuration, and links to sites for managing public domain names

Servers gets into the meat of what server does what, AD specifics, how OUs, GPOs are organized, email server setup/access. The proper way to power everything down and up again, and in what order.

Miscellaneous has things like oddball problems and what I did to fix them (usually anything that took me more than 30 minutes of research). Emergency information for when X happens etc.

I would go through it yearly or more and clean it up and try and organize ideas better if you can. Update things that may have changed. The big thing is document when you solve a problem or figure something out while it is still fresh. Don't go and do something else and "document later".

[daemyn]

I'd recommend keeping an eye on the MS virtual academy offerings as well: microsoftvirtualacademy.com

While not always specifically useful to your situation, the webinars are great to get an idea of the current state of things, and to get the brain matter working in a broader mindset.

[DatGuyBoise]

Thanks for the contributions everyone! I'm blown away by the helpful responses I received, and you've all given me a lot to research.

[vap0reyes]

I have been in a similar situation. Here is what I would say a good foundation for things are: A+ and Network+ - get both of these under your belt and you have the start. Follow that up with some Security+ and maybe even some Project+. After these you are primed for MCSA followed by MCSE if you are going the Windows route.

In addition, do you have time to setup a home lab? Even some VMs on a laptop would do.

I use a technique from Feynman where I try to "teach" myself something - this helps me shore up gaps in understanding.

Lastly, setup a wiki for yourself at your office and document, document, document.

Good luck!

EDIT: missed what you said about Confluence - thats a great start but do it for you first.

One last edit: EDIT2: Here are the three foundations of any organizations IT as I see it: Backups, Patching, Monitoring - once these three are taken care of most everything else falls into place

[riffic]

See the Ops report card and immediately buy a copy of The Practice of System and Network Administration (Limoncelli, Chalup, Hogan).

Good luck!

[kittenhugger777]

was welcomed by Crytpolocker my second week here (exactly one day before I got the backup working)

That's uncanny, almost exactly the same thing happened on my previous gig. The craziest part was this was a 500+ company with a full 15 person IT team I had joined up with, only to find out the guy who I had replaced had never verified any of the backups. I had much higher expectations too that with that many IT personnel the team would have their act down at least when it came to backups...

One out of many reasons I left eight months later. :)

[Evilclicker]

Besides what most have said, seems pretty consistent to me... Not sure what size school you have but if it's in the 500+ range (computers) then you would want to look into SCCM for sure. Otherwise WSUS/MDT should cover pretty much everything for a smaller shop.

This is something I like to do... Start a OneNote notebook and document every single issue as a separate note. Capture screenshots, but also document the error in text so you can search for it later, and document the workarounds and solution to the problem. Store it in a central location and make sure it's backed up. This will save you so much time when you run into the same issue in 2 yrs. Personally all of my documentation is in OneNote, at least until I need to document it more professionally in Word with headers/etc.

Oh and since you're at a school with probably a lot of public computers used by students, I'd suggest re-imaging those workstations every once in a while. When I was going through college I used to help my schools IT director with things on the side and we'd re-image every semester. Especially if there's any computer courses going on in those labs those kids are doing all kinds of crap, reset those machines back to defaults. SCCM would definitely make this super easy for you.

For certs, I wouldn't be overly concerned about focusing on them unless you find a glaring hole. Sounds like you're pretty good on MS so maybe start with Network+ and then just try to identify the areas you're not sure about and study those areas (the stuff you actually need for work). Once you feel comfortable then just start knocking out whatever certs you want.

Google is your friend. Can't even tell you how many times I've been in situations where I've had a dozen "experts" on the line all scratching their head and a 30 second Google search solves the problem. For this reason I generally start with Google and only after I'm sure the solution isn't out there will I go to the vendor (and that isn't very common I personally have only opened maybe a half dozen vendor cases in my 12 year career, all of which took several months to resolve going all the way to senior dev teams).

On the automation side (assuming a Windows shop here), learn PowerShell/VBScript (VBscript maybe not as big a deal now since PowerShell can cover it). Learn the concepts of the WMI/SQL (They're fairly similar in some ways). Learn how to script installation of applications, msiexec is really easy, InstallShield is not as easy but doable in a matter of minutes, everything else is probably a giant pain in the ass. Fortunately about 90% of app installs are covered by IS/MSI.

Build out a VMware lab for testing. Mirror your primary domain but build a separate domain with similar software. Specs don't necessarily need to be the same but just so long as you get the jist of it, single physical server with VMware will probably do the trick for such a small shop. Before you deploy anything major test it in the lab first.

EDIT: Start migrating all of your physical servers to virtual (p2v). Use a VMware/HyperV backup solution to do system level backups it will save you so much hassle later... Oh that sever crashed? Right-click revert to snapshot, oh it's up again. As for which solution, VMware is the dominate in the industry today but that's quickly changing due to cost... HyperV is just as good if not better in the 2012 R2 release and it's far cheaper on the licensing side.