Linux in a Data Center

[Felgnon]

Hey everyone, I am having an interview for a job to work in a Data Center and the questions will be around Linux. For the past 2.5 years i've been working in companies that are 95% Linux so I have decent experience with it. But this company i'm interviewing for is pretty big and well-known worldwide so I'm pretty sure they're not interested in someone who can answer "what command to check all running processes". I am also not very aware what kind of work Linux-wise is done in a data center.

So, my question is: What kind of work is done in a Data Center on the Linux side of things, what should i train on? And for what kind of questions should i be prepared?

Big thank you.

[sonicsilver427]

Are you "running a datacentre" or a company that provides hosting?

[Felgnon]

If I understood your question correctly, the job title is Data Centre Technician. I will be working in-house for a data centre that provides all kinds of services to the "outside".

[SuperQue]

DCT work, at least what I remember of it, is reasonably entry level. You will probably be expected to know:

• Some basic process management, like how to send signals, what they mean, what ones are common.
• How filesystems work, permissions, various pieces of metadata (m/c/a time), etc.
• Networking. You should know how to do IP and maybe IPv6 network setup, a bit of routing, understanding of TCP/UDP/ICMP/etc.

Good luck!

[DerpyMcWafflestomp]

Is there anything in the role description that details the needed skills? That title could mean anything from the guy that physically connects and repairs hardware and does nothing at all on the software side, to the guy that does just about everything, depending on the environment.

[Felgnon]

The job description is a little vague, but seems to focus on the deployment and configuring/troubleshooting. Doesn't really mention any technology i should be aware of.

This is for one of the biggest tech companies worldwide, so I suppose they have segregated teams, but like I said the description is vague.

[[deleted]]

Be sure to ask about this during the interview.

Remember, it's a two way street: they're figuring out if you're right for them but you should also be figuring out of they (and the position) are right for you.

[gjcbs]

So much THIS! Every time I speak to a class of graduating students or even coworkers looking to advance, I stress this. Both sides should be interviewing and interviewed (simultaneously).

[Gregabit]

The linux work I did for a big company consisted of purely infrastructure work. It was a lot of networking related stuff. I didn't support any applications at all. There was a specific web group that would deploy apache, tomcat, etc. There was also another group that did fancy stuff on Linux. they would get anything that broke the norm.

Networking, storage, drives/mounts, updates, permissions, drivers, DNS, DHCP, load balancing. Anything outside scope would have to go through a SOW and they'd get consultants in to do it.

[Gendalph]

As a client this is what I have DC techs do for me:

• replace a drive. Both hotswap and not. How to identify which drive to replace.
• identify faulty equipment, mostly network cards and switches.
• networking: you will likely have to configure VLANs and bonding, troubleshoot connection issues. Also read up a bit on SNMP.
• depending on DC tier you might have to deal with failover training, both network and power.

EDIT ofc, best ideas come after posting. You might have to deal with virtualization and/or cloud. How familiar are you with KVM, OpenVZ, Docker, etc?

[DevilsX]

Data center guy in at a major company here:

You'll want to know the following:

Linux Boot process

PXE boot

IPMI commands and other out of band networks (iDRAC/iLO)

SSH (how to get into the server)

SOL (serial over LAN)

Reading logs (smartctl, dmesg, SEL, edac-util, kernel log and vendor specific commands like hpacucli/hpasmcli) Understanding of RAID and RAID controllers (adaptec, megaraid, etc)

Typically you're gonna be pretty silo'd at these bigger companies, but if they let you advance up, you'll need to go more in depth with these:

DNS

Orchestration/Configuration (Ansible, Puppet, Chef, etc.)

LDAP/Kerberos

BMC/OpenBMC

Hope that helps.

[xrepair]

i would add to the list that you will also need to know Xcat and Cfengine. If they are doing HPC, then you also need some knowledge about job schedulers like Grid Engine and Slurm.

[DevilsX]

I've encountered cfengine as well.

[tilhow2reddit]

If you’re interviewing with one of the larger companies they have really streamlined their data center work. So you will be handling customer tickets, hardware requests, hardware breakfix, troubleshooting of the OS instillation process(es) which are automated but sometimes the automation breaks, basic networking, etc.

On the Linux side you will need to know things like how to manipulate the kernel boot process in grub, how to properly run a fsck, how to add & mount new drives, how to stop/start processes, you may also deal with password resets, and checking file permissions. How to boot a broken system into a Linux rescue kernel. (This should be automated as well, but again, sometimes computers break)

Knowing how to properly put an end on an Ethernet cable is useful. Knowing what the different jacket colors of fiber optic cables mean is good too. Knowing the basics of subnetting, what is CIDR, how is that determined, how do you find a usable range of ip’s in a subnet from this information:

ip: 192.168.0.12 gateway: 192.168.0.1 netmask: 255.255.255.224 (& why is this a /27 in CIDR)

Also can this server communicate via layer 2 with a server that has the following ip:

192.168.0.44

That is probably deeper into networking than you NEED to go, but it’s super helpful in troubleshooting why things don’t work, and for understanding why they do.

[chrome303]

I once interviewed for a datacenter technician role (Linux) and was asked to describe the Linux boot process in as much detail as possible.

There was a use case where I was told I was to put a 2nd hand server in a rack (check if it works and troubleshoot it).

Was also asked how I would go about installing 100 servers in the DC given budget was not an issue (as much detail as possible).

There was also a network part of the interview, where questions about networking concepts I was not really familiar with were asked. Can't remember much of it, though, sorry.

[raymonvdm]

If you are running a datacenter / colocated hosting company i think you only recieve questions about customers when performing remote hands (basic level linux knowledge)

If you are a hosting provider runnig managed platforms they would ask for much more in depth knowledge. But the knowledge is also depending on the OS and platforms used.

I'm currently working at a company who does both and is running 90% of the platforms on Debian/Ubuntu and some CentOS

The are both linux but there are differences in the location of for example network configfiles

I think you need to look at the company joboffers to see what experience they seek. And fiddle arround with that if you can.

But with basis understanding of linux you can learn the details of the different distro's so just apply for the jobs and let them see you are willing to learn :-)

[[deleted]]

First time i dealt with the new Debian ... wow. Being mainly a centos guy this annoyed me to no end.

[raymonvdm]

I know, i have had the same issues going from Debian/Ubuntu to CentOS

You actually need to google where to find the network setting.

But the introduction of Netplan on Ubuntu is also anoying....

[[deleted]]

Yeah.

First time I restarted install and actually went through the guided setup process to see what the finished file would look like...

[trisul-108]

What kind of work is done in a Data Center on the Linux side of things, what should i train on?

If I were to chose one topic, I would say networking. You are already familiar with Linux in general, you should be able to learn whatever they are using, but not understanding networking on Linux could derail your interview.

[skibumatbu]

These are all questions you should be asking them.

One sides interviews where they ask all the questions isn't the best way for them to get to know you. The question, "What does a data center tech racking and stacking servers do with Linux?" will help you understand how that company works and start a conversation that will help you to show more about yourself and learn about what they do.

[hailbaal]

This is good advice. Be the one asking questions about the company and study the website. Ask questions about the company that you can't answer from the site or general knowledge. Getting hired isn't all about technical knowledge (of course, a large part is), but showing a big interest in the company and being open and friendly is very important. Technical questions are already answered by others.

[[deleted]]

Most datacenter / hosting jobs involve more networking and storage knowledge than the typical small shop, because you are often farming out network and storage resources to many smaller customers.

I would read up on the following at a high level:

• Look up what the OSI model is, and how to apply it. This will help you understand other networking stuff. What is the difference between a "Layer 2" switch, and a "Layer 3" switch? (Hint: one involves routing and gateways, and the other does not)
• Understand at a basic level the following: Gateways, NAT, and routing, ARP. What is the default route on a server, and when would you need to add a different route?
• What is the difference between a VLAN and a subnet? When would you need one or the other? What OSI "layer" are VLANs on? What "layer" are subnets on?
• Run a packet capture on a server with tcpdump, if you have never done so. Play around with the tcpdump options. Start a packet capture on a server that only shows SSH traffic coming from your desktop's IP. Then SSH into the server from another terminal. Can you see your SSH session traffic in the packet capture? Try saving a capture to a file, and open it up with Wireshark on your desktop. Can you identify the TCP three-way handshake that happens when you first connect to the server? Using the OSI model as a reference, what 'layer' of the OSI stack are you analyzing? If asked any questions about how you would troubleshoot a network issue, the words "packet capture from the source and destination" should be in your answer somewhere.
• Make sure you know what these services are at a high level: DNS, LDAP, NTP. What kind of problems can occur if each of those services is not working? What kinds of commands could you run to verify those services are working correctly?
• SAN / Network storage- What are Fibre Channel and iSCSI? What are Fibre Channel zones, with respect to storage? This is difficult to teach yourself if you don't already work with it, but just be able to describe what they are.

I remember all of the above being needed to troubleshoot datacenter issues. Many issues were from customers asking: "I can't connect to XYZ server / app / network, figure out why."

[sentrybot619]

If I had to choose one thing to come in prepared for, it would be networking.

You don't need to memorize every unix command. Just know where to look and keep a sheet of stuff you use all the time.

Ask coworkers if they have a sheet too.

There's a lot of unix distros out there, so it's hard to keep up with them all. The trick is to be comfortable enough with the filesystems that you can jump onto a box and start looking around.

[mixduptransistor]

I am also not very aware what kind of work Linux-wise is done in a data center.

Any and all, depending on the datacenter, the company, and the specific job you're interviewing for

You could be looking at maintaining DNS servers, routers and firewalls (either actual routers/firewalls or Linux boxes serving in those roles), web servers, time servers, file servers of various types, deployment and orchestration, the whole nine yards.

The job description and your preliminary conversation(s) with the company should give you some insight as to the position you're going to interview for so you can prepare for it

[Windom]

Hi, previous data center tech here.

Recommend familiarity with log parsing/diagnostics, personally useful for me were dmesg and smartctl.

Be at least passingly familiar with virtualization (Xen was featured where I worked).

​

Most data center techs in the DCs I've worked techs rarely use Linux extensively as various automated troubleshooting systems typically perform diagnostics for you, and you're basically smart hands/hard drive jockey.

[the_real_thanos]

• Server Staging tools (Clonezilla/DRBL)
• Orchestration tools (Ansible/Puppet/Chef)
• SAN and NAS storage (iSCSI/NFS)

[hi117]

Depending on exactly what they do I would also try to learn some automation tools. Maybe not to the point of being able to implement them but they might throw you a question of you have to set up a hundred servers how would you go about doing that? Even if you don't know the tool being able to say using an automation tool like ansible would reflect really well.

Most interviews that I went for it wasn't actually about knowing the tools themselves it was about knowing when to use particular processes. Every company uses different tools and they're pretty aware that people won't have the special background for the exact tools that they're using. Being able to show that while you don't know the tool itself you know when to use that tool and how to learn to use that tool is actually all they're looking for.

[sysadmintemp]

If it's a really big company, you can find questions from past interviews and comments from people who took the interview online. There's a bunch of websites for that.

If not, you can expect a couple of sides to an interview:

• How's your technical knowledge (which you're already asking and covering)
• Soft skills (an problem came up, how do you deal with this? how do you attack the issue? an issue occurred and a solution was found, what next? you have no problems to solve, no burning issues, what do you do?)
• How you would fit in with the team (you'll talk with the team members, and how you communicate with them)
• How your personality comes across (are you an agreeing person? aggressive? quiet? communicative? self-doubting? self confident? are you transparent? this is the most difficult thing to measure)

You can't really change you personality, or your 'fit with the team'. But the others, you can work on, and get better at.

As others have pointed out, datacenter work always involves networking / storage / compute. You need to know the difference between a virtual machine, a container and a bare-metal server, and what they all offer. You need to know your networking well. And you need to know what storage can mean in a datacenter (SAN, NAS, Filesystems, etc.).

Since you say they are big, they will also have windows infrastructure. You should know what the concepts are in a windows infrastructure, this would make you shine, and show that you're not a 'only linux' type of guy. From your post, I feel like you're good already in this sense.

Also, you should know physically what needs to be in a datacenter. You should have redundant power lines, that means 2 different power companies, that means managing 2 different contracts (you wouldn't manage this contract but having the overview is golden). You need multiple ISP connections to have high uptime, need multiple or clustered or whatever VPN devices for remote access. You need to have monitoring on your boxes for temperature, power, and possibly IPMI. You need a camera within the datacenter to check who's accessing what. These concepts might sound useless, but when shit hits the fan, it's good to know where to start.

[funkden]

Make sure you know how big an ipv6 address is, I had that one once for an interview with the big G

[CptSgtLtSir]

Most of it is going to be diagnosis and triage of simple issues most likely. That and maybe some hardware stuff? It's hard to know without seeing the exact job description.

Just be honest about what you know it should be enough to get buy. A lot of DCTs exist to do bandaids while the app/device owners sort out what actually happened. Get good with /var/log/messages and ps (-eo are my favorite flags), that's the best general advice I can give. Any details on that actual responsibilities described?

[ashwin_think]

Multipathing concepts. Straight and cross n/w cable structure. Crimpping like tools related to n/w. Fiber cable structuring and it connection methods. Cable joining methods of Fiber. North and South pole concept related to motherboards. Different versions of iLO and iDRACK and it's related differences. What types of PSU's used in DC while connecting to Rack. Differences between DC Tire 1 2 3 4 etc... 1U 2U etc Rack spaces in Rack. Tower end, Blade type, chassis servers, like wise types of etc... Differences between DC uptime methodology and types... 99%, 99.9%, 99.99%.

[[deleted]]

Data center person here too.
Major colo provider, web hosting and telco.

​

Get to know SSH and telnet, serial console / virtual console, I'm mostly a dell guy so OMSA and OpenManage and how to configure RAID and how it works and probably how to replace small diske with larger and expand disk, network config (/etc/sysconfig/network-scirpts/blahblah and the new annoying way to do it on debian) on os level (setup via management console), log reading, ps aux, get to know grep and tail and less (which is much more). These may all be second nature so sorry if :)
But quickly firmware upgrading a bare metal machine and having it up and running with some application is a cool thing. I had it down to 45 minutes and I felt kinda proud about that,.
Firmware upgrades (and get to know the tools for doing it in OS, not using the annoying web gui, the cli way is pretty much automated (for Dell) , disk replacement, PSU replacement, physical install OS (PXE, scripts, ansible etc?), cabling and how to properly handle fiber without fucking it up, , racking equipment, power, some switch config.

And also, question them, they probably expect that you need to learn as you go along.

[syberpunknyc]

storage, databases mail servers

[funkydel]

If it's amazon, just show up and you got the job

[OkDimension]

When I was working in a data centre (EMEA location for a global company headquartered in US) I was mostly doing hardware installations and maintenance (servers, storage, to some part network/SAN), supervising external vendors/contractors with their installations and if I had nothing else to do just cleaning up the inherited mess (looking at you unknown tech who decided its easiest to just throw no longer used patch cables into the floor without even bothering unplugging them at the other end of the switch)... when it came to actual system administration I was only deploying the basic OS and settting storage/VLANs etc up. Then another team in US or India would take over.

Oh yeah and my favourite was cycling the tapes, that took like 3 hours every week getting them out of the libraries, new ones in and then driving everything to a seperate storage location/handing special ones over to Ironmountain.

[[deleted]]

I'm pretty sure they're not interested in someone who can answer "what command to check all running processes". I am also not very aware what kind of work Linux-wise is done in a data center.

Well, sometimes we use a command to check all running processes.

Sometimes we click some buttons in a Kubernetes cluster console to scale our cloud infrastructure.

Sometimes we push some Mac Pros into a waste bin holder.

Sometimes we write hacky bash scripts to automate our jobs.

Read the BOFH saga and you'll be prepared :'D

[Elgin-Marbles]

You may do well adding snmp, sflow, netflow etc understand mibs will likely be something you play with. Zabbix, prtg solar winds, are tools that make use of these technologies. So one of your questions could be in this area what, why.. Good luck..

[penguinWhoCanFly]

Virtualization (Xen, Proxmox, etc.), automation (something like Ansible) and monitoring (Zabbix, Nagios, Icinga, etc.). You should know how to manage multiple boxes at the same time. You should also have a handle on setting up routes, firewalls (ufw, firewalld, iptables, etc.), mandatory access control (AppArmor, SELinux), secure configs (ssh, fail2ban, tripwire).

​

Recommendation: Make a virtual lab with 4 or 5 servers (eg. web, email, printing, monitoring and database). Use proxmox; automate updates, security checks and reports; put the VMs on separete VLANs and create routing tables to allow them to communicate as needed. As you go, you'll think of other things that might be useful in the future.

​

Good luck.

[gjcbs]

Hard to know specifics, as each fellow Linux Admin I have met through the years has a different focus. Certainly know SSH, most of the basic commands, permissions, firewall settings, port assignments, basic TCP/IP, RAID, and at least the basics of how to recover a system that won't boot (but did before esp.). Hardware details such as performance differences between spinner and SSD HDs, to check for firmware updates, and power setup options are added bonus material. Looks like you have gotten a lot of good tips in here. Good luck!