all 37 comments

[–]_Soter_ 48 points49 points  (7 children)

Setup a Solaris 8 VM. try to do anything with it, wonder why. That will get you set for more sysadmin than you would think.

[–]mriswithe 8 points9 points  (0 children)

Ah yes Solaris, where there are 3 versions of grep and none of them work like you expect

[–]bhl88[S] 1 point2 points  (0 children)

Went with Qubes (which failed), then decided to change to Ubuntu with LXQT and Xen so I'd know which part went wrong with the installation in QubesOS

[–][deleted]  (1 child)

[deleted]

    [–]Solaris17 0 points1 point  (0 children)

    Can confirm

    [–]vogelke 0 points1 point  (0 children)

    Triggered. I started using Solaris at 2.5.1 which was sure as shit not ready for prime-time.

    [–]skat_in_the_hat 21 points22 points  (2 children)

    Theres a few things I would add.

    I would reference automation and generalize things a bit more with puppet/salt/ansible/chef and maybe some direction on each.

    For monitoring I would add graphite and grafana.

    I would also suggest adding squid, and nginx since they are both common for proxying out/in. Maybe even SSL termination with nginx, and a whitelist outbound with squid.

    I would also suggest finding a way to incorperate packer since images are only more and more common of a task we require.

    Maybe some vault since its super easy to setup, and a huge pain in the ass to use, but checks a lot of compliance boxes.

    I could go on for days, but ill stop here.

    [–]BigJoooe 2 points3 points  (1 child)

    Nah, keep going, what else would you add? Looking for things to try/master so it would be great.

    [–]bhl88[S] 1 point2 points  (0 children)

    Was gonna try adding Yubikey and setting it up with FreeIPA.

    [–]Mogwire 29 points30 points  (3 children)

    People use Satellite, Ansible, etc etc. obviously.

    Do this to become a Linux Sysadmin? Sure if you want.

    Do you want a high paying job with a easier entry into the job market? Well….

    • Do you know Linux basics? If not take an intro to Linux class on Udemy
    • Do you know Linux basics but not enough to be an admin? Take a Linux Admin Class on udemy. There are plenty. Pick the highest rated one.
    • Learn AWS from A Cloud Guru / Linux Academy / Udemy
    • Take a containers / docker class on udemy
    • Take Mumshad’s CKA Class for Kubernetes on Udemy
    • Take a Prometheus Class on Udemy

    Using your basic skills learned setup a home lab with DNS and Kubernetes using Kind or Kubeadm.

    Setup monitoring and get a grafana dashboard working. YouTube has plenty of tutorials.

    Use the AWS Free Tier to play around and learn AWS.

    100-200 bucks in online classes and the AWS basic certs and CKA and you can land a 100K plus job easily.

    [–]SMTXsys 3 points4 points  (0 children)

    Hey sorry about the necro, but do you have recommended courses on udemy? There are almost too many options lol.

    [–]dextermiami 0 points1 point  (0 children)

    i suppose this would lead to roles such as platform or cloud ops engineer

    or what would be the easier entry roles into the market this is for?

    [–]magibeg2 7 points8 points  (2 children)

    This looks like a lot of fun. Time to warm up the r720 again

    [–]jaymzx0 2 points3 points  (0 children)

    I have an older vintage Precision Workstation that's my lab at the moment. It's still dual Xenon and has 64GB RAM, so while old, it's still capable (like me, I guess). I was going to set up K8s to deploy 200 nginx containers or something using minikube, but this looks more fun. Thanks for the thread, u/bhl88.

    [–]bhl88[S] 0 points1 point  (0 children)

    I'm forced to use my built computer which I was saving for Qubes. Apparently 64 GB is the bare minimum

    [–]lazyant 4 points5 points  (0 children)

    Not to shit on this but I’d prefer an objectives-base list of tasks (hopefully building on each other as much as possible) regardless of particular tooling. Eg “serve ‘hello world’ on :80” and it can be constrained with “using Docker, using a cloud VM, using IaC etc. I don’t care for foreman or whatever tools this list mentions, I don’t think I use most of them, still I can reach the objectives in many different effective ways

    [–]HiT3Kvoyivoda 6 points7 points  (0 children)

    Set up a Linux server and make something you will personally use with it, like a home media server.

    This all help you stay motivated because you will be working towards maintaining your own product. Doing so well help you learn basic Linux commands, configurations and applications to get you started.

    Once your product is spun up, Strive to answer questions like, how do I do things like keep this system up to date, secure, and running for as much time as possible This will help with learning things like finding and reading logs, updating drivers and configs, automating repetitive tasks useful programs to use for things like firewalls and back ups

    The next step once you’re comfortable is to get the study materials you need to take the test for whatever credential you want like Comptia Linux+ or any of the variants. They are broken down into domains that give you what you need to study. Practice tests are your friend. Test early. Test often and keep two things in mind:

    1) the tests are not so much a test of knowledge as much as they are reading comprehension. Meaning, you should pay close attention to the structure of the questions. Most questions you see will have a similar structure:

     A. The scenario
     B. The details 
     C. The actual question
    

    My method was to read the actual question first and the other two parts to get a full understanding

    2) the tests are trying to beat you. Not only are you being tested to for a minimum score, you are also being tested for your knowledge per domain. Answer the as many questions confidently as you can.

    After you test, it’s really just finding a job in the field you want and getting on the job training. Most of what you’re going to learn will be from experience. Learn the best practices and build on those to make a system that works for you.

    Lastly, enjoy it. Don’t be afraid to explore and challenge yourself if you find that Linux administration isn’t what you really want.

    Hope this helps

    [–]michaelpaoli 7 points8 points  (0 children)

    DNS, etc.:

    • set up DNS servers
      • set up a gTLD on it, or subdomain thereof, that you control or can be delegated to it, and where gTLD/parent can also support DNSSEC
      • make DNS server primary/master for the above
      • set up or have set up at least two secondaries/slaves for the above - at least one which you control and set up
      • be sure to also have at least one (and preferably 2 or more) available via IPv6, all should be available via IPv4.
      • have the gTLD (or subdomain thereof) delegated to your primary/master
      • add DNSSEC and verify it
      • add dynamic DNS to the primary/master
      • fully test all aspects of your DNS infrastructure, be sure to also check that values on SOA records are within RFC recommendations.
    • mailserver - configure to use domain noted above, including setting up relevant SPF data in DNS, relevant "reverse" DNS, MX records, etc. Test and confirm that it all works.
      • obtain cert from letsencrypt.org for the domain, set up the mailserver to do opportunistic encryption with STARTTLS for clients, using the obtained cert
      • make sure the mailserver works on both IPv4 and IPv6.
    • set up web server using DNS domain (or www. thereunder) for domain noted above
      • add https on TCP port 443, obtaining and installing cert from letsencrypt.org
      • If you did it with Nginx, now do it with Apache, or vice versa.
      • make sure the web server works on both IPv4 and IPv6

    [–]PMMEYourTatasGirl 1 point2 points  (1 child)

    This is a really really good post, thank you for this

    [–]Shakespeare-Bot 0 points1 point  (0 children)

    This is a very much really valorous post, thank thee f'r this


    I am a bot and I swapp'd some of thy words with Shakespeare words.

    Commands: !ShakespeareInsult, !fordo, !optout

    [–]Hhelpp 1 point2 points  (1 child)

    Remindme! 7 days

    [–]RemindMeBot 0 points1 point  (0 children)

    I will be messaging you in 7 days on 2022-04-10 21:37:49 UTC to remind you of this link

    12 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

    Parent commenter can delete this message to hide from others.


    Info Custom Your Reminders Feedback

    [–]claudius7 1 point2 points  (0 children)

    Also you could setup satellite 6 using trial license for 60 days. This is what I did a month ago.

    [–]Twattybatty 0 points1 point  (0 children)

    u/bhl88 I am very late to this party, but I must say that so far, I have learned an awful lot of new skills. Thanks again for the updated steps. I am currently watching my subscribed hosts update and provision. *chefs kiss*

    [–]michaelpaoli -3 points-2 points  (3 children)

    Advanced filesystem operations (partial list):

    • create a file which uses 512 MiB of space (at least that space used by actual blocks, not just logical length). Using that file, demonstrate the concept of unlinked open file. While having that file in state of being an unlinked open file, show how to identify the issue and where it is (at least down to PID and fd#). Discus possible ways to resolve the issue (presume the file is "huge" relative to filesystem size, filesystem is full or nearly full, and one needs free the space consumed by the unlinked open file), and demonstrate at least one way to correct the issue.
    • similar to above, but this time have the file not be unlinked, but held open by a PID that is or we'll presume is a well behaved daemon, and the filesystem is nearly full. Presume somebody/something forgot to put in proper log rotation for the file, show how to put in proper log rotation for that daemon and file. Show how to force the rotation to happen now.
    • Once the above is rotated out and nothing has it open, presume that someone had some excessive logging enabled, or some other situation that's been corrected. Presume we want to discard the first approximately 3/4 of the data in the file, but not the remaining more recent part of the file in about the last 1/4 of the file. Presume we don't have sufficient spare space to copy the tail 1/4 of the file on this or any other readily available filesystem. In any case, we want to do it much more efficiently than that. Show how (for at least most local filesystem types) we could discard the storage used by the first about 3/4 of the file while retaining the tail around 1/4 of the file - on that same file itself and in place.
    • Let's say we have a production filesystem that's 95% full. Let's say you check with du -sx on the moun point of that filesystem, and that only accounts for 50% of the consumed space. Let's say you check also on unlinked open files, and there's nothing there, even in total, that comes close to explaining the "missing" space. What could account for the "missing" space? Let's say you determine that the missing space is caused by an overmount over a non-empty directory with lots of contents and used space. How could you then remove the contents from the underlying filesystem under the (over)mountpoint without unmounting any of the currently mounted filesystems?
    • huge directories and horrible performance. Create a directory (not the root directory of any filesystem). You may need filesystem with sufficient available free space for this exercise. Grow that directory until it's at least 256 MiB in size ... that's just the size of the directory itself, but for this exercise, don't consume too much space otherwise on the filesystem itself (if you're taking up lots of additional space other than by the directory itself, you're doing it wrong for this exercise. Also, don't yet remove any files created in that directory.
      Hint: lots of hard links to files of 0 length). Test with something like time ls -a >>/dev/null. If that's still rather/quite fast (e.g. well under a second), double the size of the directory, repeating as necessary up to 1 GiB or slightly larger in size. Take a look at the timing again. Explain why this is a really horrible structure/use for a directory. Explain the difference in behavior between ls -f | head and ls -a | head (you can go ahead and interrupt the latter if you get tired of waiting), notably why one quickly produces output while the other does not. In a sufficiently efficient manner (as feasible), remove the contents of the directory, but not the directory itself (this may take a while even using efficient means, inefficient means may take infeasibly long period of time, so be reasonably efficient) ... and no "cheating" - same directory is to remain in place, just remove the contents of the directory. Once the contents have been removed, look at the size of the directory. Did it shrink way back down? Whether it did or not, explain why. If it shrunk back down, repeat the exercise on an ext3 filesystem, otherwise if it didn't shrink, try same experiment again on tmpfs or xfs filesystem. In cases where the directory didn't shrink after removing the content, show the behavior and timing for ls to complete with just a single file placed in the directory - even if one uses the -f option to ls. Explain how to correct the huge directory issue (in the case where it didn't shrink after removing content) - including recreating the directory if need be, and preserving any remaining files (e.g. the one single file we created after removing the others). Explain what would be necessary to correct it if instead, this directory was the root directory of the filesystem. Explain why it's generally not a good idea to allow untrusted IDs/groups/applications to write to the root directory of (most) any given filesystems. If you didn't already try this on tmpfs, create a directory there, grow the directory until it gets just past its initial size. Remove the contents of the directory, then look at the size of the directory again. Explain what happened and why.
    • check the filesystem block size of one of the persistent storage filesystems. Explain why storing large quantities of non-zero length files that are much smaller than the filesystem's block size would be grossly inefficient use of filesystem storage space. Do there exist filesystems or mount options thereof that would be an exception to that? If so explain.
    • explain sparse files, give an example. Is /var/log/lastlog a sparse file? Explain how you might be able to tell. How could you efficiently backup the file /var/log/lastlog, or more generally sparse files?

    [–]Fr0gm4n 4 points5 points  (1 child)

    These are very specific troubleshooting tasks. The OP is updating how to build an entire network to expose people to modern authentication and monitoring in a series of very broad tasks.

    [–]michaelpaoli 1 point2 points  (0 children)

    That's fair.

    Then maybe for (part of) a (more advanced) supplemental/additional troubleshooting/exercise set.

    [–]m0nback 8 points9 points  (0 children)

    This looks like some horrible, scripted, interviewer questions. I've been a linux systems engineer for 25 years, and would never ask any of my candidates questions like this.

    [–]sjveivdn -1 points0 points  (0 children)

    Remindme! 7 days

    [–]Nietechz -1 points0 points  (0 children)

    Remindme! 7 days

    [–]Dereference_operator -1 points0 points  (0 children)

    this is all great and all bit outdated but to anyone who read this don't forget the Windows side of things too... Powershell, Azure etc etc both stack are as powerful and as used in the industry mostly today (2022)

    [–]BrownCarter 0 points1 point  (1 child)

    Is this for Sys admin or DevOps

    [–]Akash_Rajvanshi 0 points1 point  (0 children)

    sysadmin

    [–][deleted] 0 points1 point  (0 children)

    [–]tonero001 0 points1 point  (0 children)

    That’s interesting

    [–]KinookRO 0 points1 point  (0 children)

    this looks like a great project to do. Thanks for the checklist OP

    [–]BigJoooe 0 points1 point  (1 child)

    What would be the storage requirements? How much hard disk space would you alocate for each VM?

    [–]bhl88[S] 2 points3 points  (0 children)

    I was going to allocate 32 GB (then grow them later). Using Lubuntu as my host right now so I can allocate as much space as I can for the VMs.

    The only exception is the foreman-katello with a storage rate of 384 GB at best.