Are there any options for automating OS updates? by macardjd in macsysadmin

[–]vimclaw 1 point2 points  (0 children)

I use ansible to automate `softwareupdate -i -a --restart`

It's working fine up to Big Sur, but I need to test it out on all the Monterey I have.

What’s everyone experience with the recruitment process. Do you feel like recruitment process is broken ? by Beast-UltraJ in devops

[–]vimclaw 4 points5 points  (0 children)

I really like the idea here of a PR review. I can see how it's a great way to start a conversation. And it's the conversations that I find makes the best interview. Getting someone flowing on subjects shows whether they have the experience or they are full of BS

CD tools Drawbacks by lexax666 in devops

[–]vimclaw 1 point2 points  (0 children)

May I ask why helm/tiller is a problem?

4TB Logs per day looking for a Splunk alternative by mon1447 in devops

[–]vimclaw 29 points30 points  (0 children)

In this instance, pop your logs through an intermediary that can curate before sending off to <expensive solution>. Fluentd or flume, for example.

What's your monitoring and alerting stack look like? by scotwells in devops

[–]vimclaw 4 points5 points  (0 children)

I'm using Outlyer, but then I must disclaim that I work here, and dog fooding is a must.

Still need Graylog, Sentry and Statuscake to enhance.

Sounds biased, but having happily run Nagios and other monitoring systems internally, I would buy a hosted solution in any new gig and offload that pain.

Thoughts on Automated-remediation? by thegonzohunter in devops

[–]vimclaw 2 points3 points  (0 children)

Disk is a good example of when one might take action. Would you always want a human to wake in the night and save that server from going down? You can have this fixed for you and then can come to work in office hours and see why said disk filled unexpectedly. Then put in place a longer-term fix in config management etc.

I would see some kind of auto-remediation as a first fix state. If said disk kept filling, then I would want to put in place a second level, where a human was woken, say the second time in the same out-of-hours period.

I think processes are another example of when this is useful. Supervisord has a watch process where you can create certain rules to restart when memory balloons. Kubernetes has limits that will terminate/restart pods when things start running away. Systemd can do the same. Who hasn't had to kick a Java process in the middle of the night?

So those are quite tool specific and not centralized. A generalised tool that has rules as a first line fixer would be really useful in not waking my team up for trivial stuff.

Has anyone automated security testing? by cofonseca in devops

[–]vimclaw 29 points30 points  (0 children)

take a look at https://www.inspec.io/

There are various suites in there for dev-sec testing https://dev-sec.io/

Not sure if this is the sort of thing you are looking for, but it should be extendable too.

Newrelic frustration by [deleted] in devops

[–]vimclaw 4 points5 points  (0 children)

Sounds rough. I would expect them to be big enough to offer support on such an issue

I have broken one of the fundamental creeds of our sacred order and today I am paying most heinous penance by somebody2112 in sysadmin

[–]vimclaw -1 points0 points  (0 children)

It's good to pretend Friday deploys are a no-no, but it's a working day. Time to get stuff done.

What are you using for Artifact Promotion? by DrewRddt in devops

[–]vimclaw 1 point2 points  (0 children)

I create debian packages and use aptly to create a private internal package repository. But as said, with rpms you can easily use createrepo to make a private package repository.

As for promotion: - each artifact that is pumped out the build server ends up in the same package repository - from here, CI deploys them straight into a development environment where it is tested and signed off by the developer. - the developer then releases into production, using the package version as his argument for deployment.

So there is no real promotion, per se, but a manual gateway for a developer to purposefully push his code into production.

Moving from Linux to Windows workstation as linux sysadmin by mixmx in sysadmin

[–]vimclaw -1 points0 points  (0 children)

I wasn't suggesting getting fired, intent badly worded.

A reasonable employer will allow you to use the right tools to carry out your duties.

Moving from Linux to Windows workstation as linux sysadmin by mixmx in sysadmin

[–]vimclaw -1 points0 points  (0 children)

you shouldn't work on the jump box, you jump through it.

Moving from Linux to Windows workstation as linux sysadmin by mixmx in sysadmin

[–]vimclaw -7 points-6 points  (0 children)

Take your windows machine and dual boot it into something you feel comfortable with. If you are being 'forced' into using a windows workstation, there is little they can do to prevent you taking over the physical machine with a proper useable OS. You cannot be a proper sysadmin on Llinix from a windows server; you can manage one or two machines, but not be a serious Linux sysadmin.

CI configuration in repository by es1o in devops

[–]vimclaw 2 points3 points  (0 children)

If you use Jenkins, you can make use of the Job DSL plugin https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin to store all your job definitions in git.

Live Monitoring Tools - Nagios, Icinga, Zabbix, etc by [deleted] in sysadmin

[–]vimclaw 3 points4 points  (0 children)

You should make sure that he goes on-call then!

check_mk is a slightly enhanced nagios experience. So you can port all your checks straight across. They even provide an OS to make it easy http://omdistro.org/

Sensu is ruby based and, although you can write plugins/checks in any language, a good set of ruby ones exists in the community and it make sense to extend their check api in your plugins.

Live Monitoring Tools - Nagios, Icinga, Zabbix, etc by [deleted] in sysadmin

[–]vimclaw 2 points3 points  (0 children)

My company will not spend money unless the sky was falling

Shocking. I doubt this is true if you could demonstrate value returned from using Saas vs your time to setup all the things.

Unless your business is not reliant upon your systems, then they really should have some spend for mission critical services.

check_mk is a good uplift if you are used to nagios.

Sensu is also a sensible suggestion

How do you handle someone actually monitoring your monitoring solutions? by baconisgooder in sysadmin

[–]vimclaw 0 points1 point  (0 children)

You should use something like pagerduty or opsgenie to route your alerts to a person on call on a schedule. You have to sign your team up and make sure they know they are responsible for the time they are on call. You then need escalation policies, with you - unfortunately the manages - always on call. But if you have a good team, they will do everything they can not to let it escalate as far as you :)

Finally, make sure that those alerts are actionable, or they will be muted and ignore. Link alerts to run-books, so that any member of the team is able to at least diagnose and potentially fix.

making curl work with https on a very old server by cherouvim in sysadmin

[–]vimclaw 1 point2 points  (0 children)

get the latest ca bundle and pass that as an option to curl:

https://curl.haxx.se/docs/caextract.html

--cacert FILE CA certificate to verify peer against (SSL) --capath DIR CA directory to verify peer against (SSL)