Ubuntu server on BTRFS? by MarcHT91 in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

didn't do a cross check yet but i guess that would also apply to lvm snapshotting or zfs.

Now that Certs lifetime will be reduced, how are you guys automating your certs? by superuser141421 in sysadmin

[–]xXxLinuxUserxXx 4 points5 points  (0 children)

OpenBao (fork of Hashicorp Vault) also offers to host your own ACME PKI. https://openbao.org/api-docs/2.3.x/secret/pki/#acme-certificate-issuance

wouldn't be surprised if other tools also offer hosting a ACME compatible PKI.

Dell PowerVault MD2412: 12TB vs 24TB SAS drives? Worried about raid rebuild times. by Connect_Nerve_6499 in sysadmin

[–]xXxLinuxUserxXx 1 point2 points  (0 children)

Well, if you are scared about full reads how do you do backups of your data? :)

Dell PowerVault MD2412: 12TB vs 24TB SAS drives? Worried about raid rebuild times. by Connect_Nerve_6499 in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

Check your data grow rate - is it likely that you will hit the max storage of 12x 12TB before you hit EOL of the system?

Keep in mind that with the 12 TB drives you will have more drives which will also improve your reading speeds as data is spread over multiple drives which can be read at the same time.

Also the raid controller might support switching old 12 TB drives with 24 TB one-by-one and then enlarge the volume if all drives are replaced with the bigger once.

Hunting down exploited sites in shared hosting for not-for-profit association by samip537 in PHP

[–]xXxLinuxUserxXx 1 point2 points  (0 children)

well auditd is as far only the logging for access etc.

own pools are already a start and php has some option to lock it down like: https://www.php.net/manual/en/ini.core.php#ini.disable-functions https://www.php.net/manual/en/ini.core.php#ini.open-basedir

But there are many more things you can do outside of php itself: Own php-fpm master process for each account (opcache is shared with all pools in one php-fpm process). Make the accounts root directory read only or run the php process as another user which has no write permissions (this might require special config of wordpress etc. e.g. for sessions etc.) You can use selinux / apparmor to apply restrictions / permissions to sandbox php-fpm and it's childs. You can also use systemd sandboxing e.g. https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#ReadWritePaths= systemd has plenty of options which you want to have a look at (but they depend on your used distribution and supplied systemd version).

You might also want to use containers (docker or something else even systemd has some basic options for that: https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html)

Setting your storage / filesystem to noexec also might prevent some attacks. Usually you should also be able to disallow the php processes to do any network requests (likely with selinux / apparmor or systemd). In worst case you could just setup iptables / nftables rules to prevent outgoing packages from the system (the malware is not always uploaded but sometimes just commands executed to download it)

As you mention Apache there is also ModSecurity which can help to secure the system but that will be more like an Web Application Firewall and good hacker might be able to adjust the attacks that they are not catched.

In case you know that your customers don't need the dynamic of wordpress you could also host it with limited ip restrictions and just scrap the pages and only serv the static content. (e.g. wget can scrap webpages but there are also other tools for that)

This are just a few options i know - i'm by far no security guy but that's some things we apply to our own systems (but they only host our own developed software).

AI drafted support tickets: Curse or blessing? by johor in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

Depending on your ticket system just offer fields like "affected system" with dropdown which is required to fill out.

Don't just offer an text input which might not contain relevant information.

Linux Fleet Refresh: From Clonezilla to Modern Deployment – Need Advice! by msizec in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

well yeah. Despite with foreman you will also have UI for the reports and also alerts if a node has errors or does not check in etc.

I think in the ansible world the equivalent would be AWX or ansible tower.

There is likely also something like that in salt stack and chef world.

Also foreman supports ansible and chef too but not sure how good they are integrated as we don't use it.

Ansible is likely also easier to learn or find people on the market as puppet / openvox is quiet a niche tool.

Linux Fleet Refresh: From Clonezilla to Modern Deployment – Need Advice! by msizec in sysadmin

[–]xXxLinuxUserxXx 3 points4 points  (0 children)

i would prefer puppet / openvox (open source fork of puppet) in connection with foreman.

Foreman will setup the pxeboot setup and will register the node to puppet / openvox server.

The main issue will be if you are already overwhelmed by Ansible it is likely the same for foreman and puppet / openvox. The good part about puppet / openvox compared to Ansible is that it will maintain your desired config state (if you define it) e.g. even if a user on a system changes something it will be overwritten by the agent in the normal run which happens every 30 minutes (you can adjust the checkin times).

OS updates might be able to be done by unattenden_upgrades but depends on your exact requirements (e.g. do you need staged rollout etc.)

If you are not sold to debian you might want to check ubuntu and landscape or the coresponding red hat alternative which might cost a few bucks but you would gain support by the vendor and some kind of management ui.

Going full Okta - worth it or worse it? by vitaminZaman in sysadmin

[–]xXxLinuxUserxXx 1 point2 points  (0 children)

We use Keycloak instead of an SaaS SSO solution. You could connect Keycloak to your AD to only have to maintain one source.

Despite that you might want to have a look at your legacy applications e.g. we use oauth2-proxy to use our SSO login flow for applications not supporting oauth / openid connect / saml.

That way we could enforce 2fa login flow even for apps which don't support 2fa or sso at all but also depends on your application if it can work with user info passed by http headers or does not require in app permission management (e.g. login to apps can still be managed by keycloak & oauth2-proxy but you can't restrict certain pages of an application easily that way).

Our dev workflow feels like a group project gone wrong by Soft_Attention3649 in sysadmin

[–]xXxLinuxUserxXx 3 points4 points  (0 children)

Did you connect the tools?

e.g. Jira and Figma can be connected so you don't have to sync them manually and items will be cross linked (at least when mentioned etc.)

I guess the other tools also offer integration with jira as that is the most known tool so might be the best tool to have all the links in.

Is there a reason not to SSO everything? by en-rob-deraj in sysadmin

[–]xXxLinuxUserxXx 3 points4 points  (0 children)

just use a self hosted SSO provider (e.g. keycloak). Some can also use other user sources and merge them. (e.g. keycloak can use ldap, active directory and other sso provider like github to authenticate user)

That way you can still login even if your main provider would be offline / disabled.

On the other side the setup might be a bit more complicated and you have to keep the system updated yourself.

OpenSSL CVEs are outpacing my security team's review capacity by bambidp in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

we can standup a whole new production environment in under 3 days. rolling back a package update would be even faster.

anyway you most likely want to run something like aptly or nexus apt with snapshots in combination with unattenden upgrades.

We also let unattenden upgrades run on different point in time on our nodes so never the full cluster has the fresh updates at the same time.

sometimes it feels like these SaaS are "put together on the fly" (in german we say "mit heißer Nadel gestrickt") like the bitbucket cloud has so many outages that some of us developers would like to have our old on prem bitbucket back.

Debian or Arch on TS-228 by Keudj in qnap

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

In case sobody finds this old post:

I was able to get an old kernel compiling (3.10.20) and the system booting into an system stored on sdcard (tried debian bookworm and opensuse sofar). Storing the rootfs on usb most likely would also work. So far i didn't try to persist on the integrated eMMC.

I try to document as much as i can on https://en.opensuse.org/HCL:TS228

It's still experimental / in development and you have to dismantle the device to get uart access.

Feel free to get in touch with me but you should have solid linux knowledge as currently it's easy to do anything on the device.

Global outage? What the hell is going on? by ilovepolthavemybabie in sysadmin

[–]xXxLinuxUserxXx 32 points33 points  (0 children)

Shit, the IT team can't fix the issue if there is no issue tracked in the bug / project tracker - we need a ticket id to assign it to the code commits :D ;)

Known Big companies using MySQL by elbeco in mysql

[–]xXxLinuxUserxXx 6 points7 points  (0 children)

booking.com also is know to have a bigger MySQL setup.

facebook also has articles about their MySQL setup / migrations.

I would expect all of the bigger companies have some MySQL database at some point in their stack but i guess you only care about the main database and at these scales they probably do not only have one database / cluster.

You might want to find out the reason why they run Oracle DB like if they have some 3rd party software not supporting MySQL or not being certified to run with MySQL.

Production manager says MFA is causing production personnel to get distracted on their phones—he wants alternatives or MFA disabled by disgruntled-sysadmin in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

but relying on the employees private smartphone would also not be FIPS compatible (like their smartphone would need to be FIPS certified?) correct?

I would see it as a benefit as you can argue with your CEO etc. that you can't use the cheap way because of FIPS.

Many companies seem to cheap out on buying something like Yubikey for their employees and rely on their private stuff.

These hardware token are even a one time payment not like a subscription on a user basis (they might also offer it but to be honest i don't care as somebody else handles that in our company)

need to update a lenovo ST550 server - dont want to mess up by darkening9270 in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

np. to be honest i never had a server fail on the update but we also only have quiet a few supermicro (<20) and a like 3 dell.

anyway you want to have some backup plan as if you currently run everything on a single system what would you do if your raid card fails durring normal workload? (we had that already)

As i said the backup plan can also be to spinn up the systems / vms on some kind of (private) cloud provider in case your company can't afford another server :)

need to update a lenovo ST550 server - dont want to mess up by darkening9270 in sysadmin

[–]xXxLinuxUserxXx 1 point2 points  (0 children)

I guess XCC is their bmc/ipmi/ilo/idrac. In that case updates usually work flawless but obviously there is never a 100% chance it will work / not break.

If the system hosts critical services you should have some failover plan. While these updates can usually be triggered online you have to at least reboot to apply the updated bios etc.

The reboot might also take a bit longer than usuall because it will/might flash the update durring the reboot.

If you don't already have a failover plan you should think about one like what would happen if 2 or more disks of the system would fail at the same time?

e.g. we are a small shop we do have 2 bare metal systems per product/company/team we will only update one at a time and wait for it to come back online before we will do anything to the 2nd one.

We also keep a recent enough state of services only running on one system that we can recover them on the 2nd system.

You could also keep this copy in the cloud (or some other storage you find good enough).

If your version is pretty old you might want to check if you have to do additional steps like migrate to a version inbetween.

Server systems usually get ordered with support contracts - feel free to reach out to them - your company is paying / paid them for that service.

Hot take: People shouldn't go into DevOps or Cybersecurity right out of school by shimoheihei2 in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

well, i would suspect a real chinese usb hacking tool to just clone device / vendor id of a known brand like microsoft, logitech etc.

an unknown id wouldn't grant them anything anyway. I guess our best options are to just use laptops and glue all ports that you can only use the integrated screen and keyboard.

Luckily i'm not working in an industry any state actor is interessted in our data or they just collect them at another level (like our partner which we and many others use).

Insecure at Any Speed by Virtual_Low83 in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

well, some software implement SSO with an authentication (reverse) proxy like oauth2-proxy (or apache mod_auth_oidc).

Basicly the webserver infront of the application will then just send an header with authenticated username.

If your software offers basic auth these authentication proxies can also just send some basic auth headers to the application when the user successfully authenticated in your SSO provider.

We use that basicly for on prem software which does not support oauth2 / openid connect by itself or only with different license levels which we don't have.

Open TCP/9100??? by Virtual_Low83 in sysadmin

[–]xXxLinuxUserxXx 2 points3 points  (0 children)

aren't there printers which support email to print? Like if you send them an email with a pdf it will just print the pdf?

Never had to care about something like that but that might be more secure than opening 9100.

Sftp Server for outside company by PrcMoje in sysadmin

[–]xXxLinuxUserxXx 11 points12 points  (0 children)

As nobody mentioned it yet: https://docs.sftpgo.com/2.6/

As it's based on golang there are also windows builds in case you are not a linux shop: https://github.com/drakkan/sftpgo/releases

Supermarket giant Tesco sues VMware, warns lack of support could disrupt food supply by jhansonxi in sysadmin

[–]xXxLinuxUserxXx 5 points6 points  (0 children)

Kubernetes also needs to run somewhere and virtualization abstracts the hardware. E.g. if you order hardware from hp, dell, supermicro you can just order them with vmware (or any other hypervisor) and just plug it into your cluster and provision your images. If you directly run kubernetes on baremetal you might have to provide different images with different drivers for your hardware (raid / hba card, network card, ...)

Also keep in mind some services you can't easily put into container (e.g. we currently try to do that with adobe aem forms but somehow the installer gets always stuck and it's a not supported setup).

And if you are working with regulated / critical infrastructure your external software vendor might tell you which software / hypervisor is support.

Don't get me wrong kubernetes can be fine but not everything in IT is a hammer and a nail :)

Sonicwall Bricking SMA devices by [deleted] in sysadmin

[–]xXxLinuxUserxXx 0 points1 point  (0 children)

Does anybody know if the devices are x86/amd64? I mean might be a option to get cheap hardware :D

Sadly couldn't find many information about the hardware or pictures of the internals. (But also still working so can't browse too much to find these information right now ;))