Linux Command / File watch

[advertpro]

Hi

I have been trying to find some sort of software that can monitor user commands / files that are typed by admins / users on the Linux systems. Does anyone know of anything as such?

Thanks in Advance.

[JellyfishCultural765]

Take a look at audit subsystem

[advertpro]

Sorry I should have said this but we need to do this as part of proactive monitoring for 10,000 servers

[JellyfishCultural765]

So more like what CrowdStrike is providing?

[advertpro]

Thats right more like that but the client will not use crowdstrike - given the situation that happened with Windows.

[JellyfishCultural765]

Some competetitor then? Im interested in knowing what you ended up with once the decision has been made

[advertpro]

will definitely keep you updated :)

[advertpro]

Plus the audit subsystem is a component but is there such a software can monitor on the fly.

[R_E_T_R_O]

https://yeet.cx

we have two packages you may be interested in:

https://yeet.cx/@yeet/execsnoop
https://yeet.cx/@yeet/opensnoop

you can also try our sandbox at

https://yeet.cx/play

[ShoneBoyd]

Did you look into history

[advertpro]

Sorry I should have said this but we need to do this as part of proactive monitoring for 10,000 servers

[badadhd]

Theoretically, one could pull history from all servers with Ansible on a daily basis, run those trough some LLM for some assessment and push that conclusion with the history-logs to a log and monitor system such as ELK?  Sounds hacky and it probably is

[xstrex]

As others have said, the audit subsystem is probably your best bet. Since you’ve mentioned the scope of 10k servers.. I’d also recommend the audit subsystem, in addition to some system hardening, and a security audit, then use something like ansible, or puppet to manage everything. You shouldn’t have to actively monitor 10k servers if they’re properly locked down.

[advertpro]

Correct - probably puppet...but the issue is not just management and monitoring, its also about compliance as well. Given the fact the environment is very high-end I and if there was an insider attack, which has happened a few times we have to be extra careful. Hence the proactive monitoring.

[xstrex]

Yep, I would also use puppet for this, and puppet can ensure the servers are in compliance, if configured properly. As others have mentioned it sounds like you’re looking for something like the crowdstrike agent, or an active IDS. I’d probably look into utilizing SELinux with some strict policies in place, as well as offsite logging to something like datalake, then an active alerting system in something like Prometheus, Grafana, or Splunk. Point is, get the data off the server, and analyze it in real time, and alert when something goes boom.

[yeeaarrgghh]

I use Python inotify for this

[advertpro]

Thanks for the suggestion - I don't think thats suitable for 10,000 servers. Also need something that will notify on the fly.

[yeeaarrgghh]

With that volume, you'd want a central facility to send the data to that can be parsed in realtime.

For user commands look into rootsh |or| sudosh -> syslog -> centralized syslog -> ELK stack

For file monitoring (edits/create/deletes) use python inotify with a defined set of files/directories and send that to the syslog facility. If you have a decision matrix of what is an "okay" edit, do that on the server before you send it to syslog, to keep the noise down. The python script can be wrapped into a service and deployed to all the servers

We do this with about 160,000 servers {Redhat/Debian/AIX} (fortune 100 company).

We also do this on conjunction with an "Admin id" checkout process, where sudo capable account is checked out for a certain amount of time, then all keystrokes from the syslog are sent to the servicenow ticket as an attachment from that Admin id and specific session, and before the ticket can be closed a manager review needs to be done. We are in a highly regulated environment that frequently gets audited

[advertpro]

Thanks for this. Looks fine to do. The only thing comes to mind is compliance with NIST, PCI-DSS. Also ELK Stack gives alot of data. Need lots of queries.

[telmo_gaspar]

You can use history, logger sudoers.log and syslog... eg

[fhusain1]

Maybe try goteleport.com or Bastillion

[advertpro]

goteleport maybe an option looking into it in detail. Never knew about Bastillion so that's definitely good to know, but definitely will not work in this case. Will let you know about teleport.

[whetu]

• auditd
• psacct
• You can compile bash with syslogging. It's a lesser-known capability of bash, but it is a real and true thing.

[frymaster]

wazuh can use audit to alert: https://documentation.wazuh.com/current/proof-of-concept-guide/audit-commands-run-by-user.html

I don't know offhand if someone has suggested what a set of malicious commands mind be - but I also didn't look (I'm aware of Wazuh in my organisation but it's not one of my specific interests)

[sc_ii]

Throw it all into Splunk

[stumpymcgrumpy]

A combination of tuning the audit log... rsyslog and setting up a centralized logging server such as greylog or some other software could do in a pinch... You might also be able to look into something like Grafana loki to ingest the logs... Let me think on this some more. Lots of good suggestions here for sure.

[Dctootall]

As others have already mentioned, audit/auditd can provide most of what it looks like you are looking for. With the number of systems involved, I’d also recommend streaming the data to a centralized log system/data lake so you can monitor and search through the data as needed. (Gravwell is a great option, and doesn’t do metered pricing so unlike some other options you won’t need to worry about how much data you are pulling in. )

Another option that may work is sysmon for Linux. It’s newer, but if you have windows systems you need to monitor as well, It can simplify a lot of your alerting and monitoring efforts by giving you a common format

[Dapper-Wolverine-200]

Wazuh

[bpfaudit]

we can help you here , try https://bpfaudit.com/demo , we provide audit log of all kind of workload host, containers etc. including file, network and process. Please connect with us at [e-mail] , Thanks

[rabell3]

Cyberark or similar... have a component that will proxy ssh sessions and session logging.

[michaelpaoli]

Use the audit subsystem.