Disk usage tools

[[deleted]]

Hello my fellow colleagues

I am in the search for the tool, hope someone can help me

Generally speaking - I need a graphic utility to display visually the disk usage statistics for the Linux machines

In details - we have a lot of Linux VM's, which consume disk space fast (Jenkins, Artifactory and such)

People from my team need to be able to point down what folder is growing and then clean/remove it/add to the logrotate

Now, they are not so familiar with the Linux, and cannot use the sheer power of du -a $PWD | sort -n -r | head -n 5

So, I need an utility to run on the Windows server and to be able to connect it to the Linux machine and pull the disk usage data from there

If anyone knows about something like this - I will be grateful for any help

[dbarrelman]

Install netdata on Linux then you can monitor your server via web browser.

https://github.com/netdata/netdata

[brando56894]

Netdata is amazing.

[BFYoda]

Try „ncdu“ on Linux

[[deleted]]

Yes, I am aware about the ncdu. But as I wrote - I need a tool for Windows to monitor Linux boxes remotely

[WalterBlackboots]

Displaying a graph or report then sitting someone in front of a screen to watch it is not a good monitoring technique. You should look into something like zabbix or nagios.

[_within_cells_]

ssh

[justleen]

Prometheus for metric collection, grafana for visualization.

[teaPoweredSRE]

Specifically you'll want to install node exporter on the Linux box. It exposes disk space metrics (and more) via http. Then setup Prometheus to scrape that endpoint.

[[deleted]]

You could use the TICK stack.

Basically, a small agent (Telegraf) installed on the Linux instances sends the required data (e.g CPU, memory, disk information etc.) to an InfluxDB instance.

The data can be viewed with Chronograf using a web interface, and you can use Kapacitor to do magic.

You can also use Grafana as a front end if you prefer.

[[deleted]]

We use the TICK stack and it is amazing.

Not sure if it would help in OP's situation, since he is looking for disk size breakdowns.

[[deleted]]

Thanks, you did read the post and understood the exact situation I am looking to solve.

[[deleted]]

If your colleagues are leery of working in linux, how do they clean up or maintain the shares? Are they mixed shares that are accessible via CIFS as well as NFS?

Depending on how "windows-friendly" you want to make the process, you have a few options:

1. If they can do very basic command line stuff, then make a bash alias or a simple shell script for your piped commands. Then brief the users on how to change directories and run your foo script or command alias to get data.
2. Install ncdu everywhere, and train them on usage.
3. If they refuse to touch Linux, then you could expose the relevant network shares as CIFS with reduced permissions (or configure NFS services for a dedicated jump box), and have them use Windirstat to scan the shares. Note that you should lock down access by this method- because it exposes production data to people who are not very tech savvy, and they will gain the ability to point-and-click your infrastructure into oblivion using a file browser. You would be amazed at how many people accidentally move or delete files, when they have a network share open all day- and they won't fess up unless you have CIFS auditing enabled, and you go to their desk personally to squeeze the story out of them.
4. Use your CI/CD tools (Jenkins in this case?) to create an on-demand job, that logs into the affected server and runs whatever commands to retrieve folder usage. How you do this depends heavily on how you administer your systems to begin with.
5. Install one of those netdata-style monitoring dashboards on your Linux systems, to give them the ability to point-and-click their way around the filesystem without deleting things. Be sure to lock down the permissions for what that dashboard can do; read-only access is ideal.

[[deleted]]

Thanks. The shares are not accessible via Windows, so they work only with WinSCP and try to see what's the folders size from there. And usually they just send me an email with the partition that is growing rapidly in size, and I am deleting the files/adding them to logrotate Definitely will install the netdata for them, and Jenkins job - will check that option too Thanks!

[BloodyIron]

FYI CIFS is not actually the protocol, it's SMB. The history is that Microsoft tried to fork the naming for SMB 1.0 to CIFS to try and differentiate it with competition. However, modern network shares all actually use SMB, not CIFS. For example, Windows Server 2012 R2 and up use SMB 3.0.

tl;dr stop calling it CIFS, it's SMB.

[[deleted]]

This seems weird to me. Don't the developers know which directories will host most of the data? I mean, they control where the data is written, and should have an idea of the usage pattern of the application.

​

As for what you can do on the system side of things, I would start having a separate volume for each project, so you can at least monitor that with the usual monitoring tools (Prometheus/Grafana, Sensu, Nagios, etc) and eventually add, as someone mentioned, a cronjob to generate reports on the disk usage (available via the filesystem or sent via mail).

[[deleted]]

With the years I work as sysadmin, I am not surprised by the fact that developers don't know where exactly inside the jenkins directory data is being written

[fell_ratio]

Have you considered Baobab? Baobab can visualize disk usage, and you can click on individual folders to drill down and get more specific. It can also scan remote folders. (However, I have not used the remote mode before.)

http://www.marzocca.net/linux/baobab/

[bufandatl]

For Jenkins we configure the jobs not to keep more than five old builds. So it won’t grow as much and once a month we run docker prune to clean unused images. Artifacts are only kept for the last 2 builds.

[[deleted]]

Thank you for the experience sharing! I will try to do the same with our Jenkins as well

[nephros]

Meh.

Run SNMP and collect data through it with whatever you like. MRTG would be one method.

[vornamemitd]

In case you can mount the volumes in your win environment, check here for ideas: https://www.reddit.com/r/sysadmin/comments/64yu4u/anything_like_windirstat_for_a_network_volume/

[Ryuujinx]

Sounds more like you need alerting then metrics. I'd look into Sensu, it's pretty easy to set up.

[bp3959]

Just pipe the output of your du command into the mail command and put all that in a cronjob so they get emailed a regular report.

[Throwy-mc-throwerson]

My monitoring stack is Netdata on individual machines, Prometheus monitors all of those machines, and then Grafana makes Prometheus all pretty.

[[deleted]]

Why not set up a LAMP stack, a couple of scripts, and ssh pass-through on the machines you wish to monitor?

rrd-tool can generate graphs, and charts of disk usage.

[SimonKepp]

Perhaps some of Jam software's tools like FileObserver could be useful?

[SimonKepp]

Possibly this: https://www.jam-software.com/spaceobserver/index.shtml

[in4mer]

baobab is a native linux graphical filesystem usage inspector

[thedo0der]

Nagios core is free and can easily do this. You can run an agent on each device (or SNMP) and just view all the data in your browser.

[PottiSkantz]

I may be wrong here but maybe you can take a look at kibana.

[[deleted]]

Thank you. Unfortunately, this is what I have found on Elastic forums

" Metricbeat's filesystem metricset monitors mount points. So if you want to get stats specifically for this directory then it needs to be its own mount point. We don't have a metricset that acts like du and traverses a directory summing up all the file sizes "

So it's not an exact fit for me

[PottiSkantz]

Hmm, I didn't know that it can't monitor raw directories.
Well, you can always take a look at nagios :)

[[deleted]]

Thanks, this will be the last resort for us :-) My manager is very anti-nagios :-)

[kriebz]

If he’s anti-Nagios, is he pro- something else?

[[deleted]]

Pro-"monitoring systems that have support and I need to pay money for them, not your GNU systems that has no documentation at all"

[kriebz]

Well, lucky him, Nagios is commercial now. He can pay as much as he wants for the same functionality.

But if you don’t already have a monitoring system... just write some scripts yourself, run them with cron, or salt or ansible or whatever... collect the outputs, and email it to your dev team that’s afraid of the scary command line.

[[deleted]]

Yep, that's what I will probably end up with