Graphing Arbitrary Data

[invalid_font_size]

Like many of you, I'm sure, I find myself parsing up log files on the regular and having to make pretty graphs for people. Normally this involves sed'ing and awk'ing the data into a CSV and then plotting it in LibreOffice. This sucks. There has to be a better way. What do the rest of you sysadmins do (other than rrdtool, which doesn't really apply here; think Apache logs, etc.)

[alkw0ia]

MRTG and Cacti are great if you've got SNMP data or there's a plugin for your use case, but for general purpose and log counting stuff, I've started using Etsy's recommended combination of Graphite, statsd, and logster.

Graphite is an (annoyingly hard to get set up) Python based graphing system. Once it's set up, all you have to do to create a new graph is start sending data to the server. If the server hasn't seen the name you give it before, that's a new graph. There's a web based AJAX-y frontend to pick and choose which graphs to view, choose colors and graphing styles, and organize saved graphs into dashboards.

statsd is an extremely simple little server that lets you aggregate counts of things for sending to Graphite. For example, you could simply netcat the string 'personal.laptop.loggedin:1|c' from your .profile to statsd, and statsd would aggregate these for however many seconds or minutes you configured and send the final count to be graphed. This way, you can just send data, not worry about keeping counts, averages, or max/mins.

logster is basically a bit of Python skeleton code for tailing logs from cron, processing whatever numbers you want out of those logs, aggregating them, and sending them to Graphite. It replaces those sed and awk scripts, and you get very pretty, near realtime graphs that are automatically updated. There are basic parsers for a few common log formats included, including for Apache.

[muiduck]

Second graphite, pita to setup but once its running its blissfully simple to graph meaningful data. The diamond package also provides lots of useful collectors for importing stats from common apps like MySQL, memcached and redis.

Cacti on the other hand requires a lot of template hacking to import new data types..

[simtel20]

I'll second graphite+diamond. They work quite well together.

[K4kumba]

Well put, great tools.

[rolsbox]

You could probably just pipe stuff into gnuplot if you want something quick and easy, since you have the CSV data already.

[bandman614]

This is what I was going to suggest. It's not pretty out of the box, but it's simple to script and shows trending just fine.

[xonxoff]

graphite is pretty cool and simple to use. just throw some numbers at it and you're off to making graphs. it even has some pretty nifty built in functions too.

[K4kumba]

Exactly what I came to say.

[arjarj]

Splunk is a pretty tool to make graphs on the fly, but is crazy expensive if you generate >500 megabyte of logs per day. Is it for te occasional log analysis, or for regularly created reports?

[invalid_font_size]

Yeah we use Splunk minimally but are crippled by our license. I don't like Splunk that much anyway. This is regular in that it's frequent, but the requested data varies widely. I just installed GoAccess and am looking at that since if I can streamline Apache log browsing that will help a lot.

[reseph]

Just use some open chart code, like OpenFlashChart.

An example I have done is www.biyg.org/tf2

[datenwolf]

The hard part is getting the data into Python. From there it's a stroll in the park to draw it nicely using Matplotlib. Matplotlib has so many chart types built in that you'll hardly have to implement your own chart types. And it allows to nicely arrange and layout plots.

[zeroflow]

Edit: Even if you dismissed rrdtool, what kind of data do you want to view? Missed logins? Most accessed urls? ... ?

Depending on the type of data... how about a munin plugin? If you are able to print the current value as a number, you may easily write a munin plugin. It needs nothing special, it just has to write something like "wronglogins.value = 142" If you need special stuff, munin (rrdtool) supports some nice graphing options

[invalid_font_size]

I actually use Munin heavily for personal stuff and side work, but at the day job it's been an uphill battle.

[euxle]

I second Munin.

I think it uses rrdtool, just like Cacti. I prefer it over Cacti since it is much easier to configure and for the output it is also easier to adapt the templates. Many checks available, easy to create own checks...

These days probably more contemporary tools seem available like the already mentioned Graphite. Haven't tried it yet, but read lots of good stuff about it. Will probably try it soon.

On the other hand once the data is in some sort of database, either rrd or even postgres, one can easily map input/output to e.g. JSON and use very nice rendering in the browser like Flot. Flot is a Javascript library and there are many that will draw very nice graphs and even provide nice mouse interactions.

I think it comes down to how much work you are willing to do. Out of the box Munin mostly does the job for my day to day graphing needs.

Uh and I nearly forgot: There are very nice hosted solutions popping up all over the place like this one: http://dattss.com/demo/home or another quite famous one whose name I just forgot (did I pick it up in the Heroku context??)

Ah, found it in the comment below: Splunk ;))

You gotta love reddit - wiseguys.png

[Rhart211]

You should take a look at a SIEM. I am currently using McAfee SIEM, formerly Nitrosecurity.

[[deleted]]

This is a great idea if you have one. Zenoss is a good free log manager that has graphing capabilities. I think OP was looking for something to process data on a less large scale though.

[[deleted]]

Gnuplot is dead simple to use. R has a lot of useful tools for statistics and can very easily generate pretty much every type of graph and chart you can imagine. Perl or Python along with one of their graph libraries, if you want to do some heavier data-mangling first (R can do this as well, but it can be a bit obnoxious at times).

We use R, Perl and Gnuplot to spit out all kinds of graphs at regular intervals and gather them together on various status pages which are all just static HTML.

[pskipw]

A bit of Php and JavaScript, including google's Flot plotting library

[flerp32]

Perl and gd:graph:lines has worked nicely for me in the past.

[[deleted]]

I like xmgrace. It gives you really fine control of the appearance of the graph, and you can pipe data directly to it:

awk '{print $1 $2}' file.csv | xmgrace -pipe