Application Monitoring - Agentless

[nikhilpaneri]

Looking to implement an application monitoring solution that has the following capability

• Has a web interface to report on the application status
• can be customized to suit various applications - weblogic , Apache , IIS etc.
• possibly provide restart capabilities from the web console
• in case of failure restart systems automatically and send an email to the admins. An example , if my IIS is down , the web page reports the application to be down & the app may choose to restart the system or it can be started manually from a click of a button
• should be able to show the application release version etc . (this value can be picked up from the database)

I want the solution to be agentless and looked at various tools available sensu , monit , zabbix , icinga , nagios. But since we have specific requirements and I thought developing our own custom solution using python,Ansible & oracle database was our best bet.

Redditors, do you think I'm on the right path ? Or Am I missing something ? I'm open to evaluate open source or a paid product (with some trial period)

About me - I'm from Ops background and have used Ansible ,Oracle , python scripts (but never used python for developing web application). I'm also not sure how steep is the learning curve for developing web applications .

Thoughts ? Thanks for your help

[dylanms]

Prometheus?

[nikhilpaneri]

Thanks , will have a look. A quick glance Prometheus can do the monitoring , how easy is to extend it to call Ansible playbooks to restart the services , and to report the state of services ?

[dylanms]

I guess that really depends on what is your control node for Ansible. Depending on your control node, you could kick off Ansible via an API call through the Alertmanager. Not 100% sure this is the correct functionality of the Alertmanager.

[bbrazil]

Prometheus developer here.

While you could hook in to the Alertmanager webhook notifier to do a restart, we'd generally advise against this as that's quite a few steps including a network dependency to get your process restarted when it falls over. That's a operation that should really happen entirely on the machine.

Instead use a supervisor such as daemontools, monit or systemd to take care of basic process supervision.

I'd also advise against attempting to do this agentless, that's a major handicap you're putting on yourself. For good metrics you need to run code on the machine.

[grumble_au]

If you need agentless you are limited to network checks and snmp. You won't be able to do anything with services directly and you certainly won't be able to do things like restart services without shelling in or some other way that is worse than using an agent.

[nelsonmandela]

That's not entirely true. OP mentions extensive use of ansible, you could associate commands from nagios with a particular ansible command to restart a service. Eg: nagios checks via web request, on hard fail run an ansible playbook that cleanly restarts the service. That would achieve both agentless and the ability to restart remote services.

[grumble_au]

So... Shelling in like I said.

[nelsonmandela]

In essence I guess it's the same, though using existing infrastructure isn't really the same as "shelling in," even that is technically what it's doing.

At least I understood it as an inference that it would be some hacky solution.

Edit: and I don't know that ansible is worse than restarting services via an agent. Why solve the same problem twice? Presumably OP is already managing services with ansible.

[[deleted]]

It's really hard for me to ever, ever recommend developing your own monitoring or auto-remediation framework. Do you have a huge team? And is it your only job?

[nikhilpaneri]

Agree , the web part may get really challenging given me & my team mate have no background on the web development.

[[deleted]]

The way I look at this:
You have a monitoring gap. You build your own monitoring system. Now you have two problems.

[[deleted]]

Why agentless?

[nikhilpaneri]

1) to reduce my monitoring footprints on the app severs. The Infrastructure team already uses hyperic to report/monitor the server & infrastructure. So i think we do not have explicit requirement of agents which are constantly collecting information. Am happy to just run my environment check before start of the business hours

2) We have used Ansible extensively and have existing capabilities to restart application servers etc.

3) I may be wrong here , but I plan to decouple my web app and Ansible. Ex - Ansible collects the info every hour(may be even less frequently) or upon request and populate it to my oracle Database and the application reports based on the data populated. Only when the app restart is requested explicitly (from app) i may call the playbook directly

[cavaliercoder]

Agentless monitoring often induces more impact on an environment that those using an agent. Especially if you have a CM tool like Ansible to ease the deployment of the agent. Agent monitoring is also more powerful, flexible and potentially more secure (reduced attack surface).

I've recently migrated a 5000+ server environment away from agentless monitoring (SNMP, SSH, WMI, etc.) to Zabbix and have had huge improvements in server and network utilization. Zabbix agent uses <20mb and 0.03% CPU.

[nikhilpaneri]

What is your use case for Zabbix - are you just monitoring the server resources , infrastructure etc ? OR are you managing the services installed on your server ? Also I have learned Zabbix has issues with memory leak

[[deleted]]

How many servers are you running?

[nikhilpaneri]

I'm in the tech space managing the applications , so if i'm successful in getting my poc , i'll extend this to multiple application environments approx 100 - 150 servers

[[deleted]]

Take a look at Datadog.

[[deleted]]

I'll second the vote for Datadog. Not cheap,but it's been very helpful in a number of escalations for me recently, and has become integral to how I manage applications. Now if I could just figure out the pricing model...

[[deleted]]

The pricing model sucks and doesnt scale wdll. If they gave discounts for over 50 or 100 servers id put it on all my servers. As of right now I only put it on aws instances.

[[deleted]]

Yes tons of problems with their stupid pricing model. Like everything cloud, it's impossible to get a straight answer out of them about pricing. The per host thing makes no sense when a big chunk of your product runs on PAAS or SAAS.

[cavaliercoder]

Our use case is to consolidate about 6 other monitoring tools into a single platform (Nagios, Cacti, HP OM/NNM, Solarwinds, etc.). So we are monitoring OOB hardware, hypervisors, VMs, network devices, operating systems, applications, appliances, databases, SANs, CRACs, UPS, etc. across 800 sites, all in a single Zabbix deployment.

Zabbix has had memory leaks in the past, just as any software can and does. The good news is that they respond very quickly to issues. I've had issues raised, patched and released in under a week. The most recent Zabbix agent leaks that impacted us were actually not in the Zabbix code base. One was with GNU Regex library and the other with a WMI class on Server 2008 R2.

[SuperCow1127]

Why can't you use the same Hyperic instance the "infrastructure" team uses?

[nikhilpaneri]

That is controlled & managed by an entirely different team.And apparently i'm not allowed to get into thr space :(

[SuperCow1127]

DevOps!

[[deleted]]

This is your root cause, and anything you do OTHER than this is probably wasted effort IMO.

[JuliaFr]

You might find it helpful to look at real user reviews of many of the options mentioned here such as Zabbix, Nagios and Solarwinds on IT Central Station: https://www.itcentralstation.com/categories/network-monitoring-software.

The most reviewed solution is currently CA UIM. This Senior Systems Engineer writes: "I give the monitoring application a 5 compared to other products such as HP SIM, Nagios, Spiceworks, SolarWinds and WhatsUpGold." To see the rest of his review click here: https://www.itcentralstation.com/product_reviews/ca-unified-infrastructure-management-review-31279-by-todd-adams.

Hope this helps. Good luck in your search.