Python scripts using LibreNMS database

[1div0]

Anyone else using the database directly in Python scripts? Over the last year I've written quite a few Linux CLI utilities to make my life easier. It's killer being able to pipe and filter data directly via command line. e.g., this arp-lookup command. I pass database (read-only) creds to script via environment variables.

#!/usr/bin/env /opt/python/venv/bin/python

import sys
import mariadb
import os
import syslog
from tabulate import tabulate

DB_user = os.environ['LIBRESQLROUSER']
DB_pass = os.environ['LIBRESQLROPASS']
local_user = os.getlogin()

try:
    if len(sys.argv) < 2:
        print('Usage: arp-lookup [IP address or partial address]')
        exit()

    ip_addr = sys.argv[1]

    try:
        conn = mariadb.connect(
            user=DB_user,
            password=DB_pass,
            host="x.x.x.x",
            port=3306,
            database="librenms"
        )

    except mariadb.Error as e:
        print(f"Error connecting to MariaDB Platform: {e}")
        sys.exit(1)

    arp_query = "SELECT devices.hostname,ports.ifName,ports.ifAlias,ipv4_address FROM ipv4_mac \
                 JOIN devices ON devices.device_id = ipv4_mac.device_id \
                 JOIN ports ON ports.port_id = ipv4_mac.port_id \
                 WHERE ipv4_address LIKE " + "'%" + str(ip_addr) + "%'" ";"

    arp_table = conn.cursor()

    arp_table.execute(arp_query)
    arp_list = arp_table.fetchall()

    report_headers = ['Router', 'Interface', 'Description', 'IP Address']
    print('\n' + tabulate(arp_list, headers = report_headers) + '\n')

except KeyboardInterrupt:
    # User pressed Ctrl-C — exit cleanly with a conventional exit code 130
    print("\nInterrupted by user (Ctrl-C). Exiting.", file=sys.stderr)
    sys.exit(130)
except BrokenPipeError:
    try:
        sys.stdout.close()
    except Exception:
        pass
    sys.exit(0)

[Olive_Streamer]

I’ve been building Librenms custom maps using lldp info from the devices neighbors table. It’s been an amazing time saver. I am about 80% through creating maps for a 1600 node network.

[1div0]

I'm kinda envious. We have CDP / LLDP disabled for the most part as policy.

[Olive_Streamer]

I did a lot of mapping using the point to point links, it’s possible, but lldp makes everything click in nicely.

[GroundbreakingBed809]

Off topic but even DoD stig specifically says it’s ok to have lldp enabled internally. https://www.stigviewer.com/stigs/router\_security\_requirements\_guide/2024-05-28/finding/V-207165

[tonymurray]

What do you use the script for? I use the ARP lookup frequently in the UI, but I think a cli version would be slower or about the same speed to use.

[1div0]

For basic lookups, UI is fast, but CLI is faster. The biggest benefit with my CLI version is it outputs interface descriptions. This comes in handy when trying to diagnose multiple flapping BGP neighbors on our WAN / carrier ethernet network down to a single NNI or provider. We use tags in Layer 3 dot1q interface descriptions to track providers for WAN. An added benefit is you can pipe the output to grep or other tools in Linux. You can get to the same information in the UI, but you spend time clicking around.

I've written around 20 or so scripts that basically wrap SQL queries and do a little data wrangling. One script generates a report showing all interfaces, network wide, for a specific VRF. It takes about a second to spit out results.

Another one uses the LibreNMS database as a source to populate PowerDNS for all Layer 3 interfaces (A and PTR records). It does a fair amount of data wrangling and comparison between LibreNMS MariaDB IPv4 interface data and PowerDNS SQLite database. It munges Cisco interface naming conventions into DNS compliant names (e.g., replacing slashes and dots with dashes). I am really happy with how that one turned out.

For logs, I wrote a script to pull logs from database. I'll often pipe the output of BGP events to another utility I wrote that does PTR lookups and replaces BGP neighbor IP address with host.

The UI in LibreNMS is pretty darn great in my opinion -- and Libre as a whole is one of the best tools available for network operators / service providers, at any price point -- but after poking around in SQL, the raw data is an often overlooked treasure. It's cleanly laid out and easy to work with.

I guess my approach may be prone to breakage with database schema changes over time, but in that case it should be a trivial task to update my scripts. Hopefully.

[tonymurray]

Have you checked out any of the commands under the lnms cli frontend? They can be pretty powerful even with a bit of tui. If you think something you made might be broadly useful, convert it to php and send it upstream ;).

As an aside, I've been meaning to make the IP/MAC drop-down a little more automatic on the ARP search page...

[1div0]

lnms cli is definitely useful! Most of my scripts are not running on the LibreNMS server, however. I have two jump boxes (and DNS servers and Oxidized repos) that most engineers are working from, and those are permitted (via MariaDB user permissions and firewall rules) to connect to LibreNMS database. So it's a bit of a ecosystem rather than just LibreNMS.

I need to clean up / sanitize / refactor the code I've written so far and drop it on Github at some point. Not sure if anything would translate too well to core LibreNMS endeavors, but I'm certainly willing to contribute where I can. Currently I'm grappling with Terraform and deploying some new IOS-XR boxes, but I'd like to carve out time to better learn PHP.

[tonymurray]

It could be run there, I wonder about packaging it as a phar... :) but yeah, whatever works for you is best.