I used to use Observium for network monitoring. After evaluating a couple of options – Cacti, Zabbix, Nagios – I found Observium, and loved its beautiful, well organised and extremely usable interface. Observium has two main drawbacks for me though:

• Like many FOSS products, it has a ‘community’ version and a ‘commercial’ version. Even though the commercial version is only ~$500 p.a, the community version is  – suspiciously – not up to date with it (and does not accept community patches..), and missing important features like any kind of alerting.
• The main developer has a bit of a history of being unstable, and withdrawing support for devices if he finds the users annoying. It’s his prerogative, but I don’t really want to invest in that.

A few months after running Observium, and being largely happy with it despite the shortcomings above, I found LibreNMS. LibreNMS is an earlier forked version of Observium, and basically exists to ensure the project is run by friendly, community-minded folks. They’ve also implemented a lot more functionality, making it a much more complete and rounded product.

On my LibreNMS server, I also installed and configured Oxidized, for version control of network device configuration. While tried-and-tested solutions like RANCID exist, Oxidized is a Ruby-based application with a web component and a RESTful API, and it integrates into LibreNMS.

Both LibreNMS and Oxidezed are easy to install out-of-the-box, by following the instructions. The integration of LibreNMS and Oxidized took a bit more fiddling. The end goal is to have:

• Network utilisation (traffic and bandwidth) monitoring, loss and outage monitoring, uptime and availabilty monitoring
• Alerting on rule-based condition triggers
• Configuration backup using Oxidized
• Up/down monitoring of 3G hosts, by disabling all polling modules except O/S.
• Notification via Slack transports, to #netadmin Slack channel

Oxidized Configuration

After installing Oxidized, including creating the local git repository, an API key must be created for it within LibreNMS by navigating to http://librenms-server/api-access/. The API token is needed so Oxidized can obtain a JSON list of devices from LibreNMS.

Now, Oxidized is configured by editing the YAML file from: /.config/oxidized/config:

In this YAML document  the source: designates the origin of the devices. The output: designates the git repository for storing device configurations.

One annoyance is that at the moment, all network devices must use the same credentials (line 1 and 2 above) which is not ideal. LibreNMS will probably never support the adding of device credentials. Perhaps key-based login – or a separate database of device logins – would be a workaround in future.

Email configuration

Fortunately, really easy. I just pointed LibreNMS at Sendgrid. Sendgrid offer 12k emails for free each month for free, so for anything where you need good deliverability, it’s a no-brainer.

Slack integration

For Slack integration, we first just create an ‘Incoming WebHook’ in Slack:

Then we configure that hook, channel and username in LibreNMS as follows:

Other Tweaks

For monitoring 3G connections, I often don’t want or need the entire SNMP tree to be polled each time. In order to save on bandwidth (which is scarce on some o these 3G sites), I’m content to just use ICMP pings. To do this, you need to disable every poller and discovery module in LibreNMS for a given device other than the module ‘os’.

I’ve found that if I poll 3G devices, I often get a lot of false up/down alerts. To mitigate against this, I set a higher fping timeout and retries value in config.php:

Lastly, I found that every few months, I would start getting a HTTP ‘error 500’ messages and the web interface was unavailable. This was happening because PHP was exhausting its allowed memory limit. Increasing the limit in php.ini each time was a bandaid solution to making it work.

I think it is the result of LibreNMS generating ports each time a VPN interface on a Mikrotik goes up/down, which is very often – resulting in thousands of extra ports in LibreNMS. These can be purged from the //nms-server/deleted-ports/ page.

On the advice of the guys in #librenms I added the following lines to config.php, which prevents the extra ports being generated:

To fix some odd graphing errors, I also enabled RRDTune under LibreNMS’s global options.