LibreNMS syslog.ibd cleanup

Recently I ran into an issue where a librenms install was taking up a crazy amount of disk space. This was tracked down to the syslog.ibd file . Even thought I had set my options to be less than 10 days per this link I still was having a huge file.

Here is what I did to fix it. My root partition was too full to start MariaDB. I went into var and cleaned out enough log files to make space to start MariaDB. The following are tee commands I ran on CentOS 7 to fix it.

mysql -u username -p

Once at an MYSQL prompt after logging in I issued the following command to verify I could see my librenms database.

show databases;

The I issued

use dbname;

In may case it was “librenms”. Once connected to the database I ran the following command

DELETE FROM syslog WHERE timestamp < '2021-1-28 08:00:00';

This commands removes all syslog entries from the database before the date and time specified. In my case this was close to 40 GIG.

After I restarted MariaDB, ran and all was good.

LibreNMS and Cambium Networks

What Can Libre do for the Cambium operator?

Number of registered SMs. This AP has 7


Interface traffic

Number of GPS Satellites tracked


Libre can be setup for thresholds.  Any of the above graphs can be set up to alert based upon a threshold.  You could setup an alert that says if SMs on an AP drop by 10% kick out an alert.  This could alert you to possible frequency issues.   You could set a threshold for if traffic reaches a certain level.

The importance of Network Monitoring Systems (NMS)

One of our open tickets on MidWest-IX is a member reporting slow speeds on their exchange port. After having them send us some data and a few e-mails back and forth we began looking at their switch port on the fabric.  Right away we noticed errors on the port. After a counter reset the errors were still incrementing

 19 runts  0 giants  1210 CRC  0 no buffer
 1329 input error  0 short frame  0 overrun  0 underrun  0 ignored

This led us to look at our LibreNMS data for this port.  A quick look shows on October 31st the port started seeing input errors.

By drilling down we are able to see exactly when this started happening

We now have responded to the customer to see if anything changed that day. Maybe a new switch, new optic, or software upgrade.  By having this data available in an NMS we were able to cut down on troubleshooting by a huge margin.  We now know when the issue started and are closer to the root cause of this.  Without this data, we would be spending more time trying to diagnose and track down issues.

Cisco 2960X I/O usage

While double checking some stats on a network I came across this in Libre.   84% is usually something that would cause me to be alarmed, as Libre is trying to tell us.

After some research, I found the following.

While it is not documented, it was noted that this was by design and that it would not affect the switch as the switchport becomes more and more loaded.

The switch allocates dedicated memory to certain processes / resources by default and then additional resources when the configuration is added. This ensures proper functionality and is again by design.

The I/O Memory pool buffers information transmitted to and from the CPU, and does not affect the actual forwarding of packets on the switch.

Translation: The switch uses up these resources by default, even if they aren’t all being used.  Think of it as setting it aside for future use without dynamic allocation of them.

Thresholds for Microwave backhauls in Librenms

If you are running Librenms this video will help you learn how to adjust the thresholds for signal strength when it comes to microwave/licensed backhauls.  This video focuses on Mimosa because that was the quickest handy thing.  Cambium and other manufacturers will be slightly different due to chains being combined and other things.

Helpful Librenms commands for fixing broken installs

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.