Servers

DNS in ISP networks and why you should care

If you are not familiar with how DNS works, please go and read this article and watch the accompanying video.

In this article we will talk about the different types of DNS servers an ISP will encounter and the design considerations of implementing them into your network. Let’s jump into the three types.

Cache servers
local DNS (LDNS) servers
authoritative DNS (ADNS) servers, of which the Root and Top Level Domain (gTLD) servers are special cases

Let’s go into detail on what each of these is.  We will start at the top and work our way down.

Authoritative Name Servers

An authoritative name server, just like the name says, is the authority on the website or resource it is responsible for. Without authoritative name servers, the DNS hierarchy would not function. You would have no way of telling an incorrect answer from a real answer.

A special kind of authoritative name server is the root nameservers, which are responsible for the Top Level Domains (TLD).  These are better known as .com,.net,.org, and so on.

Local DNS resolvers
These are servers run by Internet Service Providers (ISPs) and other entities such as Google (8.8.8.8) and Cloudflare (1.1.1.1). These are machines running BIND, Tiny DNS, or other software.  It is best practice to not run your authoritative name server on the same instance as your resolver.

Local DNS resolvers sit on the ISPs Internet backbone on fast connections to the Internet.

Caching servers
Caching servers are more typical in a larger ISP network with large concentrations of customers n certain geographic areas. .  Since DNS is such an important function, ISPs want their clients to have the quickest access to DNS lookups possible. In larger networks, this means putting caching servers as close to customers as possible. DNS lookups do not take up much bandwidth, but a slow DNS lookup can affect the user experience. If you have ever entered an address or clicked a link and experienced a “Looking up xyz.com” message for more than a second or two you are experiencing a slow DNS.

So why is this important in an ISP network?
As mentioned above the quicker you can get DNS lookups done and over with the better user experience your customers will have. You do not want your clients waiting 3-4 seconds for each lookup.  While you might have the fastest Internet on the planet, your customers will always be waiting on these requests, which will cause them to think things are slow.  And technically they would be right.

To speed up a large ISP network you will need to design DNS resolvers/caches into your design.  These can be at major traffic points on your network, or at each POP location. Part of this depends on your IP design, while some of it depends on the overall design of your infrastructure.  As networks flatten out due to things like VPLS and EVPN the need for lots of resolvers in an ISP network fades away. A general rule of thumb is to deploy DNS resolvers at each customer gateway.  This can be the wireless tower they receive their signal from, the local DSLAM, or a fiber node at the edge of your network.  By deploying resolvers at the customer gateway you are giving them fast access to cached DNS lookups. This can also lead to more complex troubleshooting should a cache server fail

What do I mean by cached?
A DNS resolver has two major functions in life.  It takes a request for a website, in our case j2sw.com, to look up the appropriate IP address and returns that information to the client.  The whole www stuff is for us, dumb humans. DNS translates www.j2sw.com to 209.209.44.4. This is what the computer and routers understand.

Secondly, a DNS resolver caches previous lookups.  This can be configured for minutes, hours, or days.  This speeds up any subsequent lookups because the resolver can serve it up locally without having to go clear across the internet to look it up and wait for the return. Once it has done one lookup, it will cache that lookup for any client who requests it after.

So, how does an ISP implement all of these?

Many modern routers have the ability to cache lookups and act as a DNS resolver.  Mikrotik, Cisco, and others all have this ability.   Even a Raspberry PI has enough horsepower to be a DNS resolver.  These can be deployed at each POP site and customers can point to them as their primary DNS servers.  This means their routers or PCs will look to the closest DNS resolver first.  If the resolver has the answer in its cache, it will serve it up right away.  If not, the resolver can go on to the next step up.

The next step is local resolvers the ISP typically runs.  There are lots of arguments for why an ISP should run their own instead of pointing to Google (8.8.8.8) or someone else. We won’t go into these here.  Let’s just say it is beneficial for the ISP to run their own resolvers with root hints.

It’s all about the trickle-down or trickle-up effect. The Local resolvers cache what they can for their subset of customers.  These resolvers then look to the name server resolvers, which have larger caches, and those name server resolvers finally look up the information directly.

 

LibreNMS syslog.ibd cleanup

Recently I ran into an issue where a librenms install was taking up a crazy amount of disk space. This was tracked down to the syslog.ibd file . Even thought I had set my options to be less than 10 days per this link https://docs.librenms.org/Support/Cleanup-options/ I still was having a huge file.

Here is what I did to fix it. My root partition was too full to start MariaDB. I went into var and cleaned out enough log files to make space to start MariaDB. The following are tee commands I ran on CentOS 7 to fix it.

mysql -u username -p

Once at an MYSQL prompt after logging in I issued the following command to verify I could see my librenms database.

show databases;

The I issued

use dbname;

In may case it was “librenms”. Once connected to the database I ran the following command

DELETE FROM syslog WHERE timestamp < '2021-1-28 08:00:00';

This commands removes all syslog entries from the database before the date and time specified. In my case this was close to 40 GIG.

After I restarted MariaDB, ran .daily.sh and all was good.

PHP 7.3 and 7.4 Woes and solution

So this weekend I have been doing some LibreNMS installs and have been having to change the default CentOS PHP version. If it is Centos7 PHP 7.3 is recommend and if it’s Centos 8 then PHP 7.4 is recommended. I have some pretty good tutorials on installing Librenms that I follow. They all mention the remi repository. However, on a few of these networks the remi repository has been unreachable. So I just can’t cut and paste from the tuorials I use.

So what is a person to do? Lots of mirrors are available at https://rpms.remirepo.net/. All you have to do is some substitution.
http://mirror.team-cymru.com/remi/enterprise/remi-release-7.rpm

Done and Done. Hope this helps. You should be using mirrors anyway when it comes to remi. However, most tutorials reference remi because it’s easy.

Some WordPress tips

If you are wanting to force non SSL to SSL. Add the following to your site’s .htaccess file

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Set proper file permissions Script from https://www.ryadel.com/en/set-file-system-permissions-wordpress-web-site-centos-7-chmod/

#
# This script configures WordPress file permissions based on recommendations
# from http://codex.wordpress.org/Hardening_WordPress#File_permissions
#
# execute it with the following command:
# bash set-wordpress-permissions.sh /var/www/<site_folder>
#
OWNER=apache # <-- wordpress owner
GROUP=www # <-- wordpress group
ROOT=$1 # <-- wordpress root directory
 
# reset to safe defaults
find ${ROOT} -exec chown ${OWNER}:${GROUP} {} \;
find ${ROOT} -type d -exec chmod 755 {} \;
find ${ROOT} -type f -exec chmod 644 {} \;
 
# allow wordpress to manage wp-config.php (but prevent world access)
chgrp ${GROUP} ${ROOT}/wp-config.php
chmod 660 ${ROOT}/wp-config.php
 
# allow wordpress to manage wp-content
find ${ROOT}/wp-content -exec chgrp ${GROUP} {} \;
find ${ROOT}/wp-content -type d -exec chmod 775 {} \;
find ${ROOT}/wp-content -type f -exec chmod 664 {} \;

Interesting Network Monitoring tool

https://nav.uninett.no/

From their Web-site

NAV (Network Administration Visualized) is a free software program to monitor networks, developed by Uninett and used by universities and corporations around the world.

 Customizable dashboard
 Extensive statistical overviews
 Configuration on the fly
 Full traceability of users and equipment
 Device and vendor agnostic