DNS in ISP networks and why you should care

If you are not familiar with how DNS works, please go and read this article and watch the accompanying video.

In this article we will talk about the different types of DNS servers an ISP will encounter and the design considerations of implementing them into your network. Let’s jump into the three types.

Cache servers
local DNS (LDNS) servers
authoritative DNS (ADNS) servers, of which the Root and Top Level Domain (gTLD) servers are special cases

Let’s go into detail on what each of these is.  We will start at the top and work our way down.

Authoritative Name Servers

An authoritative name server, just like the name says, is the authority on the website or resource it is responsible for. Without authoritative name servers, the DNS hierarchy would not function. You would have no way of telling an incorrect answer from a real answer.

A special kind of authoritative name server is the root nameservers, which are responsible for the Top Level Domains (TLD).  These are better known as .com,.net,.org, and so on.

Local DNS resolvers
These are servers run by Internet Service Providers (ISPs) and other entities such as Google (8.8.8.8) and Cloudflare (1.1.1.1). These are machines running BIND, Tiny DNS, or other software.  It is best practice to not run your authoritative name server on the same instance as your resolver.

Local DNS resolvers sit on the ISPs Internet backbone on fast connections to the Internet.

Caching servers
Caching servers are more typical in a larger ISP network with large concentrations of customers n certain geographic areas. .  Since DNS is such an important function, ISPs want their clients to have the quickest access to DNS lookups possible. In larger networks, this means putting caching servers as close to customers as possible. DNS lookups do not take up much bandwidth, but a slow DNS lookup can affect the user experience. If you have ever entered an address or clicked a link and experienced a “Looking up xyz.com” message for more than a second or two you are experiencing a slow DNS.

So why is this important in an ISP network?
As mentioned above the quicker you can get DNS lookups done and over with the better user experience your customers will have. You do not want your clients waiting 3-4 seconds for each lookup.  While you might have the fastest Internet on the planet, your customers will always be waiting on these requests, which will cause them to think things are slow.  And technically they would be right.

To speed up a large ISP network you will need to design DNS resolvers/caches into your design.  These can be at major traffic points on your network, or at each POP location. Part of this depends on your IP design, while some of it depends on the overall design of your infrastructure.  As networks flatten out due to things like VPLS and EVPN the need for lots of resolvers in an ISP network fades away. A general rule of thumb is to deploy DNS resolvers at each customer gateway.  This can be the wireless tower they receive their signal from, the local DSLAM, or a fiber node at the edge of your network.  By deploying resolvers at the customer gateway you are giving them fast access to cached DNS lookups. This can also lead to more complex troubleshooting should a cache server fail

What do I mean by cached?
A DNS resolver has two major functions in life.  It takes a request for a website, in our case j2sw.com, to look up the appropriate IP address and returns that information to the client.  The whole www stuff is for us, dumb humans. DNS translates www.j2sw.com to 209.209.44.4. This is what the computer and routers understand.

Secondly, a DNS resolver caches previous lookups.  This can be configured for minutes, hours, or days.  This speeds up any subsequent lookups because the resolver can serve it up locally without having to go clear across the internet to look it up and wait for the return. Once it has done one lookup, it will cache that lookup for any client who requests it after.

So, how does an ISP implement all of these?

Many modern routers have the ability to cache lookups and act as a DNS resolver.  Mikrotik, Cisco, and others all have this ability.   Even a Raspberry PI has enough horsepower to be a DNS resolver.  These can be deployed at each POP site and customers can point to them as their primary DNS servers.  This means their routers or PCs will look to the closest DNS resolver first.  If the resolver has the answer in its cache, it will serve it up right away.  If not, the resolver can go on to the next step up.

The next step is local resolvers the ISP typically runs.  There are lots of arguments for why an ISP should run their own instead of pointing to Google (8.8.8.8) or someone else. We won’t go into these here.  Let’s just say it is beneficial for the ISP to run their own resolvers with root hints.

It’s all about the trickle-down or trickle-up effect. The Local resolvers cache what they can for their subset of customers.  These resolvers then look to the name server resolvers, which have larger caches, and those name server resolvers finally look up the information directly.

 

ISP consultants.How to hire and engage Part 1

I have been a consultant for the Internet Service Provider (ISP) space for the better part of my working life. I have dealt with the technical side of the field and some of the business and back end. I have been an owner/operator/stakeholder in several ISPs since 2000. Some of these ventures have been self-funded, while partners funded others. This history has given me a unique perspective many others have not been able to experience.

Most of the ISP operators I have worked with boil down two one of three types

The Businessperson. These operators usually have little technical knowledge but see a business need and require technical and operational personnel to help them. An outside technical consultant helps to supplement any technical expertise.

The Techie. These are folks who know the technical side of operations. They may not necessarily have Service Provider experience. Still, they can configure gear, understand spec sheets, and follow the lingo on message boards and groups.

The Operator. They may need a consultant for a few reasons. The first is that they are racing and need outside talent on demand. Second, they may need someone to supplement higher-end tasks such as BGP, CBRS implementations, or LTE help. This person is a professional who can pick up on both the business and technical sides.

Each of the above types needs different things from a consultant and should approach a consultant differently. One of the best things to do before hiring a consultant is put a list together of what you need help with. This need can be a long wishlist or a specific task. Either way, having a defined Scope of Work (SOW) is beneficial.

From my experience, the more focused the task, the easier it is for me to get up to speed. This ease is especially true of an established network. It is much easier for me to give a time estimate when someone says, “OSPF is broken between router ten and router eleven. Can you look at that?” than it is for someone to say, “My OSPF is broken. can you look at it”. As a general topic of investigation, I will try and get more specific, so I am not spending hours looking at unaffected network segments.

One of the things I think any owner should do with a potential client is to have a general call before any work starts. If you are looking for someone to fix a specific issue, this is probably a quick call. I have jumped into a client’s screen share before within a few minutes of them calling and worked through an issue without much prep work. Suppose the operator is looking to have a consultant work on several projects and work on a medium to long-term basis. In that case, the conversation needs to be longer.

Suppose you are an operator looking to hire a consultant to be with you for a while. In that case, the initial conversation I mentioned above is more like a two-way interview process. This type of conversation tends to happen with new or startup WISPs a lot. They need direction and someone to answer many questions that are not answered outside of an ISP. In an earlier article, I go over the differences between an ISP network and an Enterprise network.

In Part two we will look at more things to ask and look for from the operator’s perspective. In Part Three we will look at some of the rationale and options from the consultant’s perspective.

Cambium QOE traffic screenshots

Recently I have been using the QoE solution from Cambium Networks on some networks. This software allows for the prioritization and shaping of traffic on a service provider’s network. We will go into the workings of this in some later posts. Here are some screenshots.

Yes, it is IPv6 aware

What services should your ISP be providing?

As Internet Service has evolved so have the service offerings of Internet Service Providers (ISPs). Not too long ago the majority of ISPs were providing USENET feeds. If you remember those you have been at this for a long time.

For those of you just getting started or wanting to do an evaluation on your network here are some thoughts on services you should or should not be providing.

Let’s dive into the short list

DNS – Must Have
You should be running your own Doman Name Service (DNS) resolvers. There are many benefits to you and your customers.

e-mail – Don’t worry about offering
The advent of free or low-cost e-mail services has seen a rapid decline in the need for ISPs to their customers. For ISPs who offer e-mail, it is a service that generates many trouble tickets. From SPAM complaints all the way down to them not receiving emails. Leave offering email to customers to the GMAIL folks of the world.

Web-Site Hosting – Optional
If you are a Managed Service provider (MSP) then web-hosting can be a lucrative business. With web-site hosting also comes e-mail hosting.

VOIP
Voice Over IP (VOIP) is a hot topic these days. Many grants mention the ability to provide voice services as a requirement. Outsource this to providers such as Atheral.

Thats it! There are many other services you could extend to customers. There are also services which are not public facing you might want to run. Radius is on example of services you would run internally. As a service provider you should be spending as much effort on delivering solid access to your customers. Other services can be found by the end users themselves from dedicated services.

Yet another case for IPv6 for service providers

So recently I have been posting about the Cambium Networks QOE box I have been testing. After having this run for about a week I figured I would share this tidbit about usage.

55.39% of my traffic is IPV6 traffic. Most of this would be streaming traffic to various folks like NetFlix and Amazon and gaming traffic. My household consists of three people. No kids.

Implement IPV6 now if you are a service provider and thank me later. https://blog.j2sw.com/networking/wisps-ipv6-is-the-answer-to-some-of-your-issues/

#packetsdownrange

Harvestore WISP Install

Over the years many MidWestern WISPS have fashioned ways to use Harvestore silos for broadcast locations. With the help of one of our good clients, who did most of the work designing this, we were able to install a very solid structure on the top of a silo.

Please note. These photos were taken before wiring and final touches were installed. The backhaul dishes all need tie-backs to help prevent them from twisting in the wind. This was the biggest thing not shown in the photos.