Networking

The Mess we call BGP

Ever wonder why BGP seems to be such a complicated protocol to administer? It seems pretty straightforward to set up. Some commands, and you have a BGP session. Easy huh? BGP is one of those things where the more BGP feeds you bring in, the more complex traffic management becomes. Why? Take a look at the following graphic.

https://thyme.apnic.net/BGP/ARIN/#

What you are looking at is a small visualization of some of the AS connections to Hurricane Electric (AS6939) in North America. This is not all of them, just what I could fit on the screen for this article. Some of these are “transit ASes” which means they sit between Hurricane and another network or networks. This is important to understand because they can influence how your traffic reaches customers or resources on Hurricane electric if they are between you and them. The same thing goes for Hurricane Electric. They are a transit AS between companies and resources. Their policies in terms of BGP traffic can influence your traffic. This is just one AS. There are thousands and thousands of others.

Now imagine you have 4 upstream providers with various peerings and upstream peers. Each one of them can do various manipulations to the same destination. Your routers will pick the best path, but that path may have Congestion or a host of other influences on your traffic.

For myself, as a network engineer, being able to diagnose and troubleshoot path issues is an art, just as it is a science.

The Brothers WISP #169

Ninjas and network admins

From my early youth, Ninjas have been a fascinating subject for me. They are what attracted me to my main hobby of collecting G.I. Joes. Movies like American Ninja, Ninja Magazine, and the 80’s Ninja craze fueled this passion. Several of the stories in the comics centered around Storm Shadow and Snake Eyes, who were from the same Ninja clan.

As life progressed, I started studying the Art of Ninjitsu at age 16. Stephen Hayes’s writings on Togakure-ryū Ninjutsu inspired me to seek out training in this ancient art form. The things I learned from this directly translate into my job as a Network Engineer/Architect/Shaman. This tends to be a theme with any martial art. I just gravitated toward Ninjutsu.

Ninjutsu does not teach you pre-rehearsed Kata like many other martial arts. You learn concepts and then apply them to different scenarios. You learn to use what you know in a given situation instead of rehearsing moves. This isn’t to say arts with Katas are inferior, just a different mindset. I can directly apply this to my everyday work of trying not to get caught in too linear thinking.

This is where one would expect to say they learned patience from a martial art. I didn’t necessarily learn patience as much as I learned discipline. Discipline is knowing when to be calm and collected and when to pull out your swords and chop some stuff up. Patience is a part of the overall plan. A good foundation of patience, discipline, and drive are essential lessons to learn.

Network Engineers have different visibility in different types of organizations. In an Enterprise, I.T. is often bolted on. This has been changing some in the past few years, but it is not the focus of the business. The network is much more at the forefront in the Service Provider space. This is where the catch-22 happens for some I.T. Folks. The non-technical folks don’t understand the backend work making things run smoothly. An out-of-sight-out-of-mind kinda thing. When things break, the opposite happens. Management wonders what they have been doing. A good team of technical and non-technical keep the visibility “visible”. What goes on to make the network function is not some black hole full of jargon.


An outstanding network engineer is like a Ninja of Feudal Japan. The Ninja is behind the scenes manipulating things without really being seen. A smooth engineer “dons” his Ninja garb and gets into the real nuts and bolts. At the same time, this engineer can switch outfits, walk with the non-technical folks, and fit in. This is where some folks fail. To succeed, the Network person needs to explain to non-technical folks what is happening. It’s not because they need to understand the inner workings, but they need to understand the effort which goes into planning, implementing, and maintaining a modern computer network,

Great Job Jotform

After successfully logging into Jotform today, I received the below email. Way to go, Jotform, for successfully deploying IPv6. More companies should do this. #win #kudos

DNS in ISP networks and why you should care

If you are not familiar with how DNS works, please go and read this article and watch the accompanying video.

In this article we will talk about the different types of DNS servers an ISP will encounter and the design considerations of implementing them into your network. Let’s jump into the three types.

Cache servers
local DNS (LDNS) servers
authoritative DNS (ADNS) servers, of which the Root and Top Level Domain (gTLD) servers are special cases

Let’s go into detail on what each of these is.  We will start at the top and work our way down.

Authoritative Name Servers

An authoritative name server, just like the name says, is the authority on the website or resource it is responsible for. Without authoritative name servers, the DNS hierarchy would not function. You would have no way of telling an incorrect answer from a real answer.

A special kind of authoritative name server is the root nameservers, which are responsible for the Top Level Domains (TLD).  These are better known as .com,.net,.org, and so on.

Local DNS resolvers
These are servers run by Internet Service Providers (ISPs) and other entities such as Google (8.8.8.8) and Cloudflare (1.1.1.1). These are machines running BIND, Tiny DNS, or other software.  It is best practice to not run your authoritative name server on the same instance as your resolver.

Local DNS resolvers sit on the ISPs Internet backbone on fast connections to the Internet.

Caching servers
Caching servers are more typical in a larger ISP network with large concentrations of customers n certain geographic areas. .  Since DNS is such an important function, ISPs want their clients to have the quickest access to DNS lookups possible. In larger networks, this means putting caching servers as close to customers as possible. DNS lookups do not take up much bandwidth, but a slow DNS lookup can affect the user experience. If you have ever entered an address or clicked a link and experienced a “Looking up xyz.com” message for more than a second or two you are experiencing a slow DNS.

So why is this important in an ISP network?
As mentioned above the quicker you can get DNS lookups done and over with the better user experience your customers will have. You do not want your clients waiting 3-4 seconds for each lookup.  While you might have the fastest Internet on the planet, your customers will always be waiting on these requests, which will cause them to think things are slow.  And technically they would be right.

To speed up a large ISP network you will need to design DNS resolvers/caches into your design.  These can be at major traffic points on your network, or at each POP location. Part of this depends on your IP design, while some of it depends on the overall design of your infrastructure.  As networks flatten out due to things like VPLS and EVPN the need for lots of resolvers in an ISP network fades away. A general rule of thumb is to deploy DNS resolvers at each customer gateway.  This can be the wireless tower they receive their signal from, the local DSLAM, or a fiber node at the edge of your network.  By deploying resolvers at the customer gateway you are giving them fast access to cached DNS lookups. This can also lead to more complex troubleshooting should a cache server fail

What do I mean by cached?
A DNS resolver has two major functions in life.  It takes a request for a website, in our case j2sw.com, to look up the appropriate IP address and returns that information to the client.  The whole www stuff is for us, dumb humans. DNS translates www.j2sw.com to 209.209.44.4. This is what the computer and routers understand.

Secondly, a DNS resolver caches previous lookups.  This can be configured for minutes, hours, or days.  This speeds up any subsequent lookups because the resolver can serve it up locally without having to go clear across the internet to look it up and wait for the return. Once it has done one lookup, it will cache that lookup for any client who requests it after.

So, how does an ISP implement all of these?

Many modern routers have the ability to cache lookups and act as a DNS resolver.  Mikrotik, Cisco, and others all have this ability.   Even a Raspberry PI has enough horsepower to be a DNS resolver.  These can be deployed at each POP site and customers can point to them as their primary DNS servers.  This means their routers or PCs will look to the closest DNS resolver first.  If the resolver has the answer in its cache, it will serve it up right away.  If not, the resolver can go on to the next step up.

The next step is local resolvers the ISP typically runs.  There are lots of arguments for why an ISP should run their own instead of pointing to Google (8.8.8.8) or someone else. We won’t go into these here.  Let’s just say it is beneficial for the ISP to run their own resolvers with root hints.

It’s all about the trickle-down or trickle-up effect. The Local resolvers cache what they can for their subset of customers.  These resolvers then look to the name server resolvers, which have larger caches, and those name server resolvers finally look up the information directly.

 

The arguments against 0.0.0.0/8

The arguments against using 0.0.0.0/0 have been pretty much quashed by the following:

Allow 0.0.0.0/8 as a valid address rangeThe longstanding prohibition against using 0.0.0.0/8 dates back to two issues with the early internet.

There was an interoperability problem with BSD 4.2 in 1984, fixed in BSD 4.3 in 1986. BSD 4.2 has long since been retired.

Secondly, addresses of the form 0.x.y.z were initially defined only as a source address in an ICMP datagram, indicating “node number x.y.z on this IPv4 network”, by nodes that know their address on their local network, but do not yet know their network prefix, in RFC0792 (page 19). This usage of 0.x.y.z was later repealed in RFC1122 (section 3.2.2.7), because the original ICMP-based mechanism for learning the network prefix was unworkable on many networks such as Ethernet (which have longer addresses that would not fit into the 24 “node number” bits). Modern networks use reverse ARP (RFC0903) or BOOTP (RFC0951) or DHCP (RFC2131) to find their full 32-bit address and CIDR netmask (and other parameters such as default gateways). 0.x.y.z has had 16,777,215 addresses in 0.0.0.0/8 space left unused and reserved for future use, since 1989.

This patch allows for these 16m new IPv4 addresses to appear within a box or on the wire. Layer 2 switches don’t care. 0.0.0.0/32 is still prohibited, of course.

Source: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=96125bf9985a&fbclid=IwAR2W03PZN8b7ZzAL1jsvxe0wuI9qOWKpeWdAnMMjvgLGE7x5f48RIaCtYUw

Musings of an ISP Engineer part 2

Part 1 can be viewed here

-Always check layer1 first. Your momma told you it’s the simplest things in life.  Most times it is that simple.

-After that check layer 2. If a backhaul or fiber link can’t push data due to no connectivity then the fanciest written config in the world won’t mean a thing.

-Use a password manager. 16 character passwords should be the norm with two-factor authentication to the password manager.

-Make a decision and act on it.  Too many times I see network by committee.  By the time it’s actually implemented the parameters have changed. Do the best you can, with the information you have, in the time you have.

-Have a second and third way into your network

yo dawg i heard you like redundancy so i put redundancy in your redundancy  so you can be redundant while you're redundant! - xzibit-yo-dawg | Meme  Generator

-Get used to reading release notes and scanning forums/mailing lists for bug reports

-Your network probably is not hacked

-Quit spending time trying to navigate your network on your phone’s small screen.  Break down and buy a tablet or break out your laptop. Your phone is for quick assessments using the data you are fed.  It’s too inefficient to make config changes.

32,619 Phone Frustration Stock Photos, Pictures & Royalty-Free Images -  iStock

-This Dilbert comic strip cracks my ass up

Dilbert had it right back in 1995 : r/linuxmasterrace

-Just emphasizing how it’s the simplest things at times

-It’s fine if you know how to program in binary but you don’t need to know the inner workings of hardware for 95% of the job. A racecar driver probably knows about engines but spends 99% of his or her time just driving the car.

-Get to know your sales reps for your hardware. They are the unsung heroes in this industry. a good salesperson picks up tidbits of knowledge and lets their customers know. Things like bad hardware runs, bugs other customers have come across, and the like.

A successful maintenance window is 75% labor and 25% documentation. An unsuccessful one is 75% labor and 25% rolling into backups.

-You only know what you know and that’s okay

-The big companies are just as clueless as you. Sometimes they are worse. They just have more people who know just enough to get by.

-There are work gadgets and fun gadgets. Keep the two separate.

Hilarious Comics With Unexpected Endings By “Wtframecomics” (47 Pics) –  Global Circulate

-That optic you should have replaced when you were cleaning the fiber cable will probably haunt you tomorrow.

Part 3? Maybe…..