BGP, a single /24 and two diverse non-connected exit points

I am starting to see the following scenario more and more as IPv4 space is hard to get, but isn’t.

With ARIN it is still possible to get an IPv4 allotment. Many smaller ISPs qualify for a /24 and can get one if they wait long enough on the ARIN waiting list. a /24 of IPv4 space is the smallest block that 99% of the Internet allows to be advertised on the Capital I Internet. There are filter rules in place that drop smaller prefixes because that is the agreed upon norm.

So what happens if you are an ISP and you have a shiny new /24 but you have two networks which are not connected. Let’s look at our scenario.

The above network have no connectivity between the two of them on the internal side. These could be half way across the world or next door. If they were half way across the world it would make sense to try and get another /24. Maybe they are either side of a big mountain or one is down in a valley and there is no way to get a decent link between the two networks.

So what is a way you can use this /24 and still be able to assign IP addresses to both sides of the network? One way is to use a tunnel between your two edge routers.

Without the tunnel the scenario is traffic could come into network1, but if the IP is assigned on network 2 it will come back as unreachable. BGP is all about networks finding the shortest path to other networks. You don’t have much control over how networks find your public IP space if you have two providers advertising the same information. Some of the Internet will come in Network2 and some will come in Network1.

By running a tunnel between the two you can now subnet out that /24 into two eqal /25s and assign one /25 Network1 and one /25 to Network2 or however you want to. You can make the tunnel a GRE, EOIP, or other tunnel type. If I am using Mikrotik I prefer to use EOIP. If it’s another vendor I tend to use GRE.

Once the tunnel is established you can use static routing, OSPF, or your favorite IGP (interior Gateway Protocol) to “tell” one side about the routes on the other side. Let’s look at a fictional use.

In the above example our fictional ISP has an IPv4 block of 1.2.3.0/24. They have two networks separated by a tall mountain range in the center. It’s too cost prohibitive to run fiber or a wireless backhaul between the two networks so they have two different upstream providers. The ISP is advertising this /24 via BGP to Upstream1 from the Network 1 router. Network 2 router is also advertising the same /24 via BGP to Upstream 2.

We now create a Tunnel between the Mikrotiks. As mentioned before this can be EOIP, GRE, etc. We won’t go into the details of the tunnel but let’s assume the ISP is using Mikrotik. We create an EOIP tunnel (tons of tutorials out there) between Network 1 router and Network 2 router. Once this is established we will use 172.16.200.0/30 as our “Glue” on our tunnel interfaces at each side. Network 1 router gets 172.16.200.1/30. Network 2 router gets 172.16.200.2/30

To keep it simple we have a static route statement on the Network 1 Mikrotik router that looks like this:

/ip route add dst-address=1.2.3.129/25 gateway=172.16.200.2

This statement routes any traffic that comes in for 1.2.3.128/25 via ISP 1 to network1 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

Conversely, we have a similar statement in the Network 2 Mikrotik router

/ip route add dst-address=1.2.3.0/25 gateway=172.16.200.1

This statement routes any traffic that comes in for 1.2.3.0/25 via ISP 2 to network2 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

It’s as simple as that. You can apply this to any other vendor such as Cisco, Juniper, PFSense, etc. You also do not have to split the network into even /25’s like I did. You can choose to have os of the ips available on one side and route a /29 or something to the other side.

The major drawback of this scenario is you will takef a speed hit because if the traffic comes in one side and has to route across the tunnel it will have to go back out to the public internet and over to the other ISP.

#packetsdownrange

Preseem and Switches in switch centric design

Anyone who follows me knows I am a big fan of switch centric designs. This usually involves a router on a stick paired with a high port count switch. Recently I had a client that installed a Preseem appliance in their network.

Equipment used in this setup
-Dell R710 with a 4 Port SFP+ card running Preseem
-Cisco 3064-X 48 Port switch
-Maxxwave Vengeance router with dual QSF+ card and 4 Port SFP+ card

A visio diagram of how this looks

We have two transport links coming into the switch on the left. These are dumped into VLANs 506 and 507. We then come out of the switch into the Preseem box via 2 SFP+ ports, one for each VLAN. In this case, we just used DAC cables In the future, we can turn these into trunk ports to pass more VLANS through.

The data then leaves the Preseem box over dual SFP fibers directly into the router’s SFP+ ports. If the Preseem appliance fails we have a secondary OSPF/IBGP path from the router’s 40 GIG QSFP down to the switch. This is a bypass in case the Preseem appliance hardware fails.

If you start flowing more than 10 Gigs through a single link you can upgrade to more SFP+ ports into your appliance and a 40 Gig QSFP+ card. You then link the appliance to the spare QSFP port on your router.

Learning, Certifications and the WISP

One of the most asked questions which come up in the xISP world is “How do I learn this stuff?”.   Depending on who you ask this could be a lengthy answer or a simple one-sentence answer.  Before we answer the question, let’s dive into why the answer is complicated.

In many enterprise environments, there is usually pretty standard deployment of networking hardware.  Typically this is from a certain vendor.  There are many factors involved. in why this is.  The first is the total Cost of Ownership (TCO).  It almost always costs less to support one product than to support multiples.  Things like staff training are usually a big factor.  If you are running Cisco it’s cheaper to train and keep updated on just Cisco rather than Cisco and another vendor.

Another factor involved is the economies of scale.  Buying all your gear from a certain vendor allows you to leverage buying power. Quantity discounts in other words.  You can commit to buying a product over time or all at once.

So, to answer this question in simple terms.  If your network runs Mikrotik, go to a Mikrotik training course.  If you run Ubiquiti go to a Ubiquiti training class.

Now that the simple question has been answered, let’s move on to the complicated, and typically the real world answer and scenario.  Many of our xISP clients have gear from several vendors deployed.  They may have several different kinds of Wireless systems, a switch solution, a router solution, and different pieces in-between.  So where does a person start?

I recommend the following path. You can tweak this a little based on your learning style, skill level, and the gear you want to learn.

1. Start with the Cisco Certified Network Associate (CCNA) certification in Routing and Switching (R&S).  There are a ton of ways to study for this certification.   There are Bootcamps (not a huge fan of these for learning), iPhone and Android Apps (again these are more focused on getting the cert), online, books, and even youtube videos. Through the process of

studying for this certification, you will learn many things that will carry over to any vendor.  Things like subnetting, differences between broadcast and collision domains, and even some IPV6 in the newest tracks.  During the course of studying you will learn, and then reinforce that through practice tests and such.  Don’t necessarily focus on the goal of passing the test, focus on the content of the material.  I used to work with a guy who went into every test with the goal of passing at 100%.  This meant he had to know the material. CompTIA is a side path to the Cisco CCNA.  For reasons explained later, COMPTIA Network+ doesn’t necessarily work into my plan, especially when it comes to #3. I would recommend COMPTIA if you have never taken a certification test before.

2. Once you have the CCNA under your belt, take a course in a vendor you will be working

the most with.  At the end of this article, I am going to add links to some of the popular vendor certifications and then 3rd party folks who teach classes. One of the advantages of a 3rd party teacher is they are able to apply this to your real-world needs. If you are running Mikrotik, take a class in that. Let the certification be a by-product of that class.

3.Once you have completed #1 and #2 under your belt go back to Cisco for their Cisco Certifed Design Associate (CCDA). This is a very crucial step those on a learning path overlook.  Think of your networking knowledge as your end goal is to be able to build a house.  Steps one and two have given you general knowledge, you can now use tools, do some basic configuration.  But you can’t build a house without knowing what is involved in designing foundations,  what materials you need to use, how to compact the soil, etc.  Network design is no different. These are not things you can read in a manual on how to use the tool.  They also are not tool-specific.   Some of the things in the Cisco CCDA will be specific to Cisco, but overall it is a general learning track.  Just follow my philosophy in relationship to #1. Focus on the material.

Once you have all of this under your belt look into pulling in pieces of other knowledge. Understanding what is going on is key to your success.  If you understand what goes on with an IP packet, learning tools like Wireshark will be easier.  As you progress let things grow organically from this point.  Adding equipment in from a Vendor? Update your knowledge or press the new vendor for training options.  Branch out into some other areas , such as security, to add to your overall understanding.

WISP Based Traning Folks.
These companies and individuals provide WISP based training. Some of it is vendor focused. Some are not.  My advice is to ask questions. See if they are a fit for what your goals are.
-Connectivity Engineer
Butch Evans
Dennis Burgess
Rick Frey
Steve Discher
Baltic Networks

Vendor Certification Pages
Ubiquiti
Mikrotik
Cisco
Juniper
CWNA
CompTIA

If you provide training let me know and I will add you to this list.

Cisco and Verizon demonstrate multi-haul

As Internet traffic grows and becomes more dynamic, optical transport networks for sub-sea, terrestrial long haul and metro need more capacity. The ability to deploy capacity quickly is equally important to handle the increasingly dynamic nature of the traffic. The concept of a multi-haul transport platform, as introduced by Andrew Schmitt of Cignal AI, becomes very appealing for achieving this ability to scale with speed while maintaining operational simplicity – a single platform for all requirements. A critical element of the multi-haul optical platform is the flexibility of the coherent optics to be tuned to fine granularity in order to meet the reach-capacity target of any given network.

https://ciscocentral.blogspot.com/2019/02/cisco-and-verizon-to-demonstrate.html

Cisco 2960X I/O usage

While double checking some stats on a network I came across this in Libre.   84% is usually something that would cause me to be alarmed, as Libre is trying to tell us.

After some research, I found the following.

While it is not documented, it was noted that this was by design and that it would not affect the switch as the switchport becomes more and more loaded.

The switch allocates dedicated memory to certain processes / resources by default and then additional resources when the configuration is added. This ensures proper functionality and is again by design.

The I/O Memory pool buffers information transmitted to and from the CPU, and does not affect the actual forwarding of packets on the switch.

Translation: The switch uses up these resources by default, even if they aren’t all being used.  Think of it as setting it aside for future use without dynamic allocation of them.

Some Basic Cisco router terminology

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

Not enough file system space on Cisco nexus?

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.