Tag Archives: ip addressing

BGP, a single /24 and two diverse non-connected exit points

Posted on by

I am starting to see the following scenario more and more as IPv4 space is hard to get, but isn’t.

With ARIN it is still possible to get an IPv4 allotment. Many smaller ISPs qualify for a /24 and can get one if they wait long enough on the ARIN waiting list. a /24 of IPv4 space is the smallest block that 99% of the Internet allows to be advertised on the Capital I Internet. There are filter rules in place that drop smaller prefixes because that is the agreed upon norm.

So what happens if you are an ISP and you have a shiny new /24 but you have two networks which are not connected. Let’s look at our scenario.

The above network have no connectivity between the two of them on the internal side. These could be half way across the world or next door. If they were half way across the world it would make sense to try and get another /24. Maybe they are either side of a big mountain or one is down in a valley and there is no way to get a decent link between the two networks.

So what is a way you can use this /24 and still be able to assign IP addresses to both sides of the network? One way is to use a tunnel between your two edge routers.

Without the tunnel the scenario is traffic could come into network1, but if the IP is assigned on network 2 it will come back as unreachable. BGP is all about networks finding the shortest path to other networks. You don’t have much control over how networks find your public IP space if you have two providers advertising the same information. Some of the Internet will come in Network2 and some will come in Network1.

By running a tunnel between the two you can now subnet out that /24 into two eqal /25s and assign one /25 Network1 and one /25 to Network2 or however you want to. You can make the tunnel a GRE, EOIP, or other tunnel type. If I am using Mikrotik I prefer to use EOIP. If it’s another vendor I tend to use GRE.

Once the tunnel is established you can use static routing, OSPF, or your favorite IGP (interior Gateway Protocol) to “tell” one side about the routes on the other side. Let’s look at a fictional use.

In the above example our fictional ISP has an IPv4 block of 1.2.3.0/24. They have two networks separated by a tall mountain range in the center. It’s too cost prohibitive to run fiber or a wireless backhaul between the two networks so they have two different upstream providers. The ISP is advertising this /24 via BGP to Upstream1 from the Network 1 router. Network 2 router is also advertising the same /24 via BGP to Upstream 2.

We now create a Tunnel between the Mikrotiks. As mentioned before this can be EOIP, GRE, etc. We won’t go into the details of the tunnel but let’s assume the ISP is using Mikrotik. We create an EOIP tunnel (tons of tutorials out there) between Network 1 router and Network 2 router. Once this is established we will use 172.16.200.0/30 as our “Glue” on our tunnel interfaces at each side. Network 1 router gets 172.16.200.1/30. Network 2 router gets 172.16.200.2/30

To keep it simple we have a static route statement on the Network 1 Mikrotik router that looks like this:

/ip route add dst-address=1.2.3.129/25 gateway=172.16.200.2

This statement routes any traffic that comes in for 1.2.3.128/25 via ISP 1 to network1 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

Conversely, we have a similar statement in the Network 2 Mikrotik router

/ip route add dst-address=1.2.3.0/25 gateway=172.16.200.1

This statement routes any traffic that comes in for 1.2.3.0/25 via ISP 2 to network2 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

It’s as simple as that. You can apply this to any other vendor such as Cisco, Juniper, PFSense, etc. You also do not have to split the network into even /25’s like I did. You can choose to have os of the ips available on one side and route a /29 or something to the other side.

The major drawback of this scenario is you will takef a speed hit because if the traffic comes in one side and has to route across the tunnel it will have to go back out to the public internet and over to the other ISP.

#packetsdownrange

Common Question: Wisps, Public IPs, and assigning

Posted on by

So a question I get on a fairly regular basis is how do I assign IPs to my customers out of this /29 block my ISP gave me. As many of you know, IPV4 public addresses, commonly known as globally routable IPV4 addresses, are in short supply and obtaining more is usually very expensive. You have three options to obtain IP addresses.

Option #1 is getting them from whoever is supplying your bandwidth.  These IPs are known as Provider Assigned IP Space or PI space for short.  Typically these IP addresses are leased to you for a price or just allocated to you because you are a customer. The key here they are tied to the provider.  If you leave for someone else, your IPs are typically not portable.

Option#2 is going to the open market to buy or lease IP addresses.  There are several companies out there which sell or rent IP addresses. For business continuity buying your addresses is the best way to go.  As of this writing, IPV4 addresses in the ARIN region at going for $17-21 an Ip address.  This pricing means the smallest block you can get costs you around $5,600.

Option #3 is asking ARIN or whatever Regional Internet Registry (RIR) you fall under and getting put on a waiting list.  This wait can take quite some time and you may be waiting for a very long time.

Most startup ISPs get a small block of addresses from their provider.  The block can be a /29 ,or sometimes they can get a /28 or larger. The upstream ISP routes the block(s) to the startup. Here is where the problem comes up.  If an ISP gets a /29 this means they have 1 block of 6 IPs to use or two blocks of two IPs they can use in the form of two /30’s.  For those of you not familiar with subnetting, each time you “break down” a block of IP addresses you always lose 2 to the network and broadcast addresses.

if you want to learn about subnetting the following Youtube Videos will be helpful. Be warned there is a lot of numbers and math involved, but it’s necessary to learn about subnetting and proper IP planning.

Imagine an ISP has two customers on different towers that want a public IP address.  These addresses are for cameras, VPNs, or any number of uses. In order to assign IPs to each of those customers out of your /29 you have to either break it apart into two /30’s and route that to the appropriate router of yours the customer is attached to.  With small blocks assigned to you, this becomes very inefficient.  Even with large blocks, it can be inefficient. The below diagram shows how we can properly subnet this out given the limited IP block you have.

subnet example

We broke out 2.2.2.0/29 into the following subnets.
2.2.2.0/30
2.2.2.4/30

We still have the same number of IP addresses we had before, but due to the way subnetting works we lost a total of 4 addresses to the network and broadcast addresses.  Think of these as the beginning and end of the subnet.  We can’t use them to assign to devices, but we need them as part of the subnet.  We can note these as the following

2.2.2.0/30 – Network Address
2.2.2.1/30 – FIrst assignable address
2.2.2.2/30 – Second Assignable address
2.2.2.3/30 – Broadcast address

The next subnet would be 4,5,6,7 in the same order.  The rules of subnetting say we can not assign the network and broadcast addresses to devices or else we start breaking things.

Now if both of these customers were on the same tower, behind the same ISP router we would not necessarily have to subnetted the /29 out. We could have routed the /29 to the router and customer 1 has the .2 IP address and customer 2 has the .3 IP address.  the subnet would have IPs 4-6 available, but only for customers on the same tower.

The other way to save on public IP addresses is to do a 1:1 nat.  Natting involves doing port forwarding from the public IP to a private IP address at the customer location. If you want to learn about 1:1 Nat here are some links:
https://ciscoskills.net/2011/01/21/configuring-nat-one-to-one-mapping/
https://wiki.mikrotik.com/wiki/How_to_link_Public_addresses_to_Local_ones

So what is the real problem? If you have a /29 you can only break that down twice into the two /30s for customers going through different routers. Your network is routed instead of bridged right?

So how does the startup WISP overcome this without spending lots of money on IP addresses? You can do the 1:1 nat for each customer as explained above.  In my experience, this works 98% of the time. It is also something most I.T. folks are not familiar with.  I have had to spend hours on the phone with customers over the years explaining how a 1:1 nat works and they do have a public address to connect to, even though they have a private IP address on their router. Customers who do not get the 1:1 nat concept seem to be very leary of it and often blame the 1:1 nat when things go wrong.  Now if you are an ISP that provides a managed router, the customer does not see this layer.

Your other option is an MPLS network with VPLS tunnels.  MPLS with VPLS allows you to bridge your sites together to share an IP address pool.  However,  bridging is bad is what you hear.  MPLS has things like split-horizon built into the protocol to make bridging things together not so bad of a thing.

Another option is PPPoE.  PPPoE creates a tunnel utilizing a /32 (one ip address).  Many providers are using a shared pool to hand out IP addresses to customers who are given Public Ip addresses.

But my provider also gave me a /30 that is between my router and their’s. Can I use this?
The short answer is this is the “Glue” between you and the upstream provider.  The only real use for this is to Nat traffic out the Ip assigned to your side.  Doing this is not scalable though.

I have briefly touched on things like PPPoE, MPLS, and other things in this article.  These are such large topics there was no way this single post could do them justice.  I wanted to present the problem, a solution, and other methods to research to solve it. I will go into these in later blog posts.