Common Question: Wisps, Public IPs, and assigning

So a question I get on a fairly regular basis is how do I assign IPs to my customers out of this /29 block my ISP gave me. As many of you know, IPV4 public addresses, commonly known as globally routable IPV4 addresses, are in short supply and obtaining more is usually very expensive. You have three options to obtain IP addresses.

Option #1 is getting them from whoever is supplying your bandwidth.  These IPs are known as Provider Assigned IP Space or PI space for short.  Typically these IP addresses are leased to you for a price or just allocated to you because you are a customer. The key here they are tied to the provider.  If you leave for someone else, your IPs are typically not portable.

Option#2 is going to the open market to buy or lease IP addresses.  There are several companies out there which sell or rent IP addresses. For business continuity buying your addresses is the best way to go.  As of this writing, IPV4 addresses in the ARIN region at going for $17-21 an Ip address.  This pricing means the smallest block you can get costs you around $5,600.

Option #3 is asking ARIN or whatever Regional Internet Registry (RIR) you fall under and getting put on a waiting list.  This wait can take quite some time and you may be waiting for a very long time.

Most startup ISPs get a small block of addresses from their provider.  The block can be a /29 ,or sometimes they can get a /28 or larger. The upstream ISP routes the block(s) to the startup. Here is where the problem comes up.  If an ISP gets a /29 this means they have 1 block of 6 IPs to use or two blocks of two IPs they can use in the form of two /30’s.  For those of you not familiar with subnetting, each time you “break down” a block of IP addresses you always lose 2 to the network and broadcast addresses.

if you want to learn about subnetting the following Youtube Videos will be helpful. Be warned there is a lot of numbers and math involved, but it’s necessary to learn about subnetting and proper IP planning.

Imagine an ISP has two customers on different towers that want a public IP address.  These addresses are for cameras, VPNs, or any number of uses. In order to assign IPs to each of those customers out of your /29 you have to either break it apart into two /30’s and route that to the appropriate router of yours the customer is attached to.  With small blocks assigned to you, this becomes very inefficient.  Even with large blocks, it can be inefficient. The below diagram shows how we can properly subnet this out given the limited IP block you have.

subnet example

We broke out into the following subnets.

We still have the same number of IP addresses we had before, but due to the way subnetting works we lost a total of 4 addresses to the network and broadcast addresses.  Think of these as the beginning and end of the subnet.  We can’t use them to assign to devices, but we need them as part of the subnet.  We can note these as the following – Network Address – FIrst assignable address – Second Assignable address – Broadcast address

The next subnet would be 4,5,6,7 in the same order.  The rules of subnetting say we can not assign the network and broadcast addresses to devices or else we start breaking things.

Now if both of these customers were on the same tower, behind the same ISP router we would not necessarily have to subnetted the /29 out. We could have routed the /29 to the router and customer 1 has the .2 IP address and customer 2 has the .3 IP address.  the subnet would have IPs 4-6 available, but only for customers on the same tower.

The other way to save on public IP addresses is to do a 1:1 nat.  Natting involves doing port forwarding from the public IP to a private IP address at the customer location. If you want to learn about 1:1 Nat here are some links:

So what is the real problem? If you have a /29 you can only break that down twice into the two /30s for customers going through different routers. Your network is routed instead of bridged right?

So how does the startup WISP overcome this without spending lots of money on IP addresses? You can do the 1:1 nat for each customer as explained above.  In my experience, this works 98% of the time. It is also something most I.T. folks are not familiar with.  I have had to spend hours on the phone with customers over the years explaining how a 1:1 nat works and they do have a public address to connect to, even though they have a private IP address on their router. Customers who do not get the 1:1 nat concept seem to be very leary of it and often blame the 1:1 nat when things go wrong.  Now if you are an ISP that provides a managed router, the customer does not see this layer.

Your other option is an MPLS network with VPLS tunnels.  MPLS with VPLS allows you to bridge your sites together to share an IP address pool.  However,  bridging is bad is what you hear.  MPLS has things like split-horizon built into the protocol to make bridging things together not so bad of a thing.

Another option is PPPoE.  PPPoE creates a tunnel utilizing a /32 (one ip address).  Many providers are using a shared pool to hand out IP addresses to customers who are given Public Ip addresses.

But my provider also gave me a /30 that is between my router and their’s. Can I use this?
The short answer is this is the “Glue” between you and the upstream provider.  The only real use for this is to Nat traffic out the Ip assigned to your side.  Doing this is not scalable though.

I have briefly touched on things like PPPoE, MPLS, and other things in this article.  These are such large topics there was no way this single post could do them justice.  I wanted to present the problem, a solution, and other methods to research to solve it. I will go into these in later blog posts.

Another BGP blunder.but not THAT one

Recently, there has been much talk about the Cloudflare BGP blunder and others. The Network Collective even did a video about such things. But did you know there was one involving the entire /12 of IPV6 space? Airtel AS9498 announced the entire IPv6 block 2400::/12 for a week and no-one noticed. Someone typed a /12 instead of a /127.

So why did no one notice? I think part of it is due to the low usage of v6 space.  Sure, all kinds of people claim stats on IPV6 usage.  They talk about X number of traffic is v6, etc. There is a difference between users and connections.  A connection may not actually represent unique users.

Secondly, people are used to IPV6 being buggy.  I know many ISPs who disabled v6 as part of their troubleshooting steps.

I know there will be several folks who jump all over me about IPV6 being the wave of the future and we all should be using it.  Yes, we should, but there is no huge hurry when it comes to business cases.

Flash Briefing: April 29. Spectrum, IoT, WPA compromises

Spectrum use Article
The wireless industry has always had to deal with regular (and alarming) pronouncements that we’re somehow running out of radio spectrum. We’re not. But the misconception regardless gives many IT and network managers pause. 

It does not mention WiSPs, but is a perspective nonetheless.

IoT Links
How many of you are focusing on Internet of Everything (IoT)? I have posted some links to how healthcare and others are using IoT to further their business.  As a service provider, you should be coming up with an IoT strategy.

WPA Compromises

Microsoft and IPV6

I have written about IPV6 lately and Microsoft has published a post where they are moving their internal network to an IPV6 only network.

Microsoft works toward IPv6-only single stack network

TeamArin at CanWISP


Kubernetes and IPV6

Carriers and telcos are also interested in adopting Kubernetes — AT&T is using Kubernetes as the basis of the Airship project it will use to run 5G and public safety network services — but they’ve already deployed significant amounts of IPv6, especially for mobile networks. Around 90 percent of T-Mobile USA and Verizon Wireless traffic already goes over IPv6; for Comcast and AT&T it’s about 70 percent.

IPV6 Planning and Implementation resources for the xISP

This post is designed to give you a lot of information on implementing IPV6 into your network.  Much of this has is aimed at the service provider market, but there are resources for the enterprise market.

One of the mindsets of this article is you are treating your customers as one of two ways.  The first is a typical ISP end user.  This can be John Smith on your wireless network or the small branch office down the street. These can vary in size and I will talk about how you can deal with these.  The second type is the Enterprise or BGP peer.  These are folks who should have their own IPV6 allocations and ASN.

Anyone who has followed me for a while knows I talk a lot of network philosophy. This article has bits of philosophy mixed in this article.  Some of it is my own while others are the debates on certain aspects of IPV6 implementation. This is not a comprehensive guide to IPV6.  rather, this is designed to fill in some blanks.

Implementing IPV6
Packet Pushers has an excellent IPV6 planning podcast

Kevin Myers over at Stubarea51 has a great article on implementing IPV6 in a service provider market using Mikrotik

Typical service providers will are assigned a /36 or a /32 from their Regional Routing Registry (RIR).  Don’t think in terms of IP addresses, but think in terms of subnets.  I will emphasize this throughout the article.  Also, another rule of thumb is when you are subnetting everything has to be done in multiples of 4 when it comes to the subnet mask. As with anything, there are exceptions but they might not be best practices.

IPV6 and Point-to-Point links
There is much debate on whether you should use a /64, a /124,126, or a /127 for point-to-point links.  These would be the equivalent of /30’s in the IPV4 world.  One of my favorite articles on this topic

I typically use a /124 or /126, which seems to be what most of the larger upstream providers and content networks are using for point-to-point links. If you are using a /64 you have to contend with neighbor exhaustion attacks as well as ping pong attacks.  One of the hybrid approaches is to use a /124,/126 or a /127 pulled from a single /64. I have yet to have a major upstream or content network hand me a /127.

Point-to-point links are one of the most highly talked about IPV6 methodologies.  There are many right answers in this debate, as you can tell my the ietf article above. My advice is to pick one and go with it.  The majority of the BGP peers I work with use either a /124 or a /126.  I pull these out of a single /64.  Some ISPs will allocate a /56 or /60, which we will talk about next, and use the first /64 and pull their point-to-point from that.

IPV6 and customer allocations
For years the standard allocation to a customer was a /56.  In recent years this has been shrunk to a /60 with most of the major providers, such as Comcast.  When we break down allocations in a service provider network, we have the standard accepted school of thought.

  • Individual pops are assigned a /48 prefix.
  • Enterprise customers are assigned a /56 prefix. That gives them 256 /64 subnets.
  • Small customers are assigned a /60 prefix, allowing 16 /64 subnets.

One of the critical errors in thinking I see network engineers and architects do in IPV6 thinking is concentrating too much on IP addresses and not on the subnets.  Just like in IPV4 subnetting it is all about the math.  Assigning a customer a /64 directly is bad practice. The biggest reason is it does not allow the customer the ability to subnet the block into smaller chunks.

Some engineers think a customer will never use 18,446,744,073,709,551,616 and they would be correct. That is 18 quintillion IP addresses.  99.99 percent of us have never dealt with numbers that high.  Many of us have only heard about numbers that high in science fiction such as Star Trek.  However, a /64 is the smallest subnet assigned without breaking core functionality of IPV6. Remember earlier when we talked about point-to-point links? When we subnet out the /64 we are taking away sing a subnet prefix length other than a /64 will break many features of IPv6, amongst other things Neighbor Discovery (ND), Secure Neighborship Discovery (SEND) [RFC3971], privacy extensions [RFC4941], parts of Mobile IPv6 [RFC4866], PIM-SM with Embedded-RP [RFC3956], and SHIM6 [RFC5533].  Many new IPV6 developments also are relying on /64s being in place.

Back to our point-to-point addressing, we do not need neighbor discovery because we are only talking to one host.  We don’t need privacy because we know the one host we are talking with, and so on.

But 18 quintillion IP addresses is still too much.  Yes, but all the features relying on at least a /64 are worth the tradeoff.  The key is to think in terms of subnets, not IP addresses.

So why not assign the customer a /62 to save on space..err..I mean subnets? Remember, when subnetting the math has to be in multiple of 4. So you can assign a /48, a /52, a /56, or a /60 to the customer and be safe. We talked about why assigning a /64 is bad practice even though it fits the multiple of 4.

Provider assigned IP (PI) space
There are very few cases for assigning Provider space to a customer who has an ASN.  If the customer has an ASN they should be going to their Regional Internet Registry (RIR) and getting an allocation.  In my case, most of the requests I do go through ARIN.  Arin has a very good document on getting your first IPV6 allocation.

Anyone else I want to assign them a /56 or /60 as stated above.  These are customer not participating in BGP. One of the lessons learned in the IPV4 world is how many started with provider assigned IP space and then went through the pain of renumbering when they received allocations from the RIR. In IPV6 we would have to re-subnet, but still the same pain.

What gets assigned IPV6 space?
One of my philosophies is I treat IPV6 like public IP addresses in the IPV4 world.  I am not assigning Internet routable IP addresses to my switches, and APs unless they are doing routing.  I can still use private Ip addresses for my infrastructure such as 10.x.x.x./8

Philosophies rounded up
-Point-to-point links.  Should you use a /64,124,126,127, or even something else.  I have seen Cogent hand me a /112 in several places

-Customer Allocations. Should you assign the customer a /56 or bigger?  I believe any customer that needs more than a /56 should have their own ASN and their own IPV6 allocation. If necessary a large customer could have a /52 and still be within the bounds.

Closing notes
IPv6 is very much in flux with some conflicting methods of doing things.  However, a good solid plan you stick to and follow can save you lots of headaches and allow your network to scale. Implementing IPV6 to some degree on your network should be a priority.  There are many benefits to IPV6, which are not IP space-related which I will talk about in a later article

Further Reading

Other articles I have done on V6

Basic IPV6 Mikrotik Firewall

Netflix, IPv6, and queing

IPv6 Security tidbits

IPv6 Test Sites


About the Author Justin Wilson

#routingrf #routinglight #bendinglight

Are you ready for 768k day?

As the global routing table increases, routers use more and more memory to hold these tables in memory. Most routers use what is called TCAM memory to hold routing tables. TCAM memory is much faster than normal RAM, which makes it ideal for accessing large routing tables on a router.  However, TCAM is generally viewed as a more expensive type of memory.

According to the CIDR report, the global routing table as of April 15 2019 was 772,711 routes. But Justin, you are warning me about 768,000 routes.  This is more than that already.  The short answer is many providers attempt to do some sort of aggregation with prefixes, which shrinks this number.

So why is this number important? in 2014 a similar situation arose.  This was called “512k day”.  Many vendors released patches and advisories recommending folks to raise their limit to 768,000 routes.  Why not just raise this to a million you say? Remember that TCAM memory is expensive so it’s not like normal ram.  As more and more folks run IPV6, it takes away TCAM memory and allocates it to the IPV6 routing table. In the “old days” we just had to worry about one IPV4 table taking up the TCAM memory. Now we have an ever-growing IPV6 table, which takes up memory as well. 768,000 was recommended by many vendors as a decent tradeoff of memory utilization.

Many experts do not expect this 768k day to be as service impacting as 512k day was.  Firmware updates, newer hardware, and increased awareness are some factors more operators are aware of.  However, there is a bunch of older hardware out there. One of the biggest concerns is the TCAM memory in the Cisco 6500 and 7600 routing platforms. These platforms simply do not have more memory to allocate.

If you own a 6500/7600 platform and are taking in full routes there are a few things you can do to help mitigate this. Obviously upgrading hardware is a choice.  Not everyone can do that.  One of the methods of dealing with this is to receive a default route from your upstream providers in addition to the full routes.  If you do this you can filter out /24 routes and decrease the routing table your router has to keep track of.  Anything that is a /24, which won’t be in the routing table at this point, will be sent to the default route.  You won’t have as much control over your routes to these destinations, but at least your router won’t be puking on itself.


As more and more smaller ISPs buy /24 allocations on the secondary market, we will see this problem increase.  IPV4 is not going away. Smaller ISPs are buying blocks to service their growing customer base and can’t afford to buy large allocations all at once.  So now we are seeing ISPs end up with four or five /24’s that can not be aggregated down like they could in the past.

MikroTik Router as a Vultr host

Recently I spun up a Mikrotik instance under Vultr for the purpose of doing some v6 testing. I was running into some problems with getting IPV6 to route properly. Vultr has IPV6 setup on their side to auto configure a gateway, etc. when it comes to IPV6. They are expecting a host, not a router. Why is his a problem?

The RFC states that nodes that act as routers are NOT to use SLAAC for IPv6 address configuration. In other words, routers that derived their interface IPv6 address from SLAAC cannot act as routers on that segment. This is a pretty hard set in stone thing when it comes to the RFC.

So how do we get around this in this instance? Go to IPV6…settings and turn off IPV6 forward.

IPv6 will start working at this point. The catch is, you won’t see neither address nor default route anywhere. It’s there, but it’s hidden. If you try to ping some external address, it will work.