Wireless ISP Network IP Scheme

For those of you starting out in the WISP field here is a good overview of a scalable Ip scheme for your WISP network. There are some caveats to this, which we will discuss later. This does not address IPV6, I have done some articles on IPv6 in the past that addresses IPV6.

Some benefits of this scheme
1.Logically separates equipment into groups.
2.Cookie-Cutter design. Fast time to depoyment
3.Easy to train on for techs and office
4.Scriptable

This following scheme uses private IP space for infrastructure. This scheme does not address end-user addressing. More on this in a later guide.

For each tower I route a /16 out of the 10.0.0.0/8 IP space. For example my first tower is
10.1.0.0/16 . Your first thought is that is alot of IP space. Yes it. However, it is private IP space and you can use as much as you want. The second thought is you can only use this scheme 255 times. Yes this is true. How many WISPs are growing beyond 255 towers. Most of the wisps I know are in the 100 tower sweet spot. There are larger and smaller wisps.

So lets break this down. We are using tower 1 as an example.

Wired Infrastrudture 
1-Not used due to default being VLAN 1 on most devices. Quarantine vlan 
10.1.2.0/24

2- Power and UPSes 
10.1.2.0/24

3- Switches and routers 
10.1.3.0/24

5-9 reserved for  future uses

Backhaul Management VLANS 

10-19 
10.1.10.0/24 – Backhaul 1
10.1.11.0/24 – BackHaul 2
etc.

Infrastructure Radio VLANS (tagged management) 

20-29 – 2GHZ equipment 
10.1.20.0/24 – First 2.4GHZ AP
10.1.21.0/24 – Send 2.4GHZ AP
10.1.22.0/24 – Third 2.4 GHZ AP Management

30-39 – 3GHZ Equipment 
10.1.30.0/24 – First 3GHZ AP

40-49 – RESERVED  

50-59 -5GHZ equipment 

60-69 – 60GHZ equipment 

70-79 – RESERVED  

80-89 – 80 GHZ 

90-99 – RESERVED possible IOT 

Customer VLANS (untagged) 

120-129 – 2GHZ equipment 
10.1.120.0/24 – Customer CPE on first 2.4GHZ AP
etc.

130-139 – 3GHZ Equipment 

140-149 – RESERVED 

150-159 -5GHZ equipment 

160-169 – 60GHZ equipment 

170-179 – RESERVED 

180-189 – 80 GHZ 

Point to Point customers 

200-299 

So why so much IP space? It helps keep your routing table small.

Medium WISP Core Network Design

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

WIFI calling port forwarding

Recently I came across a need to do some port forwarding for wifi calling. I have assembled a resource guide to help you if you need to do such things. IPSEC should be allowed per RFC 5996 https://tools.ietf.org/html/rfc5996 for all wifi calling

Verizon
https://community.verizonwireless.com/t5/Verizon-Wireless-Services/What-are-the-wifi-calling-firewall-ports-and-destination-IP/td-p/1080659
UDP ports 500 and 4500 open to sg.vzwfemto.com and wo.vzwwo.com

TMobile
https://www.t-mobile.com/support/coverage/wi-fi-calling-on-a-corporate-network
IPv4 Address Block: 208.54.0.0/17 and 66.94.0.0/19:
UDP Ports 500 and 4500
5061 for SIP/TLS
TCP port 443 and 993
Also whitelist the CRL server for DIGITS OTT and WFC 1.0: crl.t-mobile.com 206.29.177.36

AT&T
https://www.att.com/support/article/wireless/KM1114459/
UDP Ports 500 and 4500
TCP Port 143

Whitelist the following:

  • epdg.epc.att.net
  • sentitlement2.mobile.att.net
  • vvm.mobile.att.net

Sprint
UDP Ports 500 and 4500

Any of the above is subject to change.

Network troubleshooting tools

Recently, there was a thread on the NANOG list asking what were somne favorite network troubleshooting tools. I have taken many of these tools and created the following list.

http://ping.pe/
Simple pingport and dig commands

https://mtr.sh/
BGP Looking glass

https://perfops.net/mtr-from-world
Traceroute from various hosts on the net

http://www.traceroute6.net/
IPV6 tools (ping,traceroute,etc)

https://dnsviz.net/
Carious DNS tools

http://irrexplorer.nlnog.net/
Routing Registry object explorer

https://mxtoolbox.com/
DNS and Mail tools

WISPs: IPv6 is the answer to some of your issues

Many Wireless Internet Service Providers (WISPs), especially newer startups, struggle with nat issues and having enough public Ip addresses to go around. Invariably, you start running into double nat issues pretty quickly. Then you get the dreaded gamer call:

Many times they don’t know why they are even calling. They just know the magic box is saying this is bad. This is related to how many layers of nat between your edge and them. Many times you are natting at the edge, then you are natting at the customer router. If you have multiple customers behind the same nat at the edge this compounds it even more.

So what is the fix? Give the customer public addresses. But IPv4 is hard to get! I didn’t say IPV4 I said public addresses. IPv6 is a public address. When given the choice between v4 and v6 most modern streaming and gaming platforms will prefer v6. Xbox has supported a protocol called Teredo for a long time. You can learn all about Teredo in this PDF. Basically, it is a tunnel in which the Xbox speaks ipv6 over the tunnel. The ISP does not have to support v6, which does away with the above-mentioned nat issues.

Great! I don’t have to worry about IPv6, Microsoft has it taken care of for me. There are two problems with this statement. Problem number one. There are more companies out there than Microsoft. Sony Playstation Online, Apple gaming, and Steam are just a few. Second, you have overhead of tunnels. In the world of who can pull the joystick quicker, milliseconds count. You don’t want them wasted in tunnel overhead. Plus, v6 is beneficial for other service such as Netflix.

Any other service that runs into port issues behind nat can be solved with Ipv6, This can be voip, cameras, and other type services. This is providing the product or service supports v6 addresses.

So what is an ISP to do?
Awhile back I put together a resource guide for ISPs. You can find it at https://blog.j2sw.com/networking/ipv6/ipv6-planning-and-implementation-resources-for-the-xisp/

Internet Routing Registry Resources by j2sw

What is a routing registry?
From Wikipedia https://en.wikipedia.org/wiki/Internet_Routing_Registry
The Internet routing registry works by providing an interlinked hierarchy of objects designed to facilitate the organization of IP routing between organizations, and also to provide data in an appropriate format for automatic programming of routers. Network engineers from participating organizations are authorized to modify the Routing Policy Specification Language (RPSL) objects, in the registry, for their own networks. Then, any network engineer, or member of the public, is able to query the route registry for particular information of interest.

RFC2622 Routing Policy Specification Language (RPSL)

RFC2650 Using RPSL in Practice

RFC7682 Considerations for Internet Routing Registries (IRRs) and routing Policy Configuration

General IRR Information

http://www.irr.net/
Includes links to various registries, FAQs, and other info

https://www.gin.ntt.net/support-center/policies-procedures/routing-registry/ntt-route-registry-frequently-asked-questions/
NTT route registry FAQ

https://www.seattleix.net/irr-tutorial
Seattle Internet Exchange IRR Tutorial

https://archive.nanog.org/meetings/nanog51/presentations/Sunday/NANOG51.Talk34.NANOG51%20IRR%20Tutorial.pdf
NANOG Routing registry tutorial

General How-Tos

https://fcix.net/whitepaper/2018/07/14/intro-to-irr-rpsl.html
A Quickstart Guide to Documenting Your Prefixes with IRR. This mainly uses the older ARIN e-mail templates.


Arin Specific

https://www.arin.net/resources/manage/irr/userguide/
Arin’s userguide for working with their IRR

https://www.arin.net/resources/manage/irr/irr-online-implementation
Notes on working with ARINs web-based


Other Regional Registries

African Network Coordination Centre (AFRNIC)
https://afrinic.net/internet-routing-registry

Asian-Pacific Network Coordination Centre (APNIC)
https://www.apnic.net/manage-ip/apnic-services/routing-registry/

American Registry for Internet Numbers (ARIN)
https://www.apnic.net/manage-ip/apnic-services/routing-registry/

Latin American and Caribbean Internet Addresses Registry (LACNIC)
https://www.lacnic.net/innovaportal/file/3512/1/internet-routing-registries.pdf

Reseaux IP Eauropeens Network Coordination Centre (RIPE NCC)
https://www.ripe.net/manage-ips-and-asns/db/support/managing-route-objects-in-the-irr

Tools

https://github.com/6connect/irrpt
A collection of tools which allow ISPs to easily track, manage, and utilize IPv4 and IPv6 BGP routing information stored in Internet Routing Registry (IRR) databases. Some of these tools include automated IRR data retrieval, update tracking via CVS, e-mail notifications, e-mail based notification for ISPs who still do human processing of routing information, and hooks for automatically deploying prefix-lists on routers.

https://www.radb.net/query
The RADB whois server provides information collected from all the registries that form part of the Internet Routing Registry. 

https://github.com/irrdnet/irrd
Internet Routing Registry daemon version 4 is an IRR database server, processing IRR objects in the RPSL format.

ARIN resources and the Service Provider

Internet Service Providers (ISPs) can be intimidated by all of the facets of working with the American Registry of Internet Numbers (ARIN). I have put together a guide that outlines common things you, as a service provider, need to do.

This guide is not an end-all how-to. Throughout, I am posting videos and links taken from the ARIN site to help. This article is more of an outline of what a service provider needs to do.

The majority of the steps below will be done through ARIN’s online ticketing system.

This is broken down into the following Sections
1. Create a Point of Contact (POC) record
2. Creating an Organization (ORG-ID)
3. Requesting an Autonomous System Number (ASN)
4. Requesting IPv6 space
5. Requesting IPV4 space
6. Source Validation
7. Reverse DNS
8. Routing Registry
9. RPKI
10. Notes and tips

Creating a Point of Contact (POC)
Point of Contact (POC) records are the foundation of your ARIN account. This record is the way you manage your resources. There are different types of POC accounts. https://www.arin.net/resources/guide/account/records/poc/ will tell you everything you need to know about POC records. Creating this record will take mere minutes to make.

Creating an Organization
Once you have a POC record created, you will create an Organization and associate your POC with that ORG-ID. ARIN will attach your resources to your org-id. You will need your federal EIN and your registered business address for this stage. This stage takes a few days to get verified due to ARIN needing to verify you are who you say you are

Requesting an ASN
An Autonomous System Number (ASN) will be the first resource an ISP will request. The ASN allows you to participate in BGP by advertising your IP blocks to peers. The ASN will require to state your routing policy, usually BGP, and at least two peers, you will be establishing BGP. If you don’t have two peers, say your plans in this section.

Once you have met the criteria and you will be asked to fill out an officer attest paper. This statement is a paper stating the information you have submitted is correct and truthful. Once you will out this form and submit it you will then receive an invoice. Once this invoice is paid, you will receive your ASN. This stage can take several days, depending on how much back and forth goes on, asking to clarify information.

Request IPv6 space
I put this as the next stage for a few reasons. The first is you should be moving toward IPv6. At the very least, dual-stack your network. Second, requesting IPV6 space will get you familiar with how ARIN looks at requests.

You are required to state how your network is laid out, what type of network, and how you plan to deploy addresses. Be prepared to give a diagram of your system. You may have to go back and forth a few times, depending on how much detail you provided on your first request.

Just like your ASN, you will be required to sign another office attest, pay the bill, and then the Ip space will be allocated.

Requesting IPV4 space
Requesting IPV4 space is pretty close to requesting V6 space, but ARIN is more strict on their criteria these days due to the shortage of space. If you are looking to transition you can get. /24 of v4 for your v6 transition.

If you choose to request IPV4 space you will be put on a waiting list with others who have also requested space. Details on the waiting list can be found at https://www.arin.net/resources/guide/ipv4/waiting_list/ . ARIN is currently doing quarterly distributions to folks on the waitlist*. I put an asterisk on the previous statement because there are several variables listed at the waitlist site linked above. Some include:

  • Only organizations holding an aggregate of a /20 or less of IPv4 address space may apply and be approved.
  • The maximum-size aggregate that an organization may qualify for at any one time is a /22.

The site says they do quarterly distributions. I believe this gives ARIN time to reclaim IP space and do a cleanup on it. Depending on when you submit you may have to wait several months or longer for an allocation.

As with V6 space and ASN, you have to do another officer attest, pay your invoice, and then it is allocated.

Origin AS
Origin AS validation is a check and balance. From Arin’s https://www.arin.net/resources/registry/originas/
The Origin Autonomous System (AS) field is an optional field collected by ARIN during all IPv4 and IPv6 block transactions (allocation and assignment requests, reallocation and reassignment actions, transfer and experimental requests). This additional field is used by IP address block holders (including legacy address holders) to record a list of the Autonomous System Numbers (ASNs), separated by commas or whitespace, from which the addresses in the address block(s) may originate.

This is simply a field you fill in on your ARIN account. When you get IP space from ARIN this is *usually* automatic.

Reverse DNS
You will need to point your IP blocks to your or hosted DNS servers for the reverse entries. Many different entities pay attention to reverse DNS entries. If you have clients who run mail servers or similar services, you will need a reverse DNS entry. More information at https://www.arin.net/resources/manage/reverse/

Routing Registry
More and more companies, such as Hurricane Electric, are requiring routing registry entries. I did a pretty in-depth article on routing registries. https://blog.j2sw.com/networking/routing-registries-and-you/
ARIN now has a web-based system for setting up route objects. This web mehtod takes some of the learning curve out of adding things into the ARIN registry. Many exchanges, including FD-IX, are moving toward routing registry support.

RPKI
RPKI is another validation method for verifying you are the proper owner of resources, especially IP blocks. https://www.arin.net/resources/manage/rpki/ . Hosted RPKI is the easiest way to get started with RPKI.

I did an article related to RPKI at https://blog.j2sw.com/networking/bgp/hurricane-electric-now-requires-irr-and-rpki/

Notes
Working with ARIN is a pretty straightforward, but sometimes confusing for the newbie. I offer a package for $799 (plus ARIN fees) where I do all the above for you. I have done this so much over the years we have templates and other shortcuts for the various things done.

If you choose to do this on your own some tips.
1. Don’t be afraid to provide more detail than asked.
2. The ARIN helpdesk is actually helpful. If you get stuck call or e-mail them. They have probably answered your question before and are willing to help.
3. Be prepared to provide information about your network, especially with IPv4 requests. ARIN is wanting to know if you are/will be using resources efficiently.

If you get IPv4 space I would recommend adding the new IP block to your advertisements. Allow it to be learned by the various reverse Geolocation folks. After a week check your blocks using the links on this page: http://thebrotherswisp.com/index.php/geo-and-vpn/. This applies to space allocated from ARIN or purchased from a broker.

If you are looking to purchase blocks for a broker, yu need to get pre-approval from ARIN. Learn more at https://www.arin.net/resources/registry/transfers/preapproval/

Hurricane Electric now requires IRR and filters invalid RPKI

If you are a Hurricane Electric customer you may be receiving e-mails like the following:

Dear ASXXX,

Routing Security Report for ASXXX

Hurricane Electric cares about your routing security.  We filter all BGP sessions using prefix filters based on IRR and RPKI.

This report is being sent to help you identify prefixes which may need either their IRR or RPKI information created or updated 
and to also help you identify possibly hijacked routes you may be accepting and reannouncing.  

Routes with RPKI status INVALID_ASN strongly indicate a serious problem.

IPv4 SUMMARY

Routes accepted: 3
Routes rejected: 3
Routes with RPKI status VALID: 0
Routes with RPKI status INVALID: 0

IPv6 SUMMARY

Routes accepted: 1
Routes rejected: 0
Routes with RPKI status VALID: 0
Routes with RPKI status INVALID: 0

We currently do not have a valid as-set name for your network.  Please add an export line to your aut-num ASXXXX 
that references your as-set name.  For example,

export: to AS-ANY announce your-as-set-name

If you do not currently have an as-set, we recommend you create one named ASXXXX:AS-ALL

Your as-set should contain just your ASN and your customers' ASNs and/or as-sets (not your peers or upstream providers).

What does this mean for you as a service provider? If you use Hurricane Electric as transit or peer with them on an exchange you will need to have ROAs for your blocksand have routing registry objects. I did a tutorial based upon Arin which can be found at: https://blog.j2sw.com/networking/routing-registries-and-you/

In short you need to do the following:

  • Create a mntner object (equivalent of a user account) to give you the ability to create IRR objects in your selected IRR database
  • Create an aut-num to represent your autonomous system and describe its contact information (admin and technical) and your routing policy
  • Create an as-set to describe which autonous system numbers your peers should expect to see from you (namely your own and your transit customers)
  • Create a route/route6 object for every prefix originated from your network
  • Update your peeringdb profile to include your IRR peering policy
  • Generate RPKI https://www.arin.net/resources/manage/rpki/roa_request/#creating-a-roa-in-arin-online

Clarification:
Some folks are confusing having valid ROAs with your router supporting RPKI with route origin validation in real-time. These two are separate things. You create ROA records with your RIR, such as ARIN, which has nothing to do with route validation on your router.

Also, HE is filtering any RPKI INVALID routes. Does this mean they are requiring RPKI? You be the judge.



New Speed Test server for Patreons

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon at $4 or more
Already a qualifying Patreon member? Refresh to access this content.

OpenVPN, rooter project, and Mikrotik

Over the past couple of weeks, I have been fighting with getting an LTE device running The Rooter Project to establish an OpenVPN connection with a Mikrotik router. Apparently, OPENVPN is the only option when it comes to VPNs on The Rooter Project. For the purpose of this article, I am going to refer to the software as “the rooter”. This is just to denote the device running The Rooter Project software. In my case, this is a GL.iNET GL-X750 LTE device.

There are two parts to this setup. The OpenVPN setup on the Mikrotik and the setup on the rooter.

Mikrotik Setup

The Mikrotik setup is pretty straight forward. There are some great tutorials out there for a more in-depth setup. The RouterOS version I used for this setup is 6.47.

Creating Certificates
You will need to create 3 certificates on the Mikrotik.
1. cert_export_ca-certificate.crt
2.cert_export_client-certificate.crt
3.cert_export_client-certificate.key

/certificate
add name=ca-template common-name=example.com days-valid=3650 key-size=2048 key-usage=crl-sign,key-cert-sign
add name=server-template common-name=*.example.com days-valid=3650 key-size=2048 key-usage=digital-signature,key-encipherment,tls-server
add name=client-template common-name=client.example.com days-valid=3650 key-size=2048 key-usage=tls-client

Signing Certificates
Once you have created the above certificates you will need to sign them with the following

/certificate
sign ca-template name=ca-certificate
sign server-template name=server-certificate ca=ca-certificate
sign client-template name=client-certificate ca=ca-certificate

Exporting Certificates
Run the following commands to add a passphrase to your key certificate and export them to files

/certificate
export-certificate ca-certificate export-passphrase=""
export-certificate client-certificate export-passphrase=j2sw123com

This will give you three files: cert_export_ca-certificate.crtcert_export_client-certificate.crt, and cert_export_client-certificate.key. Download these out of “files” from the Mikrotik to the same computer you have access to the rooter on. I like to rename them to ca.crtclient.crt, and client.key so I can keep track of what is what.



Rooter Client Setup

Caveats
I could not find out how to make the operating system read a config file I would edit by hand. Even after a reboot, the config file would not be read. I am not sure if there is a command to read it into the running-config. If someone knows, let me know and that will make this process much easier.

client
dev tun
proto tcp
remote example.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
cipher AES-128-CBC
auth SHA1
auth-user-pass
redirect-gateway def1
verb 3

In my rooter, the config is in /var/etc. I would cat this occasionally to make sure I did not have any extra options turned on. Since I could not make my edits the file stick, I would make the below changes in the GUI and verify they matched up to my above file.

If your OpenVPN is using a username and password create a file named passowrd.txt and put the username on the first line and the password on the second.

You will need that file along with the three files you generate on the Mikrotik above.

Log in to the router and create you an open VPN instance. In my case, I named it Nexstream because this is who I was working for on this project. You can name it anything you want.

Click on edit and you will be brought to the following screens. Fill them out as shown.

When you get to the bottom this is where you upload your password.text and your cert and key files. If you see anything missing go to the bottom and select the field and click add.

Make sure to hit save and apply before proceeding. Click on “switch to advanced configuration”. Match up your configuration with the following screenshots, which match up with the above config file. You are just basically making the proper checkboxes to match the plain text config I posted above. Again, if anyone knows how to get OpenVPN. on the rooter to read the config in let me know.

Once you have the GUI part done and the certs uploaded to the rooter you will need to deal with the keyphrase via the command line. Simply SSH to the rooter. The below code is a generic code for changing the client.key to not ask for a passphrase anymore.

cd /etc/luci-uploads/
openssl.exe rsa -in client.key -out client.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

Couple of things to note about the process.
1. Your location may vary. You must either be inside the directory with your keys or provide the path to the keys in the OpenSSL command

2.when I uploaded the keys it changed them to cbid.openvpn.FRIENDLYNAME.key.

what my actual code looked like to change the passphrase

cd /etc/luci-uploads/
openssl.exe rsa -in cbid.openvpn.vpnout.key -out cbid.openvpn.vpnout.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

If everything goes well you will be rewarded with the following screen on your OpenVPN main page. If, for some reason, it does not start the system log is actually pretty informative on what is going on.