CCR1016 BGP route pull down

This morning I had a Mikrotik CCR1016 where I had to change the router ID, which caused all the sessions to reset. The following is a screenshot of the time it took to re-learn all of the peers. Obviously, the smaller prefixes were learned pretty quickly. It took about 10 minutes to learn two full IPv4 route tables and about 5 minutes to learn the IPv6 routing tables.

This is why I always get full routes plus a default from the upstream when it warrants full routes. This way I can have slow convergence time like this and still have traffic flowing.

OpenVPN, rooter project, and Mikrotik

Over the past couple of weeks, I have been fighting with getting an LTE device running The Rooter Project to establish an OpenVPN connection with a Mikrotik router. Apparently, OPENVPN is the only option when it comes to VPNs on The Rooter Project. For the purpose of this article, I am going to refer to the software as “the rooter”. This is just to denote the device running The Rooter Project software. In my case, this is a GL.iNET GL-X750 LTE device.

There are two parts to this setup. The OpenVPN setup on the Mikrotik and the setup on the rooter.

Mikrotik Setup

The Mikrotik setup is pretty straight forward. There are some great tutorials out there for a more in-depth setup. The RouterOS version I used for this setup is 6.47.

Creating Certificates
You will need to create 3 certificates on the Mikrotik.
1. cert_export_ca-certificate.crt
2.cert_export_client-certificate.crt
3.cert_export_client-certificate.key

/certificate
add name=ca-template common-name=example.com days-valid=3650 key-size=2048 key-usage=crl-sign,key-cert-sign
add name=server-template common-name=*.example.com days-valid=3650 key-size=2048 key-usage=digital-signature,key-encipherment,tls-server
add name=client-template common-name=client.example.com days-valid=3650 key-size=2048 key-usage=tls-client

Signing Certificates
Once you have created the above certificates you will need to sign them with the following

/certificate
sign ca-template name=ca-certificate
sign server-template name=server-certificate ca=ca-certificate
sign client-template name=client-certificate ca=ca-certificate

Exporting Certificates
Run the following commands to add a passphrase to your key certificate and export them to files

/certificate
export-certificate ca-certificate export-passphrase=""
export-certificate client-certificate export-passphrase=j2sw123com

This will give you three files: cert_export_ca-certificate.crtcert_export_client-certificate.crt, and cert_export_client-certificate.key. Download these out of “files” from the Mikrotik to the same computer you have access to the rooter on. I like to rename them to ca.crtclient.crt, and client.key so I can keep track of what is what.



Rooter Client Setup

Caveats
I could not find out how to make the operating system read a config file I would edit by hand. Even after a reboot, the config file would not be read. I am not sure if there is a command to read it into the running-config. If someone knows, let me know and that will make this process much easier.

client
dev tun
proto tcp
remote example.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
cipher AES-128-CBC
auth SHA1
auth-user-pass
redirect-gateway def1
verb 3

In my rooter, the config is in /var/etc. I would cat this occasionally to make sure I did not have any extra options turned on. Since I could not make my edits the file stick, I would make the below changes in the GUI and verify they matched up to my above file.

If your OpenVPN is using a username and password create a file named passowrd.txt and put the username on the first line and the password on the second.

You will need that file along with the three files you generate on the Mikrotik above.

Log in to the router and create you an open VPN instance. In my case, I named it Nexstream because this is who I was working for on this project. You can name it anything you want.

Click on edit and you will be brought to the following screens. Fill them out as shown.

When you get to the bottom this is where you upload your password.text and your cert and key files. If you see anything missing go to the bottom and select the field and click add.

Make sure to hit save and apply before proceeding. Click on “switch to advanced configuration”. Match up your configuration with the following screenshots, which match up with the above config file. You are just basically making the proper checkboxes to match the plain text config I posted above. Again, if anyone knows how to get OpenVPN. on the rooter to read the config in let me know.

Once you have the GUI part done and the certs uploaded to the rooter you will need to deal with the keyphrase via the command line. Simply SSH to the rooter. The below code is a generic code for changing the client.key to not ask for a passphrase anymore.

cd /etc/luci-uploads/
openssl.exe rsa -in client.key -out client.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

Couple of things to note about the process.
1. Your location may vary. You must either be inside the directory with your keys or provide the path to the keys in the OpenSSL command

2.when I uploaded the keys it changed them to cbid.openvpn.FRIENDLYNAME.key.

what my actual code looked like to change the passphrase

cd /etc/luci-uploads/
openssl.exe rsa -in cbid.openvpn.vpnout.key -out cbid.openvpn.vpnout.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

If everything goes well you will be rewarded with the following screen on your OpenVPN main page. If, for some reason, it does not start the system log is actually pretty informative on what is going on.

Mikrotik Routeros 7.0beta7

What’s new in 7.0beta7 (2020-Jun-3 16:31):

!) added Layer3 hardware offloading support for CRS317-1G-16S+RM more info here: https://wiki.mikrotik.com/wiki/Manual:CRS3xx_series_switches#L3_Hardware_Offloading
!) enabled BGP support with multicore peer processing (CLI only);
!) enabled RPKI support (CLI only);
!) ported features and fixes introduced in v6.47;
!) routing updates, complete status report: https://help.mikrotik.com/docs/display/ROS/v7+Routing+Protocol+Status
!) system kernel has been updated to version 5.6.3;
*) other minor fixes and improvements;

Mikrotik BGP firewall rules for security

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

Simple Mikrotik DNS cache flush script

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

PTP Backhaul Complete link kit #1

I have been asked a few times on what I would pick for a PTP link. here is what I would recommend as option #1.  You can use the RF Elements link planner to judge how far you can go. If you want to dive deeper you can use the Cambium LinkPlanner, which is a download. This is an unlicensed 5GH link. if you want some real-world data on this link you can visit https://blog.j2sw.com/xisp/the-addition-of-rf-elements-horns-to-a-ptp550-link/

2x Cambium PTP 550
https://www.ispsupplies.com/Cambium-Networks-C050055H001A
PTP 550 Data Sheet

 

 

 

2x RF Elements Ultrahorn
https://www.ispsupplies.com/RF-Elements-UH-CC-5-24
Ultrahorn Data Sheet

 

 

4x LMR Jumpers
https://www.ispsupplies.com/NM-NM-L4-36

 

 

 

8x Cold Shrink
https://www.ispsupplies.com/COLDSHRINK-12MDN

 

 

There are several other distributors to buy from as well. I choose ISP supplies because they refer business to me and do not have a consulting program that competes with my services.

 

How many customers on an ap? wrong question

Several years ago, I did an article on How many customers can I fit on an AP? I figured with the introduction of MU-MIMO and other things, it was time for an update. Several concepts still apply, but we now have Multi-User MIMO, better filtering, and better technology. One of the biggest questions I hear is, “How many customers can I put on an Access point?”. In this article, I will explain some of the ways to answer this question. Some of this will be geared toward certain products but will be an overall way of answering the question.

Thinking in terms of how many customers you can put on an Access Point is flawed thinking. What you really should be thinking of is how much capacity do I have to sell on an AP. From this, you can apply a formula to know how many customers an Access Point can support with quantifiable data.

Firstly, some things to know.  This article applies to mainly point-to-multipoint radios.  Most of your multipoint radios you come across are half-duplex radios.  The radios receive or transmit, but not at the same time. The over the air rate vs. real throughput come into play as a result. More on this later. Before we get into everything we have to know what affects the customer data rates.  I will break this into two sections. Ideal environment and the real world.

The Ideal Environment
This mainly has to do with radio specs and such.  You have channel width, data rates, and signal to noise to worry about.

Channel width is the first thing to consider. The bigger the channel, the more bits you can flow. If we want to use an analogy, we could compare this to a road or a water pipe. The bigger the road, the more cars that can drive down that road at faster speeds. A larger water pipe can flow more water. As with anything, there are drawbacks. The larger the channel, the more susceptible you are to interference.

Data rates and modulation are the next factors.  The higher the data rate the more capacity the client radio has.  Data rates are influenced by the channel width, radio limitations, and environmental factors.  Think of data rates as the top speed of your client radios. Just like a car road conditions are a huge influencer.

Signal to noise is one of the most critical factors overlooked. I have included this in the ideal and real-world sections for a couple of essential reasons. In the ideal environment, radio manufacturers publish the signal to noise needed to achieve max modulation. Modulation should be looked at first when it comes to a radio not performing as well as it should. The first thing I always look at is what is the current signal to noise.  For example, a Cambium 450M (Medusa) access point states,in the Spec sheet, that in order to achieve an 8x modulation, which is 256QAM you have to have a signal to noise ratio of 32dB.  This chart means if your noise floor is a -80, you have to have a signal of *at least* -48.  In the real world, this isn’t always achievable. Physics can fickle that way. If you want to geek on what QAM is you can watch the following video

The real-world environment
As many of you know the real world can be totally different than the lab environment.  Let’s discuss some factors which can alter the modulation rates, which then affect your overall throughput on an AP.

RF Landscape of a link

The RF “landscape” is the most significant influencer. In other words, how noisy is the spectrum? How many other devices does your access point “hear”? I always use the crowded room analogy. If you have a couple of people in a room, it’s easy to hear them and more comfortable to talk faster (modulation rate). As more people enter the room, you have to find a corner with a smaller group to talk (change channels). As the room becomes even more crowded, you have to speak a little slower because those around you are noisy and a distraction. Your modulation rate has to lower to have an intelligent conversation.

Line of sight is the next major issue. If a customer has any obstruction between them and the AP, the modulation level to drop because it has to deal with the extra noise. This is simple physics. Not only does the signal get degraded if it has to pass through objects or even dense air, but it is also deflected. This deflection is referred to as multipath. Other factors that influence modulation are the quality of antennas, the quality of any cables between the antenna and the AP, environmental factors such as bodies of water, and many other items. these are beyond the scope of this article.

On to determining the total capacity of an AP

Let’s take a Cambium ePMP 3000 ap as an example. This is a 4X4 Multi-User MIMO radio.   What this means is it can transmit four streams to a user at once.  This increases the bandwidth to the client. So where does the multi-user part come in? Most clients are not able to take advantage of the Access Point’s (AP) full capacity so the AP talks to multiple clients at once because it has the capacity to do so.

So let’s run some numbers.  The published spec sheet of an ePMP 3000 radio is a total capacity of 1.2 Gbps.  This radio is a TDD system. This means you over the air rate is half of your actual throughput due to the half-duplex nature of the radio.  It can only send or receive at one time, not both.

Now that we know our radio will do approximately 600 megs of capacity minus some overhead we can factor in oversubscription.

Oversubscription
Oversubscribing in the ISP world has been going on since the dial-up days. When managed properly, it is not a bad thing. The theory is that not every user is online at the same time doing the same things. Out of ten households doing things on the Internet at any given moment in time, you may have three or four streaming Netflix, two watching Youtube videos, three checking Instagram/Facebook/Twitter, and one just reading webpages. Let’s say each of them is paying for a 25 meg down by 5 meg up speed package. Out of those 10 accounts the Netflix streamers may be using 5 megs, the Youtube watchers may be using 3, and the rest are using a combined 5 meg. Out of 250 megs of sold capacity, those 10 accounts only use 31 megs at that point in time. Out of those users, only the streaming services are using that bandwidth the most. In an earlier article, I did a video on a Netflix stream at my house. As customer plans have more bandwidth available, they are grabbing data less frequently because they can grab bigger chunks at a time. This blog post illustrates this as well as this video

Here is where oversubscription becomes a moving target. Not every household is the same. Some may have two or three devices that stream at the same time.  Some may only have one.  Some may watch streaming services very little.

So how do you plan for oversubscription?
In today’s world of streaming a 3:1 oversubscription ratio is a pretty safe bet.  Depending on your customers you might be able to go 4:1, 5:1, or even more.  The faster your plans the less time the customer gets on and off the connection.

Formula
So let’s put it all together.
600 megs of AP capacity at a 1:1 ratio
1200 megs of AP capacity at a 2:1 ratio
1800 megs of AP capacity at a 3:1 ratio

For easy figuring, we will say we are selling 20 meg packages.
1:1 we can sell 30 20 meg packages
2:1 we can sell 60 20 meg packages

Will these numbers hold up in the real world? In most cases, they will not due to the real world conditions mentioned earlier in this article.  If you keep all of your customers at high MCS rates you should expect 70-80 percent capacity numbers in a real-world scenario.  Your mileage may vary. So let’s adjust our numbers.

70 percent of 600 megs is 420 megs
420 at 1:1
840 at 2:1
1260 at 3:1

Those same 20 meg packages
1:1 we can sell 21
2:1 we can sell 42
3:1 we can sell 63

Is the above formula absolute? It is just designed to give you an idea. The following link was published today. it shows 72 ePMP clients on a single AP. As I have stated the client connection isn’t the whole story.  Look at the throughput running through the AP to illustrate the formula is highly dependent on your customers and how they use the service. Remember when I talked about channel width and data rates? Pay attention to these in the video.

In conclusion think of how much capacity you have on an Access Point instead of just customer numbers.  The numbers can be impressive, as in the above video, but don’t tell the entire story.  Customer counts on an AP are nice to know and you can take the above formula to determine how many you can put on at what levels.

#packetsdownrange #epmp #rfelements #cambium