Mikrotik mAP for the WISP installer

One of the problems installers run into on a few networks we manage is having the right tools to properly test a new install. Sure, an installer can run a test to speedtest.net to verify customers are getting their speed.  Anyone who has done this long enough knows speedtest.net can be unreliable and produce inconsistent results. So, what then? Or what happens if you need to by-pass customer equipment easily? Most installers break out their laptop, spend a few minutes messing with settings and then authenticating themselves onto the network. Sometimes this can be easy, other times it can be challenging.

mAP with extenral battery pack

In steps the Mikrotik mAP.
What you are about to read is based on a MUM presentation by Lorenzo Busatti from http://routing.wireless.academy/ with my own spin on it. You can read his entire presentation on the mAP in PDF at : https://mum.mikrotik.com//presentations/US16/presentation_3371_1462179397.pdf . The meat of what we are talking about in this article starts on Page 50. If you want to watch the video you can do so at https://www.youtube.com/watch?v=VeZetH9uX_Y . The focus of this article starts around 21:00.

I have taken Lorenzo’s idea and have several different versions based upon the network.  In most of our scenarios, the ethernet ports are what plug into the CPE or the customer’s equipment, and the technician connects to the mAP over wifi.  This post covers using the mAP as an installer tool, not a traveling router. Lorenzo covers the travel option quite well in his presentation.

In this post, we focus on networks which use PPPoE. PPPoE networks usually are the ones who take much time to set up to diagnose.   What we have done is set up an uncapped user profile that is available on every tower.  Authentication can be done with local secrets or via radius.  Depending on your IP design the user can get the same IP across the network, or have an IP that assigned to this user on each tower/routed segment. We could do an entire article on IP design.

On our Mikrotik, we setup ether1 to have a PPPoE client running on it.  When the installer plugs this into the customers CPE the mAP will automatically “dial-out” and authenticate using the technician user we talked about earlier.  Once this connection has is established, the mAP is set to turn on the red “PoE out” light on the mAP using the following code.

/system leds
add interface=pppoe-out1 leds=user-led type=interface-status

Note. Our PPPoE interface is the default “pppoe-out1″ name. If you modify this, you will need to modify the led setup as well to match.

The red light gives the technician a visual indicator they have authenticated and should have internet. At the very least their mAP has authenticated with PPPoE. There are netwatch scripts mentioned in the above presentation which can kick on another LED indicating true internet reachability or other functions.  In our case, we can assume if the unit authenticates with the tower, then internet to the tower is up.  While this isn’t always the case if the Internet is down to the tower you quickly know or the NOC quickly knows.  At least you hope so. We chose the PoE out led because we are not using POE on this setup and a red light is noticeable.

Once the technician has a connection they can connect to an SSID set aside for testing.  In our case, we have set aside a “COMPANY_TECH” SSID. The tech connects to this on their laptop, and they are online.  Since this is a static profile, you can set it up just like a typical customer, or you can give the tech user access to routers, APs or other devices.  Our philosophy is you set up this SSID to mimic what a customer account experiences as closely as possible.  It goes through the same firewall rules and ques just like a typical customer.

To further enhance our tool we can set up a VPN.  This VPN can is accessible from the laptop with a second SSID named “COMPANY_VPN”. Once the technician switches over to this SSID they have access, over a preconfigured VPN on the mAP, to the network, from where they can access things customers can not, or at least should not be able to access. Many modern networks put APs, and infrastructure on separate VLANs not reachable from customer subnets.  The VPN comes in handy here. You can access these things without changing security. If you plan on using this router internally, the type of VPN you choose is not as important as if you plan to modify the config so you can travel as is the case with the above MUM presentation. If you plan to travel an SSTP VPN is the most compatible.  If it’s just inside your network, I would suggest an l2tp connection with IPsec.

Our third configuration on this is to set up the second ethernet port to be a DHCP client.  This setup is handy for plugging into the customer router for testing or for places where DHCP is the method of access, for example, behind a Baicells UE.  If your network does not use PPPoE, you could have one ethernet be a DHCP client, and the other be a DHCP server. We have found having the technicians connect wirelessly makes their lives easier.  They can plug the unit in and not have to worry about cables being too short, or getting behind a desk several times to plug and unplug things.

So why go through all this trouble?
One of the first things you learn in troubleshooting is to eliminate as many variables as you can. By plugging this into your CPE, you have a known baseline to do testing. You eliminate things such as customer routers, customer PCs, and premise wiring.  The mAP is plugged directly in CPE, whether it be wired or wireless. Experience has shown us many of the troubles customers experience are traced back to their router. Even if you provide the router, this can eliminate or point to that router as being a source of the problem if a technician needs to visit the customer.

Secondly, the mAP allows us to see and do more than your typical router. From the mAP we can run the Mikrotik bandwidth test tool from it to the closest router, to the next router inlines, all the way out to the internet. A while back I did an article titled “The Problem with Speedteststs“.  This article explains many of the issues testing just using speedtest.net or other sites.  Being able to do these kinds of tests is invaluable.  If there are four Mikrotik routers between the customer and the edge of your network all four of them can be tested independently. If you have a known good host outside your network, such as the one we provide to our clients, then you can also test against that. 

Having a Mikrotik test tool like this also allows you access to better logging and diagnostics.   You can easily see if the ethernet is negotiating at 100 meg or a Gig.  You can do wireless scans to see how noisy or busy 2.4GHZ is.  You have easy to understand ping and traceroute tools.  You also have a remote diagnostic tool which engineers can remote into easily to perform tests and capture readings.

Thirdly, the mAP allows the installer to establish a good known baseline at the time of install.  You are not reliant on just a CPE to AP test, or a speedtest.net test.

How do we make this portable?
You may have noticed in my above pictures I have an external battery pack hooked up to my mAP.   I am a fan of the Anker battery packs

Distributors such as ISP Supplies and CTIconnect have the mAP.

Finally, you will need a USB to MicroUSB cable

If you want you can add some double sided tape to hold the mAP to the battery pack for a neat package. I like the shorter cable referenced above in order to have a neat and manageable setup.

No matter what gear you use for delivering Internet to your customers, the mAP can be an invaluable troubleshooting tool for your field staff. I will be posting configs for Patreon and subscribers to download and configure their mAPs for this type of setup, as well as a road warrior setup. In the meantime, we do offer a setup service for $200, which includes the mAP, battery, USB cable and customized configuration for you.

Some WISP CPE grounding

A discussion which comes up over and over in the WISP space is grounding and proper installation of customer CPE. The folks at perfect-10 (https://www.perfect-10.tv/) were a vendor at #WISPAPALOOZA2018.  One of the best things I have seen them in a long time is the below photo they created. This is a great illustration of how a proper CPE goes.

Importance of PIM in WISP LTE deployments

PIM sweeps are a common thing in the Cellular field.   One of the first questions folks often ask is what is a PIM sweep? If you think of PIM testing as a passive test and line sweeping as an active test that is a good start.  PIM testing looks for problems with things like connectors, cables, and other “layer 1” items.  A PIM test is not a line sweep. Line sweeping measures the signal losses and reflections of the transmission system. this is typically VSWR.  A line sweep is an active test. It can not detect the same things a PIM test can.  Many HAM radio folks are familiar with a line sweep where the reflected power is measure in an antenna system. In a line sweep you deal with reflected power and all that.

What does a PIM test do?

When you do a PIM test typical two high power signals are injected into the antenna line.  You can actually pass a sweep test but not a PIM test.

I won’t go into PIM tests very much because you need high dollar units such as those from Anritsu and Kaelus. These cost 10’s of thousands of dollars new.  Sometimes you can find these used.  However, the next thing you will run into is understanding the output of such a device.  Cell crews go to week long certification classes to become a PIM certified tech from Anritsu and others.

What causes a PIM test to fail?

According to Kaelus the most common problems are:

• Contaminated surfaces or contacts due to dirt, dust, moisture or oxidation.
• Loose mechanical junctions due to inadequate torque, poor alignment or poorly prepared contact surfaces.
• Loose mechanical junctions caused transportation shock or vibration .
• Metal flakes or shavings inside RF connections.
• Poorly prepared RF connections
•Trapped dielectric materials (adhesives, foam, etc.)
•Cracks or distortions at the end of the outer conductor of coaxial cables caused by over tightening the back nut during installation.
• Solid inner conductors distorted in the preparation process causing these to be out of round or tapered over the mating length.
• Hollow inner conductors excessively enlarged or made oval during the preparation process.

Why does cable matter?

Cables do not typically cause PIM, but poorly terminated or damaged cables can and do cause problems.

Cables with Seams can cause issues.  The seam can corrode.  Plated copper, found in cheaper cables, can break away from the aluminum core. This actually allows small amounts of flaking to happen between the connector and the core of the cable.  This will cause PIM issues and is very hard to diagnose. Imagine little flakes inside a connector. You don’t see them until you break open the connector, and even then they may be pretty little flakes.

Cables can change their physical configuration as temperature varies. For instance, sunshine can warm cables, changing their electrical length. A cable that happens to be the right length to cancel out PIM when cool may show strong PIM after changing its length on a warm day, or, it can work the other way around, good when hot and bad when cold. In addition, the physical change in length can make a formerly good connection into a poor one, also generating PIM. Other environmental factors such as water in the connector or cable can be an issue, as with any RF setup.

I think I have PIM issues. What are some indications?

PIM often shows up as poor statistics from the affected antenna. One of the first and most direct indications of PIM can be seen in cells with two receive paths. If the noise floor is not equal between the two paths, the cause is likely PIM generated inside the noisy receive path.

How Do I prevent PIM issues?

Cable quality and connector quality are one of the biggest factors in the PIM quality of a LTE system.  Many WISPs are used to making their own LMR cables and putting on their own connectors.  There is a difference between a low PIM LMR-400 cable and normal LMR-400.  Same for connectors.  One of the recommendations today was to use 1/2” superflex heliax.

The easy recommendation is to buy pre-made cables that have already been PIM certified.  In a typical WISP setup, you do not have lots and lot of components in your setup. Buy already certified components from your distributors that are “Low PIM rated”.

Thresholds for Microwave backhauls in Librenms

If you are running Librenms this video will help you learn how to adjust the thresholds for signal strength when it comes to microwave/licensed backhauls.  This video focuses on Mimosa because that was the quickest handy thing.  Cambium and other manufacturers will be slightly different due to chains being combined and other things.

Capacity of a UBNT AP vs the number of clients

Note: I am in the process of updating this for AC based radios. This was published in 2014, but much is still relevant.

Almost all the time I get asked: “How many clients can an AP handle?” . My answer is always a very long and drawn out one. There is no set in stone answer. There are many factors which can affect this. I will go into some of these and then explain how to calculate this.

Some things that we will assume.
1.You are calculating on an 802.11N Ap with some kind of polling (TDMA, NSTREME, AIRMAX, etc)
2.You know the MCS values and/or data rates at channel widths.
3.When I say in an ideal situation I mean basically in the lab. This is our baseline. This means no outside noise, everything is working properly, and all the connected clients are excellent.

Before I get into what affects how many clients can an AP handle we need to shift our thinking a little. We don’t think in terms of how many clients can an AP handle. We need to think in terms of how much capacity an AP has. This is very important to think in these terms. If you do so things will become more clear and more quantifiable.

So now, on to what affects the total capacity of an AP.

1.The channel width. In and ideal situation you will get more Capacity out of a 20 mhz channel than you will a 10mhz channel.
2.Noise. In the real world you will have interference. If you have interference the noise floor drops, customer signals can’t reach maximum modulation, and there are retransmits.
3.Plain old signal. Things such as trees, distance, fresnel zone, and antenna gain all affect signal
4.The speed you are giving to each customer.
5.Overselling. The concept of overselling has been around since the dial-up days. You are betting your customers are not all online at the same exact time doing the exact same stuff. So you can oversell your capacity. I will explain this a little more in a bit how this factors in.

Okay, so let’s dive into this. I am going to use a Ubiquity Rocket M5 as an example. Again, this can be applied to any polling type N radio.

Say we have a Rocket M5. At a 20MHZ channel the best modulation this M5 will do is MCS 15 at 130 Megs of over the air. What do you mean Over the Air? Well there is a difference between actual throughput and the Wireless Data Rate (aka over the air). Your actual throughput/capacity will be 1/2 of the over the air rate minus a little for overhead. I factor in 10% overhead for easy figuring.

Back to our figuring. You have 130 megs of capacity on your AP in an ideal situation on a 20 mhz channel. If we do our math:
130 / 2 = 65 Megs of Capacity to sell on the AP.
Now here comes the overselling part.
If we oversell at a 2:1 ratio we have 130 Megs of capacity on the AP.
If we oversell at a 3:1 ratio we have 195 megs of capacity on the AP.

We can do higher ratios, but it starts to become a moving target. With the spread of Netflix, Youtube, Hulu, and other streaming services the average customer is sucking down more and more bandwidth for longer periods of time. Think of a restaurant with so many tables. If your customers are staying longer and longer, you don’t have as much seating capacity to turn over for new people to sit down and consume your food. This is for another blog post.

So, let’s say we are overselling at 3:1. We have 195 megs of capacity. We now need to think about what packages we are selling to our customers. If they are all say 5 meg packages, this means we can safely sell 39 connections to the AP. 195 / 5 = 39. You can figure up the math if you have 3 Meg, 10 meg, or a mixture.

Now to the real world (aka why do my customers hate me and my AP sucks?).

The following is a real AP in the wild.  Blacked out to protect the innocent from script kiddies.

ubnt-main-screen
Couple of things to Note (circled in Red).

20 MHZ Channel
Capacity at 45% . This is more important than anything, even CCQ.
43 clients associated.

Let’s apply our math we learned earlier. We know a 20 mhz channel nets us MCS15 – 130 Megs

Here is the kicker.  Our capacity is at 45%.  This means we only have 45% of 130 megs of Over the air capacity.  Take this in half (130 / 2= 65   45% of 65 = 29.25.
This means all 43 of these customers are sharing 29 megs of capacity on the AP.  And the quality isn’t the greatest (37%).  So this means there are retransmissions going on between the client and the AP. The client can’t talk as fast as it is capable of in most cases. This means you can’t oversell the AP as much due to the quality of the signals being poor.  It is important to note I am talking about the quality and capacity of the signals, not signal strengths.

If those 43 people are all paying for, let’s say, 2 Megs download.  That means your AP needs to support a minimum of 86 megs. Thats without overselling.  We only have 29 megs in the current state!

We need to get those capacity numbers up.  How do we do that?

1. Channel selection. A noisy channel will drag everyone down.

2. Antenna gain.  This can be done at both the client and the AP.  A higher gain or better quality antenna can cause the clients to “hear” better.  You might not get an increase in signal strengths, but you are looking for an increase in quality. I use a loudspeaker metaphor.  You can hear a loudspeaker from a far distance, but you might not always be able to make out what is being said.  If you can somehow make out what is being said more clearly, then you don’t have to have the speaker turn up the volume.

3. Shielding. This helps eliminate the amount of stuff a client or AP hears.

4. Channel Width.  Sometimes dropping the channel width down can increase signals, thus raising the overall capacity.  Keep in mind it will lessen the overall capacity of the AP.

5.Simply getting rid of customers that shouldn’t be installed.  We have all done installs that were iffy.  These can drag down the overall capacity.

I hope this has helped understand.  The biggest thing I want you all to take away from this is think in terms of the amount of capacity you have to sell, not the number of connections.

WISPAPALOOZA 2019 News and tidbits

WISPAPALOOZA 2019 is rapidly approaching October 14th-18th 2019.  I will be attending in several capacities.

FD-IX will have a booth with Athereal, Ascilis Telecom (booth 135).  Come by and say hi. We have some giveaways as well as a comfy spot for a chat.

I will be speaking and moderating a few panels.
I will be moderating
The Little Things
Wednesday – Oct 16, 2019
09:15 – 10:15am

I will be a speaker on

IPv6 – How and When?
Thursday – Oct 17, 2019
03:00 – 04:00pmHow to deploy IPv6 and when? (include ipv4 conservation here?

Some Vendors to check out

Services
Acilis Telecom
Atheral– Atheral is building customer-centric white-label and wholesale cloud solutions for Internet Service Providers
Divi Networks
Towercoverage.com – Quickly Create RF Propagation Maps for Anywhere in the World!
ToweroneInc – Climbing and personal safety

Distributors
CTI Connect
ISPSupplies – Distributor

Equipment
ALG – High-performance antennas
Alpha Wireless – designs and manufactures antenna solutions
Cambium Networks – Wireless equipment manufacturer. ePMP, PMP, and other equipment.

Some things to do
Cambium Open House – Monday, October 14th