Mikrotik mAP for the WISP installer

One of the problems installers run into on a few networks we manage is having the right tools to properly test a new install. Sure, an installer can run a test to speedtest.net to verify customers are getting their speed.  Anyone who has done this long enough knows speedtest.net can be unreliable and produce inconsistent results. So, what then? Or what happens if you need to by-pass customer equipment easily? Most installers break out their laptop, spend a few minutes messing with settings and then authenticating themselves onto the network. Sometimes this can be easy, other times it can be challenging.

mAP with extenral battery pack

In steps the Mikrotik mAP.
What you are about to read is based on a MUM presentation by Lorenzo Busatti from http://routing.wireless.academy/ with my own spin on it. You can read his entire presentation on the mAP in PDF at : https://mum.mikrotik.com//presentations/US16/presentation_3371_1462179397.pdf . The meat of what we are talking about in this article starts on Page 50. If you want to watch the video you can do so at https://www.youtube.com/watch?v=VeZetH9uX_Y . The focus of this article starts around 21:00.

I have taken Lorenzo’s idea and have several different versions based upon the network.  In most of our scenarios, the ethernet ports are what plug into the CPE or the customer’s equipment, and the technician connects to the mAP over wifi.  This post covers using the mAP as an installer tool, not a traveling router. Lorenzo covers the travel option quite well in his presentation.

In this post, we focus on networks which use PPPoE. PPPoE networks usually are the ones who take much time to set up to diagnose.   What we have done is set up an uncapped user profile that is available on every tower.  Authentication can be done with local secrets or via radius.  Depending on your IP design the user can get the same IP across the network, or have an IP that assigned to this user on each tower/routed segment. We could do an entire article on IP design.

On our Mikrotik, we setup ether1 to have a PPPoE client running on it.  When the installer plugs this into the customers CPE the mAP will automatically “dial-out” and authenticate using the technician user we talked about earlier.  Once this connection has is established, the mAP is set to turn on the red “PoE out” light on the mAP using the following code.

/system leds
add interface=pppoe-out1 leds=user-led type=interface-status

Note. Our PPPoE interface is the default “pppoe-out1″ name. If you modify this, you will need to modify the led setup as well to match.

The red light gives the technician a visual indicator they have authenticated and should have internet. At the very least their mAP has authenticated with PPPoE. There are netwatch scripts mentioned in the above presentation which can kick on another LED indicating true internet reachability or other functions.  In our case, we can assume if the unit authenticates with the tower, then internet to the tower is up.  While this isn’t always the case if the Internet is down to the tower you quickly know or the NOC quickly knows.  At least you hope so. We chose the PoE out led because we are not using POE on this setup and a red light is noticeable.

Once the technician has a connection they can connect to an SSID set aside for testing.  In our case, we have set aside a “COMPANY_TECH” SSID. The tech connects to this on their laptop, and they are online.  Since this is a static profile, you can set it up just like a typical customer, or you can give the tech user access to routers, APs or other devices.  Our philosophy is you set up this SSID to mimic what a customer account experiences as closely as possible.  It goes through the same firewall rules and ques just like a typical customer.

To further enhance our tool we can set up a VPN.  This VPN can is accessible from the laptop with a second SSID named “COMPANY_VPN”. Once the technician switches over to this SSID they have access, over a preconfigured VPN on the mAP, to the network, from where they can access things customers can not, or at least should not be able to access. Many modern networks put APs, and infrastructure on separate VLANs not reachable from customer subnets.  The VPN comes in handy here. You can access these things without changing security. If you plan on using this router internally, the type of VPN you choose is not as important as if you plan to modify the config so you can travel as is the case with the above MUM presentation. If you plan to travel an SSTP VPN is the most compatible.  If it’s just inside your network, I would suggest an l2tp connection with IPsec.

Our third configuration on this is to set up the second ethernet port to be a DHCP client.  This setup is handy for plugging into the customer router for testing or for places where DHCP is the method of access, for example, behind a Baicells UE.  If your network does not use PPPoE, you could have one ethernet be a DHCP client, and the other be a DHCP server. We have found having the technicians connect wirelessly makes their lives easier.  They can plug the unit in and not have to worry about cables being too short, or getting behind a desk several times to plug and unplug things.

So why go through all this trouble?
One of the first things you learn in troubleshooting is to eliminate as many variables as you can. By plugging this into your CPE, you have a known baseline to do testing. You eliminate things such as customer routers, customer PCs, and premise wiring.  The mAP is plugged directly in CPE, whether it be wired or wireless. Experience has shown us many of the troubles customers experience are traced back to their router. Even if you provide the router, this can eliminate or point to that router as being a source of the problem if a technician needs to visit the customer.

Secondly, the mAP allows us to see and do more than your typical router. From the mAP we can run the Mikrotik bandwidth test tool from it to the closest router, to the next router inlines, all the way out to the internet. A while back I did an article titled “The Problem with Speedteststs“.  This article explains many of the issues testing just using speedtest.net or other sites.  Being able to do these kinds of tests is invaluable.  If there are four Mikrotik routers between the customer and the edge of your network all four of them can be tested independently. If you have a known good host outside your network, such as the one we provide to our clients, then you can also test against that. 

Having a Mikrotik test tool like this also allows you access to better logging and diagnostics.   You can easily see if the ethernet is negotiating at 100 meg or a Gig.  You can do wireless scans to see how noisy or busy 2.4GHZ is.  You have easy to understand ping and traceroute tools.  You also have a remote diagnostic tool which engineers can remote into easily to perform tests and capture readings.

Thirdly, the mAP allows the installer to establish a good known baseline at the time of install.  You are not reliant on just a CPE to AP test, or a speedtest.net test.

How do we make this portable?
You may have noticed in my above pictures I have an external battery pack hooked up to my mAP.   I am a fan of the Anker battery packs

Distributors such as ISP Supplies and CTIconnect have the mAP.

Finally, you will need a USB to MicroUSB cable

If you want you can add some double sided tape to hold the mAP to the battery pack for a neat package. I like the shorter cable referenced above in order to have a neat and manageable setup.

No matter what gear you use for delivering Internet to your customers, the mAP can be an invaluable troubleshooting tool for your field staff. I will be posting configs for Patreon and subscribers to download and configure their mAPs for this type of setup, as well as a road warrior setup. In the meantime, we do offer a setup service for $200, which includes the mAP, battery, USB cable and customized configuration for you.

Some WISP CPE grounding

A discussion which comes up over and over in the WISP space is grounding and proper installation of customer CPE. The folks at perfect-10 (https://www.perfect-10.tv/) were a vendor at #WISPAPALOOZA2018.  One of the best things I have seen them in a long time is the below photo they created. This is a great illustration of how a proper CPE goes.

The importance of Network Monitoring Systems (NMS)

One of our open tickets on MidWest-IX is a member reporting slow speeds on their exchange port. After having them send us some data and a few e-mails back and forth we began looking at their switch port on the fabric.  Right away we noticed errors on the port. After a counter reset the errors were still incrementing

 19 runts  0 giants  1210 CRC  0 no buffer
 1329 input error  0 short frame  0 overrun  0 underrun  0 ignored

This led us to look at our LibreNMS data for this port.  A quick look shows on October 31st the port started seeing input errors.

By drilling down we are able to see exactly when this started happening

We now have responded to the customer to see if anything changed that day. Maybe a new switch, new optic, or software upgrade.  By having this data available in an NMS we were able to cut down on troubleshooting by a huge margin.  We now know when the issue started and are closer to the root cause of this.  Without this data, we would be spending more time trying to diagnose and track down issues.

How VRFs work

From Wikipedia https://en.wikipedia.org/wiki/Virtual_routing_and_forwarding
virtual routing and forwarding (VRF) is a technology that allows multiple instances of a routing table to co-exist within the same router at the same time. One or more logical or physical interfaces may have a VRF and these VRFs do not share routes therefore the packets are only forwarded between interfaces on the same VRF

Importance of PIM in WISP LTE deployments

PIM sweeps are a common thing in the Cellular field.   One of the first questions folks often ask is what is a PIM sweep? If you think of PIM testing as a passive test and line sweeping as an active test that is a good start.  PIM testing looks for problems with things like connectors, cables, and other “layer 1” items.  A PIM test is not a line sweep. Line sweeping measures the signal losses and reflections of the transmission system. this is typically VSWR.  A line sweep is an active test. It can not detect the same things a PIM test can.  Many HAM radio folks are familiar with a line sweep where the reflected power is measure in an antenna system. In a line sweep you deal with reflected power and all that.

What does a PIM test do?

When you do a PIM test typical two high power signals are injected into the antenna line.  You can actually pass a sweep test but not a PIM test.

I won’t go into PIM tests very much because you need high dollar units such as those from Anritsu and Kaelus. These cost 10’s of thousands of dollars new.  Sometimes you can find these used.  However, the next thing you will run into is understanding the output of such a device.  Cell crews go to week long certification classes to become a PIM certified tech from Anritsu and others.

What causes a PIM test to fail?

According to Kaelus the most common problems are:

• Contaminated surfaces or contacts due to dirt, dust, moisture or oxidation.
• Loose mechanical junctions due to inadequate torque, poor alignment or poorly prepared contact surfaces.
• Loose mechanical junctions caused transportation shock or vibration .
• Metal flakes or shavings inside RF connections.
• Poorly prepared RF connections
•Trapped dielectric materials (adhesives, foam, etc.)
•Cracks or distortions at the end of the outer conductor of coaxial cables caused by over tightening the back nut during installation.
• Solid inner conductors distorted in the preparation process causing these to be out of round or tapered over the mating length.
• Hollow inner conductors excessively enlarged or made oval during the preparation process.

Why does cable matter?

Cables do not typically cause PIM, but poorly terminated or damaged cables can and do cause problems.

Cables with Seams can cause issues.  The seam can corrode.  Plated copper, found in cheaper cables, can break away from the aluminum core. This actually allows small amounts of flaking to happen between the connector and the core of the cable.  This will cause PIM issues and is very hard to diagnose. Imagine little flakes inside a connector. You don’t see them until you break open the connector, and even then they may be pretty little flakes.

Cables can change their physical configuration as temperature varies. For instance, sunshine can warm cables, changing their electrical length. A cable that happens to be the right length to cancel out PIM when cool may show strong PIM after changing its length on a warm day, or, it can work the other way around, good when hot and bad when cold. In addition, the physical change in length can make a formerly good connection into a poor one, also generating PIM. Other environmental factors such as water in the connector or cable can be an issue, as with any RF setup.

I think I have PIM issues. What are some indications?

PIM often shows up as poor statistics from the affected antenna. One of the first and most direct indications of PIM can be seen in cells with two receive paths. If the noise floor is not equal between the two paths, the cause is likely PIM generated inside the noisy receive path.

How Do I prevent PIM issues?

Cable quality and connector quality are one of the biggest factors in the PIM quality of a LTE system.  Many WISPs are used to making their own LMR cables and putting on their own connectors.  There is a difference between a low PIM LMR-400 cable and normal LMR-400.  Same for connectors.  One of the recommendations today was to use 1/2” superflex heliax.

The easy recommendation is to buy pre-made cables that have already been PIM certified.  In a typical WISP setup, you do not have lots and lot of components in your setup. Buy already certified components from your distributors that are “Low PIM rated”.

ALG 5GHZ backhaul before and after

If you are looking for a U.S. stocking distributor of ALG products check out ISP Supplies
https://www.ispsupplies.com/search?keywords=alg&order=relevance:asc

The following are results from a series of tests of AGLcom’s parabolic dish antennas on an existing link that is 5.7 miles long. The link typically passes 80-90Mbs with a TX capacity of 140 Mbs and radios used are Ubiquiti AF5X operating at 5218 Mhz.  A full PDF with better Readability can be downloaded here..

The tests were taken in stages:

  1. 1)  The normal performance of the link was recorded.
  2. 2)  The 2′ dish at one end, B, was replaced with the AGLcom, C, dish and the link reestablished.The link performance was recorded.
  3. 3)  The 2′ dish at the other end, A, was replaced with the AGLcom, D, dish and the link reestablished. The link performance was recorded.
  4. 4)  The setting on the AF5xs were adjusted to optimize the link performance with data recorded.
  5. 5)  The 2′ dish, B was put back in the link and the performance was recorded.
  6. 6)  The ACLcom C was put back into place.

The tables below do not follow the test order as the third line of data was actually the last test performed.

Antennas:

A-Jirous JRC-29EX MIMO
B-Jirous JRC-29EX MIMO C-AGLcom – PS-6100-30-06-DP D-AGLcom – PS-6100-29-06-DP-UHP

Results:

Table 1 is the signal strength results of the various dishes on the link. The first line, A-B, is the original Jirous to Jirous. A is the first two columns of the link and are the A side and the last two columns are the B side on the link. What is of interest is that exchanging B to C in the second line brought the signal deviation between the channels to only 1db and 0 db as seen in Table 2. The third line was a result of replacing the horn on the A dish and optimizing the setting on the AF5X radios. This changed the signal by around 7db and improved the link capacity, Table 3. Clearly, the A dish had a problem with the original horn.

In the fourth line, D-B, the signal strength improved as well at the signal deviation on the two channels, Table 2 first two columns. This link was not optimized. The fifth line, D-C is both AGLcom dishes which improved the bandwidth, Table 3, and the signal deviations, Table 2. The final line, D-C, was the previous line optimized. The signal strengths moved closer together and the bandwidth improved.

Link Ch0 Ch1 Ch0 Ch1

  1. A-B  -73 -76
  2. A-C  -73 -74

A*-C -64 -66

  1. D-B  -63 -62
  2. D-C  -62 -62

D*-C -60 -60

-70 -74 -71 -71 -65 -66 -59 -59 -58 -58 -61 -61

Signal Strength (* optimized data) Table 1

Table 2 has four data columns, the first two being the measured results and the latter two being the measured difference from theory. The Jirous and AF5X calculators were used for the theory signals. Clearly the signal approached the theoritical limit with the optimization and with the change of dishes. The optimization improved the signal by ~9db for the link that we replaced the horn on the Jirous and by ~2db for the AGLcom link.

Link dSig dSig A-B 3 4 A-C 1 0 A*-C 2 1 D-B -1 0 D-C 0 0 D*-C 0 0

dSig dSig -16.5 -17.4 -17.0 -15.0 -8.0 -9.0 -13.3 -5.3 -7.0 -4.3 -5.0 -6.0

Signal strength variation from theory Table 2

The band width improvement was more obvious, Table 3, from 22 Mbs to 39 Mbs for the RX and 144 Mbs to 141 Mbs TX for the link with the horn replacement. The bandwidth improvement for the optimization of the AGLcom link was from 61Mbs to 66Mbs RX and from 211Mbs to 267Mbs for TX.

The bandwidth improvement from the original, optimized link to the AGLcom link is from 61Mbs RX to 67Mbs and from 210Mbs TX to 267Mbs. There is a clear improvement for the AGLcom link over the Jirous link.

Link BW-RX

  1. A-B  22.5
  2. A-C  39.0

A*-C 60.9

  1. D-B  61.4
  2. D-C  60.6

D*-C 66.6

BW-TX 144.6 141.4 210.0 211.0 215.0 267.6

Table 3

Conclusions:

The data supports a measurable improvement in both signal strength and bandwidth with the use of the AGLcom dishes. However, it is difficult to quantify the improvement. The Jirous dishes were identical whereas the AGLcom dishes were not. One of the jirous dishes was under performing initially but was repaired for the last tests. Additional testing is needed to provide accurate data analysis and performance comparison. The best performance tests would involve identical AGLcom dishes, ideally two links, one each of both types of dishes.

Setting up Cambium and Mikrotik for PPPoE

Cambium has a pretty good tutorial on their community forums on setting up PPPoE between Mikrotik and Cambium radios acting as clients.

https://community.cambiumnetworks.com/t5/ePMP-Networking/High-level-RouterOS-PPPoE-Server-Client-Configuration-Example/m-p/49625#M14

#packetsdownrange #epmp #mikrotik