This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
One of the problems installers run into on a few networks we manage is having the right tools to properly test a new install. Sure, an installer can run a test to speedtest.net to verify customers are getting their speed. Anyone who has done this long enough knows speedtest.net can be unreliable and produce inconsistent results. So, what then? Or what happens if you need to by-pass customer equipment easily? Most installers break out their laptop, spend a few minutes messing with settings and then authenticating themselves onto the network. Sometimes this can be easy, other times it can be challenging.
In steps the Mikrotik mAP.
What you are about to read is based on a MUM presentation by Lorenzo Busatti from http://routing.wireless.academy/ with my own spin on it. You can read his entire presentation on the mAP in PDF at : https://mum.mikrotik.com//presentations/US16/presentation_3371_1462179397.pdf . The meat of what we are talking about in this article starts on Page 50. If you want to watch the video you can do so at https://www.youtube.com/watch?v=VeZetH9uX_Y . The focus of this article starts around 21:00.
I have taken Lorenzo’s idea and have several different versions based upon the network. In most of our scenarios, the ethernet ports are what plug into the CPE or the customer’s equipment, and the technician connects to the mAP over wifi. This post covers using the mAP as an installer tool, not a traveling router. Lorenzo covers the travel option quite well in his presentation.
In this post, we focus on networks which use PPPoE. PPPoE networks usually are the ones who take much time to set up to diagnose. What we have done is set up an uncapped user profile that is available on every tower. Authentication can be done with local secrets or via radius. Depending on your IP design the user can get the same IP across the network, or have an IP that assigned to this user on each tower/routed segment. We could do an entire article on IP design.
On our Mikrotik, we setup ether1 to have a PPPoE client running on it. When the installer plugs this into the customers CPE the mAP will automatically “dial-out” and authenticate using the technician user we talked about earlier. Once this connection has is established, the mAP is set to turn on the red “PoE out” light on the mAP using the following code.
/system leds add interface=pppoe-out1 leds=user-led type=interface-status
Note. Our PPPoE interface is the default “pppoe-out1″ name. If you modify this, you will need to modify the led setup as well to match.
The red light gives the technician a visual indicator they have authenticated and should have internet. At the very least their mAP has authenticated with PPPoE. There are netwatch scripts mentioned in the above presentation which can kick on another LED indicating true internet reachability or other functions. In our case, we can assume if the unit authenticates with the tower, then internet to the tower is up. While this isn’t always the case if the Internet is down to the tower you quickly know or the NOC quickly knows. At least you hope so. We chose the PoE out led because we are not using POE on this setup and a red light is noticeable.
Once the technician has a connection they can connect to an SSID set aside for testing. In our case, we have set aside a “COMPANY_TECH” SSID. The tech connects to this on their laptop, and they are online. Since this is a static profile, you can set it up just like a typical customer, or you can give the tech user access to routers, APs or other devices. Our philosophy is you set up this SSID to mimic what a customer account experiences as closely as possible. It goes through the same firewall rules and ques just like a typical customer.
To further enhance our tool we can set up a VPN. This VPN can is accessible from the laptop with a second SSID named “COMPANY_VPN”. Once the technician switches over to this SSID they have access, over a preconfigured VPN on the mAP, to the network, from where they can access things customers can not, or at least should not be able to access. Many modern networks put APs, and infrastructure on separate VLANs not reachable from customer subnets. The VPN comes in handy here. You can access these things without changing security. If you plan on using this router internally, the type of VPN you choose is not as important as if you plan to modify the config so you can travel as is the case with the above MUM presentation. If you plan to travel an SSTP VPN is the most compatible. If it’s just inside your network, I would suggest an l2tp connection with IPsec.
Our third configuration on this is to set up the second ethernet port to be a DHCP client. This setup is handy for plugging into the customer router for testing or for places where DHCP is the method of access, for example, behind a Baicells UE. If your network does not use PPPoE, you could have one ethernet be a DHCP client, and the other be a DHCP server. We have found having the technicians connect wirelessly makes their lives easier. They can plug the unit in and not have to worry about cables being too short, or getting behind a desk several times to plug and unplug things.
So why go through all this trouble?
One of the first things you learn in troubleshooting is to eliminate as many variables as you can. By plugging this into your CPE, you have a known baseline to do testing. You eliminate things such as customer routers, customer PCs, and premise wiring. The mAP is plugged directly in CPE, whether it be wired or wireless. Experience has shown us many of the troubles customers experience are traced back to their router. Even if you provide the router, this can eliminate or point to that router as being a source of the problem if a technician needs to visit the customer.
Secondly, the mAP allows us to see and do more than your typical router. From the mAP we can run the Mikrotik bandwidth test tool from it to the closest router, to the next router inlines, all the way out to the internet. A while back I did an article titled “The Problem with Speedteststs“. This article explains many of the issues testing just using speedtest.net or other sites. Being able to do these kinds of tests is invaluable. If there are four Mikrotik routers between the customer and the edge of your network all four of them can be tested independently. If you have a known good host outside your network, such as the one we provide to our clients, then you can also test against that.
Having a Mikrotik test tool like this also allows you access to better logging and diagnostics. You can easily see if the ethernet is negotiating at 100 meg or a Gig. You can do wireless scans to see how noisy or busy 2.4GHZ is. You have easy to understand ping and traceroute tools. You also have a remote diagnostic tool which engineers can remote into easily to perform tests and capture readings.
Thirdly, the mAP allows the installer to establish a good known baseline at the time of install. You are not reliant on just a CPE to AP test, or a speedtest.net test.
How do we make this portable?
You may have noticed in my above pictures I have an external battery pack hooked up to my mAP. I am a fan of the Anker battery packs
Distributors such as ISP Supplies and CTIconnect have the mAP.
Finally, you will need a USB to MicroUSB cable
If you want you can add some double sided tape to hold the mAP to the battery pack for a neat package. I like the shorter cable referenced above in order to have a neat and manageable setup.
No matter what gear you use for delivering Internet to your customers, the mAP can be an invaluable troubleshooting tool for your field staff. I will be posting configs for Patreon and subscribers to download and configure their mAPs for this type of setup, as well as a road warrior setup. In the meantime, we do offer a setup service for $200, which includes the mAP, battery, USB cable and customized configuration for you.
Recently a client testing their 5G solution came to me asking for a solution to testing speed from their CBRS/5G/802.11ax clients. One of the requirements was it had to support greater than 1 gig speedtests as close to the devices as possible. This particular client has a small cell device which has room for a small form factor PC. The challenge was finding a small PC that could handle a 10 gig port.
In steps my buddy John from Columbus. John is up on hardware more than I am. After some talks, we settled on the following two pieces of hardware
Once we assembled this we need a router for the Internet and DHCP. We chose a RouterBoard hexS
As a not both of these will run off DC power. The Nuc comes with a 19Volt power supply so if you are running Pure DC you may want to drop from, say a 24volt battery bank to 19 volt with a Meanwell converter.
Proxmox was installed on the Nuc. Nothing crazy about this. Just make sure the thunderbolt adaptor is plugged in during install. For our purposes, we are just using the 10 gig adaptor. Proxmox recognizes the adaptor without a hiccup.
In some earlier blog posts I wrote about the self-hosted speedtests.
https://blog.j2sw.com/xisp/self-hosted-speedtest/ (Patreon Subscription Required)
I installed the self-hosted speedtest under a Centos Minimal Install. Everything was put on a 172.16.x.x network. This was done in order to prevent any conflicts with various types of Internet the Mikrotik may be plugged into. By default, port 1 is set up to be a DHCP client. In our setup, the Internet is the bottleneck, but we are not testing the Internet. We are testing clients on the 5g/CBRS/802.11ax network. Our 10 gig port on the nuc will be plugged into a 10 gig switch at the small cell, and not into our routerboard. The routerboard is just there to hand out DHCP and allow Internet access, if available.
Ubiquiti launches the Ubiquiti Speedtest, the first public test network integrated with enterprise network equipment. Ubiquiti Speedtest comprises a network of test servers and built-in speed test capabilities. Reports include uplink/downlink throughput and latency. Sharing the results is easy via email or social media.
It appears you can run this on a Ubuntu server or VM. They have an installer and a docker image. You can do browser-based speed tests or their WiFiman App.
Tests may run over LAN, Wi-Fi, or mobile networks. Ubiquiti Speedtest uses Ubiquiti test endpoints and provides automated and manual test target selection. The automated selection uses a combination of geolocation and latency measurements for determining the best servers. The algorithm may use several parallel endpoints for the best measurement accuracy.
Imagine this scenario. Outside your house, the most awesome superhighway has been built. It has a speed limit of 120 Mile Per Hour. You calculate at those speeds you can get to and from work 20 minutes earlier. Life is good. Monday morning comes, you hop in your 600 horsepower Nissan GT-R, put on some new leather driving gloves, and crank up some good driving music. Your pull onto the dedicated on-ramp from your house and are quickly cruising at 120 Miles an hour. You make it into work before most anyone else. Life is good.
Near the end of the week, you notice more and more of your neighbors and co-workers using this new highway. Things are still fast, but you can’t get up to speed like you could earlier in the week. As you ponder why you notice you are coming up on the off-ramp to your work. Traffic is backed up. Everyone is trying to get to the same place. As you are waiting in the line to get off the superhighway, you notice folks passing you by going on down the road at high rates of speed. You surmise your off-ramp must be congested because it is getting used more now.
Speedtest servers work the same way. A speedtest server is a destination on the information super-highway. Man, there is an oldie term. To understand how these servers work we need a quick understanding of how the Internet works. The internet is basically a bunch of virtual cities connected together. Your local ISP delivers a signal to you via Wireless, Fiber, or some sort of media. When it leaves your house it travels to the ISP’s equipment and is aggregated with your neighbors and sent over faster lines to larger cities. It’s just like a road system. You may get access via a gravel road, which turns into a 2 lane blacktop, which then may turn into a 4 lane highway, and finally a super-highway. The roads you take depend on where you are going. Your ISP may not have much control over how the traffic flows once it leaves their network.
Bottlenecks can happen anywhere. Anything from fiber optic cuts, oversold capacity, routing issues, and plain old unexpected usage. Why are these important? All of these can affect your results and can be totally out of control of your ISP and you. They can also be totally your ISP’s fault.
They can also be your fault, just like your car can be. An underpowered router can be struggling to keep up with your connection.Much like a moped on the above super-highway can’t keep up with a 600 horsepower car, your router might not be able to keep up either. Other things can cause issues such as computer viruses, and low performing components.
Just about any network can become a speedtest.net node or a node with some of the other speedtest sites. These networks have to meet minimum requirements, but there is no indicator of how utilized these servers are. A network could put up one and it’s 100 percent utilized when you go running a test. This doesn’t mean your ISP is slow, just the off-ramp to that server is slow.
The final thing we want to talk about is the utilization of your internet pipe from your ISP. This is something most don’t take into consideration. Let’s go back to our on-ramp analogy. Your ISP is selling you a connection to the information super-highway. Say they are selling you a 10 meg download connection. If you have a device in your house streaming an HD Netflix stream, which is typically 5 megs or so, that means you only have 5 megs available for a capacity test while that HD stream is happening. Speedtest only tests your current available capacity. Many folks think a speedtest somehow stops all the traffic on your network, runs the test, and starts the traffic. It doesn’t work that way. Your available capacity is only tested at that point in time. The same is true for any point between you and the speedtest server. Remember our earlier analogy about slowing down when you got to work because there were so many people trying to get there. They exceeded the capacity of that destination. However, that does not mean your connection is necessarily slow because people were zooming past you on their way to less congested destinations.
This is why results to a server should be taken with a grain of salt. They are a useful tool, but not an absolute. The speedtest server is just a destination. That destination can have bottlenecks, but others don’t. Even after this long article, there are many other factors which can affect Internet speed. Things we didn’t touch on like Peering, the technology used, speed limits, and other things can also affect your internet speed to destinations.
Our friend willie has a new video on a self-hosted speedtest server.