OTV and VXLAN differences

A great article on explaining what OTV is and how it compares to VXLAN.

OTV(Overlay Transport Virtualization) is a technology that provide layer2 extension capabilities between different data centers. In its most simplest form OTV is a new DCI (Data Center Interconnect) technology that routes MAC-based information by encapsulating traffic in normal IP packets for transit”

https://www.routexp.com/2018/03/vxlan-and-otv-what-is-difference.html

IPV6 Point-to-point addressing

Recently, there has been a discussion on what to use for point-to-point links under ipv6. Over the years I have seen providers use a wide variety of subnets across a point-to-point link.  Anything from a /64, to /122,124, and the 127. There are arguments for each. But let’s look at some RFCs.

https://tools.ietf.org/html/rfc6164
This was the original RFC on point-to-point links advocating for a /127. It was moved to a “historical” status in 2012 after a /127 was found to be damaging.

https://tools.ietf.org/html/rfc3627
This RFC explains some of the issues with using a /127. In a nutshell.

 Using /127 can be especially harmful on a point-to-point link when
   Subnet-router anycast address is implemented.

All of this brings us to some current RFC drafts and further reading
https://tools.ietf.org/id/draft-palet-v6ops-p2p-links-00.html

Some notes on point-to-point addressing
1. A subnet mask using something shorter than a /64 breaks some IPv6 functionality. A point-to-point link does not use the “broken” features anyway.

  1. The above 6164 RFC said vendors had to support /127s.  Many wrote code to comply with the RFC, which has now been obsoleted.
  2. A /64 could expose the link to security issues.

As of this writing, there are many approaches.  One approach is to set aside a /64 for the point-to-point but only use a /127 out of that /64.  You don’t re-use anything else out of that /64.  Other approaches involve using a  /126, /120 or a /112 are being accepted until this is all figured out.  So, why not a /122 or something? In short, it all has to do with the math of the subnet breakdown.

Everything you wanted to know about NTP

Network Time Protocol (NTP) is a service that can be used to synchronize time on network connected devices.   Before we dive into what NTP is, we need to understand why we need accurate time.

The obvious thing is network devices need an accurate clock.  Things like log files with the proper time stamp are important in troubleshooting.  Accurate timing also helps with security prevention measures.  Some attacks use vulnerabilities in time stamps to add in bad payloads or manipulate data. Some companies require accurate time stamps on files and transactions as well for compliance purposes.

So what are these Stratum levels I hear about?
NTP has several levels divided into stratum. All this is the distance from the reference clock source.  A clock which relays UTC (Coordinated Universal Time) that has little to no delay (we are talking nanoseconds) are Stratum-0 servers. These are not used on the network. These are usually atomic and GPS clocks.  A Stratum-0 server is connected to time servers or stratum-1 via GPS or a national time and frequency transmission.  A Stratum 1 device is a very accurate device and is not connected to a Stratum-0 clock over a network.  A Stratum-2 clock receives NTP packets from a Stratum-1 server, a Stratum-3 receives packets from a Stratum-2 server, and so on.  It’s all relative of where the NTP is in relationship to Stratum-1 servers.

Why are there levels?
The further you get away from Stratum-0 the more delay there is.  Things like jitter and network delays affect accuracy.  Most of us network engineers are concerned with milliseconds (ms) of latency.  Time servers are concerned with nanoseconds (ns). Even a server directly connected to a Stratum-0 reference will add 8-10 nanoseconds to UTC time.

My Mikrotik has an NTP server built in? Is that good enough?
This depends on what level of accuracy you want. Do you just need to make sure all of your routers have the same time? then synchronizing with an upstream time server is probably good enough. Having 5000 devices with the same time, AND not having to manually set them or keep them in sync manually is a huge deal.

Do you run a VOIP switch or need to be compliant when it comes to transactions on servers or need to be compliant with various things like Sox compliance you may need a more accurate time source.

What can I do for more accurate time?
Usually, a dedicated appliance is what many networks use.  These are purpose built hardware that receives a signal from GPS. the more accurate you need the time, the more expensive it will become.  Devices that need to be accurate to the nanosecond are usually more expensive than ones accurate to a microsecond.

If you google NTP Appliance you will get a bunch of results.  If you want to setp up from what you are doing currently you can look into these links:

http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html

How to Build a Stratum 1 NTP Server Using A Raspberry Pi

 

Building a Stratum 1 NTP Server with a Raspberry Pi

 

Mikrotik mAP for the WISP installer

One of the problems installers run into on a few networks we manage is having the right tools to properly test a new install. Sure, an installer can run a test to speedtest.net to verify customers are getting their speed.  Anyone who has done this long enough knows speedtest.net can be unreliable and produce inconsistent results. So, what then? Or what happens if you need to by-pass customer equipment easily? Most installers break out their laptop, spend a few minutes messing with settings and then authenticating themselves onto the network. Sometimes this can be easy, other times it can be challenging.

mAP with extenral battery pack

In steps the Mikrotik mAP.
What you are about to read is based on a MUM presentation by Lorenzo Busatti from http://routing.wireless.academy/ with my own spin on it. You can read his entire presentation on the mAP in PDF at : https://mum.mikrotik.com//presentations/US16/presentation_3371_1462179397.pdf . The meat of what we are talking about in this article starts on Page 50. If you want to watch the video you can do so at https://www.youtube.com/watch?v=VeZetH9uX_Y . The focus of this article starts around 21:00.

I have taken Lorenzo’s idea and have several different versions based upon the network.  In most of our scenarios, the ethernet ports are what plug into the CPE or the customer’s equipment, and the technician connects to the mAP over wifi.  This post covers using the mAP as an installer tool, not a traveling router. Lorenzo covers the travel option quite well in his presentation.

In this post, we focus on networks which use PPPoE. PPPoE networks usually are the ones who take much time to set up to diagnose.   What we have done is set up an uncapped user profile that is available on every tower.  Authentication can be done with local secrets or via radius.  Depending on your IP design the user can get the same IP across the network, or have an IP that assigned to this user on each tower/routed segment. We could do an entire article on IP design.

On our Mikrotik, we setup ether1 to have a PPPoE client running on it.  When the installer plugs this into the customers CPE the mAP will automatically “dial-out” and authenticate using the technician user we talked about earlier.  Once this connection has is established, the mAP is set to turn on the red “PoE out” light on the mAP using the following code.

/system leds
add interface=pppoe-out1 leds=user-led type=interface-status

Note. Our PPPoE interface is the default “pppoe-out1″ name. If you modify this, you will need to modify the led setup as well to match.

The red light gives the technician a visual indicator they have authenticated and should have internet. At the very least their mAP has authenticated with PPPoE. There are netwatch scripts mentioned in the above presentation which can kick on another LED indicating true internet reachability or other functions.  In our case, we can assume if the unit authenticates with the tower, then internet to the tower is up.  While this isn’t always the case if the Internet is down to the tower you quickly know or the NOC quickly knows.  At least you hope so. We chose the PoE out led because we are not using POE on this setup and a red light is noticeable.

Once the technician has a connection they can connect to an SSID set aside for testing.  In our case, we have set aside a “COMPANY_TECH” SSID. The tech connects to this on their laptop, and they are online.  Since this is a static profile, you can set it up just like a typical customer, or you can give the tech user access to routers, APs or other devices.  Our philosophy is you set up this SSID to mimic what a customer account experiences as closely as possible.  It goes through the same firewall rules and ques just like a typical customer.

To further enhance our tool we can set up a VPN.  This VPN can is accessible from the laptop with a second SSID named “COMPANY_VPN”. Once the technician switches over to this SSID they have access, over a preconfigured VPN on the mAP, to the network, from where they can access things customers can not, or at least should not be able to access. Many modern networks put APs, and infrastructure on separate VLANs not reachable from customer subnets.  The VPN comes in handy here. You can access these things without changing security. If you plan on using this router internally, the type of VPN you choose is not as important as if you plan to modify the config so you can travel as is the case with the above MUM presentation. If you plan to travel an SSTP VPN is the most compatible.  If it’s just inside your network, I would suggest an l2tp connection with IPsec.

Our third configuration on this is to set up the second ethernet port to be a DHCP client.  This setup is handy for plugging into the customer router for testing or for places where DHCP is the method of access, for example, behind a Baicells UE.  If your network does not use PPPoE, you could have one ethernet be a DHCP client, and the other be a DHCP server. We have found having the technicians connect wirelessly makes their lives easier.  They can plug the unit in and not have to worry about cables being too short, or getting behind a desk several times to plug and unplug things.

So why go through all this trouble?
One of the first things you learn in troubleshooting is to eliminate as many variables as you can. By plugging this into your CPE, you have a known baseline to do testing. You eliminate things such as customer routers, customer PCs, and premise wiring.  The mAP is plugged directly in CPE, whether it be wired or wireless. Experience has shown us many of the troubles customers experience are traced back to their router. Even if you provide the router, this can eliminate or point to that router as being a source of the problem if a technician needs to visit the customer.

Secondly, the mAP allows us to see and do more than your typical router. From the mAP we can run the Mikrotik bandwidth test tool from it to the closest router, to the next router inlines, all the way out to the internet. A while back I did an article titled “The Problem with Speedteststs“.  This article explains many of the issues testing just using speedtest.net or other sites.  Being able to do these kinds of tests is invaluable.  If there are four Mikrotik routers between the customer and the edge of your network all four of them can be tested independently. If you have a known good host outside your network, such as the one we provide to our clients, then you can also test against that. 

Having a Mikrotik test tool like this also allows you access to better logging and diagnostics.   You can easily see if the ethernet is negotiating at 100 meg or a Gig.  You can do wireless scans to see how noisy or busy 2.4GHZ is.  You have easy to understand ping and traceroute tools.  You also have a remote diagnostic tool which engineers can remote into easily to perform tests and capture readings.

Thirdly, the mAP allows the installer to establish a good known baseline at the time of install.  You are not reliant on just a CPE to AP test, or a speedtest.net test.

How do we make this portable?
You may have noticed in my above pictures I have an external battery pack hooked up to my mAP.   I am a fan of the Anker battery packs

Distributors such as ISP Supplies and CTIconnect have the mAP.

Finally, you will need a USB to MicroUSB cable

If you want you can add some double sided tape to hold the mAP to the battery pack for a neat package. I like the shorter cable referenced above in order to have a neat and manageable setup.

No matter what gear you use for delivering Internet to your customers, the mAP can be an invaluable troubleshooting tool for your field staff. I will be posting configs for Patreon and subscribers to download and configure their mAPs for this type of setup, as well as a road warrior setup. In the meantime, we do offer a setup service for $200, which includes the mAP, battery, USB cable and customized configuration for you.

The importance of Network Monitoring Systems (NMS)

One of our open tickets on MidWest-IX is a member reporting slow speeds on their exchange port. After having them send us some data and a few e-mails back and forth we began looking at their switch port on the fabric.  Right away we noticed errors on the port. After a counter reset the errors were still incrementing

 19 runts  0 giants  1210 CRC  0 no buffer
 1329 input error  0 short frame  0 overrun  0 underrun  0 ignored

This led us to look at our LibreNMS data for this port.  A quick look shows on October 31st the port started seeing input errors.

By drilling down we are able to see exactly when this started happening

We now have responded to the customer to see if anything changed that day. Maybe a new switch, new optic, or software upgrade.  By having this data available in an NMS we were able to cut down on troubleshooting by a huge margin.  We now know when the issue started and are closer to the root cause of this.  Without this data, we would be spending more time trying to diagnose and track down issues.

Cisco and Verizon demonstrate multi-haul

As Internet traffic grows and becomes more dynamic, optical transport networks for sub-sea, terrestrial long haul and metro need more capacity. The ability to deploy capacity quickly is equally important to handle the increasingly dynamic nature of the traffic. The concept of a multi-haul transport platform, as introduced by Andrew Schmitt of Cignal AI, becomes very appealing for achieving this ability to scale with speed while maintaining operational simplicity – a single platform for all requirements. A critical element of the multi-haul optical platform is the flexibility of the coherent optics to be tuned to fine granularity in order to meet the reach-capacity target of any given network.

https://ciscocentral.blogspot.com/2019/02/cisco-and-verizon-to-demonstrate.html

How VRFs work

From Wikipedia https://en.wikipedia.org/wiki/Virtual_routing_and_forwarding
virtual routing and forwarding (VRF) is a technology that allows multiple instances of a routing table to co-exist within the same router at the same time. One or more logical or physical interfaces may have a VRF and these VRFs do not share routes therefore the packets are only forwarded between interfaces on the same VRF

Looking Glass Links

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon at $0.01 or more
Already a qualifying Patreon member? Refresh to access this content.