Speed validation for compliance on ISP networks

Does your ISP network need a way to validate customer speeds? RDOF compliance? State compliance for broadband speeds and latency? as many of you know there are problems with using public speedtest servers to validate customer speeds.

J2networks, in conjunction with Aloremnetworks, has an on-premise solution that is easy to manage, small footprint, and scalable. Our appliance lets you add a speed test server in almost any part of your network. It’s small footprint and low power draw makes it easy to fit in remote cabinets as well as the data center.

Contact us for more details and pricing.

BGP, a single /24 and two diverse non-connected exit points

I am starting to see the following scenario more and more as IPv4 space is hard to get, but isn’t.

With ARIN it is still possible to get an IPv4 allotment. Many smaller ISPs qualify for a /24 and can get one if they wait long enough on the ARIN waiting list. a /24 of IPv4 space is the smallest block that 99% of the Internet allows to be advertised on the Capital I Internet. There are filter rules in place that drop smaller prefixes because that is the agreed upon norm.

So what happens if you are an ISP and you have a shiny new /24 but you have two networks which are not connected. Let’s look at our scenario.

The above network have no connectivity between the two of them on the internal side. These could be half way across the world or next door. If they were half way across the world it would make sense to try and get another /24. Maybe they are either side of a big mountain or one is down in a valley and there is no way to get a decent link between the two networks.

So what is a way you can use this /24 and still be able to assign IP addresses to both sides of the network? One way is to use a tunnel between your two edge routers.

Without the tunnel the scenario is traffic could come into network1, but if the IP is assigned on network 2 it will come back as unreachable. BGP is all about networks finding the shortest path to other networks. You don’t have much control over how networks find your public IP space if you have two providers advertising the same information. Some of the Internet will come in Network2 and some will come in Network1.

By running a tunnel between the two you can now subnet out that /24 into two eqal /25s and assign one /25 Network1 and one /25 to Network2 or however you want to. You can make the tunnel a GRE, EOIP, or other tunnel type. If I am using Mikrotik I prefer to use EOIP. If it’s another vendor I tend to use GRE.

Once the tunnel is established you can use static routing, OSPF, or your favorite IGP (interior Gateway Protocol) to “tell” one side about the routes on the other side. Let’s look at a fictional use.

In the above example our fictional ISP has an IPv4 block of 1.2.3.0/24. They have two networks separated by a tall mountain range in the center. It’s too cost prohibitive to run fiber or a wireless backhaul between the two networks so they have two different upstream providers. The ISP is advertising this /24 via BGP to Upstream1 from the Network 1 router. Network 2 router is also advertising the same /24 via BGP to Upstream 2.

We now create a Tunnel between the Mikrotiks. As mentioned before this can be EOIP, GRE, etc. We won’t go into the details of the tunnel but let’s assume the ISP is using Mikrotik. We create an EOIP tunnel (tons of tutorials out there) between Network 1 router and Network 2 router. Once this is established we will use 172.16.200.0/30 as our “Glue” on our tunnel interfaces at each side. Network 1 router gets 172.16.200.1/30. Network 2 router gets 172.16.200.2/30

To keep it simple we have a static route statement on the Network 1 Mikrotik router that looks like this:

/ip route add dst-address=1.2.3.129/25 gateway=172.16.200.2

This statement routes any traffic that comes in for 1.2.3.128/25 via ISP 1 to network1 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

Conversely, we have a similar statement in the Network 2 Mikrotik router

/ip route add dst-address=1.2.3.0/25 gateway=172.16.200.1

This statement routes any traffic that comes in for 1.2.3.0/25 via ISP 2 to network2 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

It’s as simple as that. You can apply this to any other vendor such as Cisco, Juniper, PFSense, etc. You also do not have to split the network into even /25’s like I did. You can choose to have os of the ips available on one side and route a /29 or something to the other side.

The major drawback of this scenario is you will takef a speed hit because if the traffic comes in one side and has to route across the tunnel it will have to go back out to the public internet and over to the other ISP.

#packetsdownrange

VMware Fusion and MacOS Big Sur

While doing some work tonight I received an update notification from VMware Fusion, which I use to run Windows 10 inside Mac OS.

• Fusion 11 can not run on macOS 11 Big Sur because it depends on Kernel Extensions which can no longer be used on macOS 11. 

Never fear there is a new version with newer pricing. Ugh!!

Finally, with this new version of Fusion, VMware is also shaking up its pricing with a new Fusion 12 Player licensing structure that is free for personal use. Fusion 12 Player users who want to use the software for commercial use will be able to purchase a license for $149, with an upgrade license available for $89 to those currently running Fusion 10 or 11.

A professional-level Fusion 12 Pro offers some additional power-user features and allows you to run on up to three devices (including Windows or Linux PCs running VMware’s Workstation Pro), with pricing set at $199 for a new license or $99 as an upgrade. Discounted educational pricing will also be available for Fusion 12, although many students will also be able to take advantage of the free personal use license.

Users who purchased Fusion 11.5 or Fusion 11.5 Pro after June 15 will be eligible for a free upgrade to Fusion 12 Player (commercial license) or Fusion 12 Pro.

https://www.macrumors.com/2020/08/20/vmware-fusion-12/

So why is the upgrade needed? Fusion 12 phases out VMWare’s use of kernel extensions by using new Apple APIs to support its containers and virtual machines.

Bad news? It is not available yet. https://blogs.vmware.com/teamfusion/2020/08/announcing-fusion-12-and-workstation-16.html

Netbox Mikrotik Ansible Config generator

So, due to Covid, weather and everything else I am quite behind on blog updates and such. this is one that kinda fell through the cracks. I meant to get this out much sooner than now. My buddy Schylar Utley has a pretty cool projects for optimizing CPE deployments and such.

Check them out at https://github.com/MajesticFalcon

I have included an old video to give you an idea. I am sure things have changed since this video was created.

Amazon Sidewalk is announced

Amazon has announced a new feature called Sidewalk.

When enabled, Sidewalk uses a small portion of your Internet bandwidth to provide these services to you and your neighbors. This setting will apply to all of your supported Echo and Ring devices that are linked to your Amazon account. 

In essence, what this does is it uses Bluetooth running in the 900MHZ band to form an adjacency with neighbors and other Sidewalk enabled devices. In the old CB terms, we might call this a sideband connection. Amazon is using a portion of a sidewalk enabled device to create a shared network in your community. Some of the examples they use are for locating lost pets with a sensor tied to their collar. As the pet passes your neighbor’s sidewalk enabled hub you will see that on a map. Another example might be a sensor that can’t see your wifi network but might see the neighbors.

You can read more at the link below, but here is the quick and dirty from their FAQ.

How does Amazon Sidewalk work?
Customers with a Sidewalk Bridge (today, many Echo devices, Ring Floodlight Cams and Ring Spotlight Cams) can contribute a small portion of their internet bandwidth, which is pooled together to create a shared network that benefits all Sidewalk-enabled devices in a community. Amazon Sidewalk uses Bluetooth, the 900 MHz spectrum and other frequencies to extend coverage and provide these benefits.

What does Amazon charge for use of the network?
Amazon does not charge any fees to join Amazon Sidewalk, which uses a small portion of bandwidth from a Sidewalk Bridge’s existing internet service. Standard data rates from internet providers may apply.

How will Amazon Sidewalk impact my personal wireless bandwidth and data usage?
The maximum bandwidth of a Sidewalk Bridge to the Sidewalk server is 80Kbps, which is about 1/40th of the bandwidth used to stream a typical high definition video. Today, when you share your Bridge’s connection with Sidewalk, total monthly data used by Sidewalk, per account, is capped at 500MB, which is equivalent to streaming about 10 minutes of high definition video.

Will I know what other Sidewalk-enabled devices are connected to my Bridge?
Preserving customer privacy and security is foundational to how we’ve built Amazon Sidewalk. Information transferred over Sidewalk Bridges is encrypted and Bridge customers are not able to see that Sidewalk-enabled devices are connected to their Bridge. Customers who own Sidewalk-enabled devices will know they are connected to Sidewalk but will not be able to identify which Bridge they are connected to. For more information, visit our whitepaper here.

https://smile.amazon.com/Amazon-Sidewalk/b/?node=21328123011&ref_=pe_41837490_547199770_pe_mp_tran_aucc_sidewalk_learn

Mikrotik Connection tracking and CPU usage

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

Wiping an ssd

For those of you who are getting more and more into ssd drives you probably have run across this. You have an ssd taken out of production, but you don’t want to destroy it. Maybe it’s a new drive with life left. Maybe you want to use it for a lab. How do you wipe it? SSDs have a limited write cycle so you don’t want to chew up drive life by a standard wipe. These typical involve writing to the disk multiple times. So what do you do?

https://kb.iu.edu/d/aiut

Podcast: Equinix, T-Mobile, IRS

Equinix publishes their global interconnection Index report.
https://www.equinix.com/gxi-report/

T-Mobile expands residential LTE broadband
interesting to note there are “no data caps” however, During congestion, Home Internet customers may notice speeds lower than other customers due to data prioritization. 
https://www.t-mobile.com/news/network/t-mobile-expands-home-internet-to-more-than-450-cities-towns-left-high-and-dry-by-att

Geek out on 100 GIG AWS networking
https://toonk.io/aws-network-performance-deep-dive/index.html

IRS under investigation

IRS Under Investigation For Illegally Tracking Americans via Their Phones

Today’s links

This podcast sponsored by Fiber Data Internet exchange (fd-ix.com)

https://www.eff.org/ Privacy news for all things digital and beyond

https://slashdot.org/ All things geeky

 

As the days get shorter

As summer comes to a close here in Indiana and the daylight fades to dusk around 5PM I find myself with more time to devote back to the writings on this blog, the Youtube videos, Podcasts and editing of the videos I already have done.

Look for more content coming soon.

Mikrotik RouterOS and CPU usage

There always is a lot of talk about Mikrotik RouterOS CPU usage. I wanted to take a few minutes and go over a real-world example and explain some of the ins and outs when discussing Mikrotik CPU usage.

Let’s talk about the router in question. This is a CCR1016-12s-1S+. This is a 16 core 1.2GHz per core and 2GB RAM tilex based router. It is currently pulling in 1,764,849 IPv4 routes. There are two transit provider BGP feeds, multiple direct peers, an Internet Exchange peer to dual-route servers. The router handles a little over 3 gigs of routed traffic at peak times. Most of the traffic is on VLANs coming from a Cisco switch to the SFPPlus port.

One of the first things people turn on is the overall CPU usage within winbox. I like to think of this as an overall view of the CPUs on this router. Keep in mind there are 16.

Th next thing to investigate when it comes to CPU is to open up System..resources. Once there clock on CPU.

Mikrotik System..resources

It will then bring up a screen that looks like the following.

Oh My we have 100% CPU! Must replace this router ASAP! Calm down, remember you have 16 cores. So, why is this CPU at 100% and what ramifications does this have?

Remember earlier when we talked about BGP? In Mikrotik, BGP is not a multi-core aware process. This means BGP is limited to just one core to do it’s work. Since there are always routes being withdrawn and re-added to the routing table it is a busy process. Lots of math calculations going on. The key thing is this is expected behavior on a router running multiple BGP peers such as this one. This is not a bad thing, but not ideal. Throwing more cores at BGP is not the answer. Optimizing the process, as it has been done in V7 is the way to go.

If we expand the CPU window we will notice other processes are multi-core aware and.or are spreading their load among different cores.

As you can see we are in pretty good shape. We have a few CPUs above 50% utilization but, only a few. I will keep reminding you of the fact we have 16 of them.

Closing notes:
Diagnosing CPU issues can get a little complicated because routers like the 3011 have some have the majority of their ports shared with a single CPU bus.  https://wiki.mikrotik.com/images/f/f3/Switch_chip_block_diagram.png. As you can tell in the diagram there are 5 ports which share 1 Gig to the CPU.  The fact that an actual switch chip with hardware offloading is in the middle helps, but the bus is still oversold.  This is one reason consolidating routers to an actual switch will make a difference.  

Janis Megis from Mikrotik had presentation at MUM, which is a little older now, still sheds a lot of light on how Mikrotik CPU works.  https://mum.mikrotik.com/presentations/US10/Megis.pdf There is some pretty interesting stuff starting on page 14

With Mikrotik switching to ARM processors we will see huge differences with them and RotuerOS7. We will see less cores, but better utilization of those cores. The new 2004 with all SFP and 2 25 gig ports only has 4 CPU.

So the next time you look at a router, take a few moments to see how utilized the entire CPU architecture is instead of just one CPU.

#packetsdownrange #mikrotik