xISP

Netbox Mikrotik Ansible Config generator

So, due to Covid, weather and everything else I am quite behind on blog updates and such. this is one that kinda fell through the cracks. I meant to get this out much sooner than now. My buddy Schylar Utley has a pretty cool projects for optimizing CPE deployments and such.

Check them out at https://github.com/MajesticFalcon

I have included an old video to give you an idea. I am sure things have changed since this video was created.

Mikrotik Connection tracking and CPU usage

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin's Patreon
Already a qualifying Patreon member? Refresh to access this content.

The problem with broadband projects in general

Before Covid I tried to attend as many meetings community leaders and towns had about bringing broadband to their communities. This is what you are supposed to in order to let the leaders know you, or in my case, my clients are there, right? Sometimes I would attend to provide my input as part of giving back to a community.

I have found some similarities in these meetings and workshops. Let’s go over them. If you are a community leader don’t let yourself fall into some of these.

The High-Level view
The high-level view starts out with noble intentions. The leaders want to get broadband to underserved areas. They have not bothered to dig deeper into seeing what is actually in the areas they want to cover. These folks may have called the ISP they have or someone their family has. they don’t actually know which providers service what areas. In their defense it’s not their job to. What they do with these meetings determines if progress is made or not. I have been in meetings where there have been four providers that service the area in question. The leaders say they must do more studies to see who is in the area. You literally have four sitting at your table who can tell you what they service. Take their information, take their maps and progress.

Bedazzled by the incumbent
Typically this person has XYZ Internet at their home and they love it. They love it so much they want it everywhere. This is great, but there are reasons that XYZ Internet is not everywhere. Otherwise, you would not be doing these meetings. Some of this is due to lack of money. Either XYZ Internet does not have enough or the return just is not there. This leader is one of the most hampering of all. I have been in many meetings where the small local company is putting their own money into investing in the community and this type of leader overlooks the small company. They even go as far to suggest the local company help XYZ become bigger in their own service area.

These leaders often invite their beau to these meetings to give their take on broadband in the area. Sometimes these companies are honest and straightforward. Sometimes they paint the picture they are the only ones who can solve the broadband issue.

The “let’s do a study” crowd
Studies are nice. They give you nice graphs, charts, and tons of fluff information about an area. It makes for good reading for those who like to learn about facts. These folks are probably the ones who know the stats of many sports figures, who lived in the prominent houses in the lcoal towns and other facts. They are willing to spend twenty thousand dollars on a piece of paper to get this information. In many instances, sitting down with the right group of people could tell you 90% of the information you need.

Unrealistic goals
Let’s face it, not everyone knows everything about the topic they are trying to address. Being able to provide gigabit to every home is a nice goal, but is hard to achieve. Not everyone needs or wants gigabit. In my county and the surrounding area, there are towns of only three or four houses. Unless lots of government money is involved fiber will not be coming to them anytime soon.

The academic
These are usually the most frustrating for the existing ISP. Terms like focus groups and thirty thousand foot view are thrown around. They are usually applying for some grants or RPF. They already have their goal and possibly the outcome in mind. They are not there to solve issues but to get the “bigger picture”. They may only know broadband from buzzwords. 5G and internet of everything are thrown around alot.

What folks do you see at these meetings? Let me know as we are working on a funny video.

Siklu Case study 80 GHZ Indianapolis Indiana

Some photos from a Siklu 80GHZ deployment in Downtown Indianapolis, Indiana. This was deployed by On-Ramp Indiana (https://www.ori.net). The problem being solved is moving video files around a network in order to get it to smart screens and projectors. This is a very urban area and wireless was pretty much the only option to get from building to building.

Siklu 80GHZ was on the shortlist due to the distances involved. Another consideration was the footprint of the equipment. The equipment had to be as low profile as possible.

Another needed aspect of this network was the ability to move traffic around at layer 2. Not all traffic is IP based in this type of network.

Equipment used
Ether Haul 1200FX
https://www.siklu.com/product/etherhaul-kilo-series/

Right above the observation windows, you can see the Siklu just to the right of the center corner

Some technical Details

Average traffic over the past 2 months

As you can see traffic is reasonably consistent in the 80-100 meg range. We needed a solution that did not slow down due to interference. A possible 10’s of thousands of visitors to this attraction in a weekend, reliability and performance were critical. When this was installed we did not know about COVID, but this is an attraction people can enjoy from their cars and social distancing. This use added to the visibility of this attraction, thus making the reliability even more crucial.

Articles about the finished product
https://www.wthr.com/article/news/local/monument-circle-get-new-light-show-time-holidays/531-ef1819ca-5f27-4886-9283-17e481c33f39

https://www.wthr.com/article/news/local/new-light-show-sound-system-entertain-monument-circle-visitors/531-576ce095-501c-41c6-913a-518a0cc05779

On-Ramp Indiana Contacts www.ori.net 317.774.2100

A Longitudinal View of Netflix

This came across the NANOG mail list.

A Longitudinal View of Netflix: Content Delivery over IPv6 and Content Cache Deployments

paper:   https://bit.ly/2toOGWP
slides:  https://bit.ly/2ZoEpap
video:   https://vimeo.com/437111302

Some highlights

Caches reachable within 6 IP hops and 20 ms
-IP path lengths shorter by 40–50%
-Latency lower by 64%
-Throughput higher by factor of three
-Latency benefits more pronounced over IPv4 compared with IPv6

Latency and IP path lengths similar between both address families
⇒ High IPv6 preference, however, slight drops during peak hours

Network troubleshooting tools

Recently, there was a thread on the NANOG list asking what were somne favorite network troubleshooting tools. I have taken many of these tools and created the following list.

http://ping.pe/
Simple pingport and dig commands

https://mtr.sh/
BGP Looking glass

https://perfops.net/mtr-from-world
Traceroute from various hosts on the net

http://www.traceroute6.net/
IPV6 tools (ping,traceroute,etc)

https://dnsviz.net/
Carious DNS tools

http://irrexplorer.nlnog.net/
Routing Registry object explorer

https://mxtoolbox.com/
DNS and Mail tools

WISPs: IPv6 is the answer to some of your issues

Many Wireless Internet Service Providers (WISPs), especially newer startups, struggle with nat issues and having enough public Ip addresses to go around. Invariably, you start running into double nat issues pretty quickly. Then you get the dreaded gamer call:

Many times they don’t know why they are even calling. They just know the magic box is saying this is bad. This is related to how many layers of nat between your edge and them. Many times you are natting at the edge, then you are natting at the customer router. If you have multiple customers behind the same nat at the edge this compounds it even more.

So what is the fix? Give the customer public addresses. But IPv4 is hard to get! I didn’t say IPV4 I said public addresses. IPv6 is a public address. When given the choice between v4 and v6 most modern streaming and gaming platforms will prefer v6. Xbox has supported a protocol called Teredo for a long time. You can learn all about Teredo in this PDF. Basically, it is a tunnel in which the Xbox speaks ipv6 over the tunnel. The ISP does not have to support v6, which does away with the above-mentioned nat issues.

Great! I don’t have to worry about IPv6, Microsoft has it taken care of for me. There are two problems with this statement. Problem number one. There are more companies out there than Microsoft. Sony Playstation Online, Apple gaming, and Steam are just a few. Second, you have overhead of tunnels. In the world of who can pull the joystick quicker, milliseconds count. You don’t want them wasted in tunnel overhead. Plus, v6 is beneficial for other service such as Netflix.

Any other service that runs into port issues behind nat can be solved with Ipv6, This can be voip, cameras, and other type services. This is providing the product or service supports v6 addresses.

So what is an ISP to do?
Awhile back I put together a resource guide for ISPs. You can find it at https://blog.j2sw.com/networking/ipv6/ipv6-planning-and-implementation-resources-for-the-xisp/

Internet Routing Registry Resources by j2sw

What is a routing registry?
From Wikipedia https://en.wikipedia.org/wiki/Internet_Routing_Registry
The Internet routing registry works by providing an interlinked hierarchy of objects designed to facilitate the organization of IP routing between organizations, and also to provide data in an appropriate format for automatic programming of routers. Network engineers from participating organizations are authorized to modify the Routing Policy Specification Language (RPSL) objects, in the registry, for their own networks. Then, any network engineer, or member of the public, is able to query the route registry for particular information of interest.

RFC2622 Routing Policy Specification Language (RPSL)

RFC2650 Using RPSL in Practice

RFC7682 Considerations for Internet Routing Registries (IRRs) and routing Policy Configuration

General IRR Information

http://www.irr.net/
Includes links to various registries, FAQs, and other info

https://www.gin.ntt.net/support-center/policies-procedures/routing-registry/ntt-route-registry-frequently-asked-questions/
NTT route registry FAQ

https://www.seattleix.net/irr-tutorial
Seattle Internet Exchange IRR Tutorial

https://www.manrs.org/resources/tutorials/irrs-rpki-peeringdb/

https://archive.nanog.org/meetings/nanog51/presentations/Sunday/NANOG51.Talk34.NANOG51%20IRR%20Tutorial.pdf
NANOG Routing registry tutorial

General How-Tos

https://fcix.net/whitepaper/2018/07/14/intro-to-irr-rpsl.html
A Quickstart Guide to Documenting Your Prefixes with IRR. This mainly uses the older ARIN e-mail templates.


Arin Specific

https://www.arin.net/resources/manage/irr/userguide/
Arin’s userguide for working with their IRR

https://www.arin.net/resources/manage/irr/irr-online-implementation
Notes on working with ARINs web-based


Other Regional Registries

African Network Coordination Centre (AFRNIC)
https://afrinic.net/internet-routing-registry

Asian-Pacific Network Coordination Centre (APNIC)
https://www.apnic.net/manage-ip/apnic-services/routing-registry/

American Registry for Internet Numbers (ARIN)
https://www.apnic.net/manage-ip/apnic-services/routing-registry/

Latin American and Caribbean Internet Addresses Registry (LACNIC)
https://www.lacnic.net/innovaportal/file/3512/1/internet-routing-registries.pdf

Reseaux IP Eauropeens Network Coordination Centre (RIPE NCC)
https://www.ripe.net/manage-ips-and-asns/db/support/managing-route-objects-in-the-irr

Tools

https://github.com/6connect/irrpt
A collection of tools which allow ISPs to easily track, manage, and utilize IPv4 and IPv6 BGP routing information stored in Internet Routing Registry (IRR) databases. Some of these tools include automated IRR data retrieval, update tracking via CVS, e-mail notifications, e-mail based notification for ISPs who still do human processing of routing information, and hooks for automatically deploying prefix-lists on routers.

https://www.radb.net/query
The RADB whois server provides information collected from all the registries that form part of the Internet Routing Registry. 

https://github.com/irrdnet/irrd
Internet Routing Registry daemon version 4 is an IRR database server, processing IRR objects in the RPSL format.