Cambium QOE traffic screenshots

Recently I have been using the QoE solution from Cambium Networks on some networks. This software allows for the prioritization and shaping of traffic on a service provider’s network. We will go into the workings of this in some later posts. Here are some screenshots.

Yes, it is IPv6 aware

What services should your ISP be providing?

As Internet Service has evolved so have the service offerings of Internet Service Providers (ISPs). Not too long ago the majority of ISPs were providing USENET feeds. If you remember those you have been at this for a long time.

For those of you just getting started or wanting to do an evaluation on your network here are some thoughts on services you should or should not be providing.

Let’s dive into the short list

DNS – Must Have
You should be running your own Doman Name Service (DNS) resolvers. There are many benefits to you and your customers.

e-mail – Don’t worry about offering
The advent of free or low-cost e-mail services has seen a rapid decline in the need for ISPs to their customers. For ISPs who offer e-mail, it is a service that generates many trouble tickets. From SPAM complaints all the way down to them not receiving emails. Leave offering email to customers to the GMAIL folks of the world.

Web-Site Hosting – Optional
If you are a Managed Service provider (MSP) then web-hosting can be a lucrative business. With web-site hosting also comes e-mail hosting.

VOIP
Voice Over IP (VOIP) is a hot topic these days. Many grants mention the ability to provide voice services as a requirement. Outsource this to providers such as Atheral.

Thats it! There are many other services you could extend to customers. There are also services which are not public facing you might want to run. Radius is on example of services you would run internally. As a service provider you should be spending as much effort on delivering solid access to your customers. Other services can be found by the end users themselves from dedicated services.

Yet another case for IPv6 for service providers

So recently I have been posting about the Cambium Networks QOE box I have been testing. After having this run for about a week I figured I would share this tidbit about usage.

55.39% of my traffic is IPV6 traffic. Most of this would be streaming traffic to various folks like NetFlix and Amazon and gaming traffic. My household consists of three people. No kids.

Implement IPV6 now if you are a service provider and thank me later. https://blog.j2sw.com/networking/wisps-ipv6-is-the-answer-to-some-of-your-issues/

#packetsdownrange

WISPAPALOOZA 2021 – Network Security talk

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

WISPAPLOOZA 2021 – Network Monitoring Talk

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

Common ISP outage causes

Over the years I have been able to narrow the most common reasons a service provider goes down or has an outage. This is, by no means, an extensive list.   Let’s jump in.

Layer1 outages
Physical layer outages are the easiest and where you should always start. If you have had any kind of formal training you have ran across the OSI model.  Fiber cuts, equipment failure, and power are all physical layer issues.  I have seen too many engineers spend time looking at configs when they should see if the port is up or the device is on.

DNS related
DNS is what makes the transition from the man world to the machine world (queue matrix movie music). Without DNS we would not be able to translate www.j2sw.com into an IP address the we-servers and routers understand. DNS resolution problems are what you are checking when you do something like:

PING j2sw.com (199.168.131.29): 56 data bytes
64 bytes from 199.168.131.29: icmp_seq=0 ttl=52 time=33.243 ms
64 bytes from 199.168.131.29: icmp_seq=1 ttl=52 time=32.445 ms
--- j2sw.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 32.445/32.844/33.243/0.399 ms

Software bugs
Software bugs typically are always a reproducible thing.  The ability to reproduce these bugs is the challenge.  Sometimes a memory leak happens on a certain day.  Sometimes five different criteria have to be met for the bug to happen.

Version mismatches
When two or more routers talk to each other they talk best when they are on the same software version. A later version may fix an earlier bug.  Code may change enough between version numbers that certain calls and processes are speaking slightly differently.  This can cause incompatibilities between software versions.

Human mistakes
“Fat fingering” is what we typically call this. A 3 was typed instead of a 2. This is why good version control and backups with differential are a good thing. Things such as cables getting bumped because they were not secured properly are also an issue.

What can we do to mitigate these issues?
1.Have good documentation.  Know what is plugged in where what it looks like and as much detail as possible.  You want your documentation to stand on its own. A person should be able to pick it up and follow it without calling someone.
2.Proactive monitoring.  Knowing problems before customers call is a huge deal. Also, being able to identify trends over time is a good way to troubleshoot issues.  Monitoring systems also allow you to narrow down the problem right away.
3.When it comes to networking know the OSI model and start from the bottom and work your way up.

Books can and are written about troubleshooting,  This has just been a few of the common things I have seen.

Justin’s List of xISP vendors and resources

I have been working on this list for a while. The following are vendors, manufacturers, and various companies I have dealt with in my career as an ISP owner and consultant. This is not a complete list by any means. These are companies I have dealt with personally and/or are sponsors of this site. Companies with the are ones that support this blog and I personally recommend.  I don’t recommend them just because they support this blog, but because they provide a good product or service. If you would like to be included on this list please contact me as I am working on more detailed lists per category.  This is a starting point for those looking to narrow down some focus of their research.

Distributors
ISP Supplies
Texas-based distributor carrying a big number of product lines such as Cambium, Mikrotik, Airspan, and many others

Baltic Networks
Chicagoland based distributor carrying product lines such as Mikrotik, Cambium, and others.

CTIconnect
Distributor of fixed wireless and telecommunications infrastructure for Internet Service Providers (ISP’s), Cable Operators, Telephone Companies

Double Radius


Billing
Azotel
Mature billing solution which can
manage all aspects of your ISP.

Sonar
Modern Billing software with many backend automation

VISP
Automation and control of your WISP customers

More Billing providers can be found at xISP billing platforms


Manufacturers
Baicells
LTE and CBRS based solutions

Cambium Networks
Manufacturer of fixed wireless products such as EMP, 450, and cnPilot wireless.

Mikrotik
Manufacturer of Mikrotik routers and RouterOS routing and switching products

Ubiquiti
Manufacturer of WISP and WIFI products. Product lines include AirFiber and Unifi.


Tower Related
TowerOne
Training and equipment to keep climbers and companies compliant and safe. Large selection of needed items such as Harnesses and rope related items for tower work.


Voice
Atheral
Unified communications with experts to help you migrate and stay compliant. Here is a link to a podcast I did with Ateral.

True IP Solutions
Unified communications solutions integrated
with access and camera solutions.


Training
Rick Frey
mikrotik training and certification as well
as consulting and integrations solutions

LinkTechs
Training on Mikrotik and distributor of related products

More info on training for the xISP 


Supporting Services
TowerCoverage
RF Mapping and Modeling for tower sites and customer pre-qualification

Wireless Mapping
Radio Mapping, two-way radio, mark study information, and Municipal broadband.

IntelPath
Microwave and Millimeter Wavechannel procurement.


Organizations, web-sites, and groups
WISPA
Trade Organization supporting Wireless Internet Service Providers=

WISP Talk on Facebook

Cambium Users group on Facebook


YouTube Channels 
TheBrothersWISP
Networking, ISP, and related topics

MSFixit


Did I forget you? Would you like to sponsor this blog and your name listed? Contact me for more information.

Preseem now supports IPv6

https://docs.preseem.com/changes

Features

IPv6

Preseem now supports IPv6 for all use cases. This includes the ability to assign subscribers a prefix of arbitrary length.

IPv4 with Prefixes of Arbitrary Length

Previously Preseem modelled subnet assignments to customers as a number of /32 assignments. For example a subscriber who was assigned a /30 would result in four internal /32 mappings. Preseem now supports assigning any prefix length to a subscriber without expanding these into /32 entries internally.

Netflix, IPV6, and affects of queing

While trying to get my Playstation to download the latest “No Man’s Sky” download quicker I figured I would share a little torch action.  This is showing my wife’s Ipad talking to Netflix while she is watching a streaming TV show. Keep in mind this is just an Ipad, not some 4k TV.

Some things to note as you watch this (no sound).

1.Uncapped the connection bursts to 50-60+ megs.
2.The slower your que the connection the more time it spends downloading data.  At slower ques the bursts last longer.
3.If you are handing out IPv6 to customers you should be queing them as well.

Just something to quick and dirty to keep in mind.