Common ISP outage causes

Over the years I have been able to narrow the most common reasons a service provider goes down or has an outage. This is, by no means, an extensive list.   Let’s jump in.

Layer1 outages
Physical layer outages are the easiest and where you should always start. If you have had any kind of formal training you have ran across the OSI model.  Fiber cuts, equipment failure, and power are all physical layer issues.  I have seen too many engineers spend time looking at configs when they should see if the port is up or the device is on.

DNS related
DNS is what makes the transition from the man world to the machine world (queue matrix movie music). Without DNS we would not be able to translate www.j2sw.com into an IP address the we-servers and routers understand. DNS resolution problems are what you are checking when you do something like:

PING j2sw.com (199.168.131.29): 56 data bytes
64 bytes from 199.168.131.29: icmp_seq=0 ttl=52 time=33.243 ms
64 bytes from 199.168.131.29: icmp_seq=1 ttl=52 time=32.445 ms
--- j2sw.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 32.445/32.844/33.243/0.399 ms

Software bugs
Software bugs typically are always a reproducible thing.  The ability to reproduce these bugs is the challenge.  Sometimes a memory leak happens on a certain day.  Sometimes five different criteria have to be met for the bug to happen.

Version mismatches
When two or more routers talk to each other they talk best when they are on the same software version. A later version may fix an earlier bug.  Code may change enough between version numbers that certain calls and processes are speaking slightly differently.  This can cause incompatibilities between software versions.

Human mistakes
“Fat fingering” is what we typically call this. A 3 was typed instead of a 2. This is why good version control and backups with differential are a good thing. Things such as cables getting bumped because they were not secured properly are also an issue.

What can we do to mitigate these issues?
1.Have good documentation.  Know what is plugged in where what it looks like and as much detail as possible.  You want your documentation to stand on its own. A person should be able to pick it up and follow it without calling someone.
2.Proactive monitoring.  Knowing problems before customers call is a huge deal. Also, being able to identify trends over time is a good way to troubleshoot issues.  Monitoring systems also allow you to narrow down the problem right away.
3.When it comes to networking know the OSI model and start from the bottom and work your way up.

Books can and are written about troubleshooting,  This has just been a few of the common things I have seen.

Red-team Hacking tool

I wanted to do something with a Ras Pi Zero that would incorporate my love for red-teaming/offensive security, and I wanted it to take as many paths of least resistance as possible, and it couldn’t be super expensive ($50-ish USD). Now that I had the basic (albeit arbitrary) parameters in place, all I had to do was come up with the problem to solve. Think, think, think…

https://blog.itpro.tv/raspberry-spy-how-daniel-built-his-own-red-team-tool/

WISP Tower leasing Resources

The following are leasing companies that I have worked with on securing vertical real-estate over the years. This is not a total list of tower companies. If you provide co-location services to Wireless ISPs and want to be included please reach out to me. Donations motivate me to update these lists. Ones with a Star next to them are WISP friendly from our dealings.


American Tower
https://www.americantower.com/

Clearview Tower
http://clearviewtower.squarespace.com/

Crown Castle
https://www.crowncastle.com/

Heartland Tower
http://www.heartlandtower.com/

Insite Wireless
https://insitewireless.com/

KGI
https://kgiwireless.com/

Melody Wireless
http://www.melodywireless.com/

MidAmerica Towers
https://midamericatowers.com/

Nexus Towers
https://nexustowers.com/

SBA
https://www.sbasite.com/

Subcarrier Communications
https://www.subcarrier.com/

Tillman Infrastructure
https://www.tillmaninfrastructure.com/

TowerCO
https://www.towerco.com/TowerSearch

Towersites.com
https://tower-sites.com/

Tower Ventures
https://towerventures.com/

Vertical Bridge
http://www.verticalbridge.com/

Wifi-enabled smart padlock

BoxLock is an internet-connected smart padlock that can be used for access control and securing any storage container that accepts a standard padlock. It’s simple to use BoxLock. With the mobile application on Android or iOS, BoxLock users can open the lock or share access by texting or e-mailing barcodes to friends and family for one-time use, multiple users or specific dates. Press the button on the top to scan a barcode. Each time a barcode is scanned your BoxLock connects to your 2.4GHz Wi-Fi to confirm whether or not it should open.This latest version of BoxLock incorporates security enhancements and improved battery life. BoxLock must be in range of a 2.4GHz WiFi network with an active internet connection for full functionality.

United States-based WISP distributors

The following is an extensive list of distributors who sell products related to the Wireless Internet Service Provider (WISP) space.  This not a total list, but an extensive list.  If you are not on this list or want to add your own description then donations are always welcome.  It takes time to make these lists and there is nothing more motivating than some Paypal donations (https://paypal.me/j2sw).

Last Updated: 10 January 2020

Justin’s List of xISP vendors and resources

I have been working on this list for a while. The following are vendors, manufacturers, and various companies I have dealt with in my career as an ISP owner and consultant. This is not a complete list by any means. These are companies I have dealt with personally and/or are sponsors of this site. Companies with the are ones that support this blog and I personally recommend.  I don’t recommend them just because they support this blog, but because they provide a good product or service. If you would like to be included on this list please contact me as I am working on more detailed lists per category.  This is a starting point for those looking to narrow down some focus of their research.

Distributors
ISP Supplies
Texas-based distributor carrying a big number of product lines such as Cambium, Mikrotik, Airspan, and many others

Baltic Networks
Chicagoland based distributor carrying product lines such as Mikrotik, Cambium, and others.

CTIconnect
Distributor of fixed wireless and telecommunications infrastructure for Internet Service Providers (ISP’s), Cable Operators, Telephone Companies

Double Radius


Billing
Azotel
Mature billing solution which can
manage all aspects of your ISP.

Sonar
Modern Billing software with many backend automation

VISP
Automation and control of your WISP customers

More Billing providers can be found at xISP billing platforms


Manufacturers
Baicells
LTE and CBRS based solutions

Cambium Networks
Manufacturer of fixed wireless products such as EMP, 450, and cnPilot wireless.

Mikrotik
Manufacturer of Mikrotik routers and RouterOS routing and switching products

Ubiquiti
Manufacturer of WISP and WIFI products. Product lines include AirFiber and Unifi.


Tower Related
TowerOne
Training and equipment to keep climbers and companies compliant and safe. Large selection of needed items such as Harnesses and rope related items for tower work.


Voice
Atheral
Unified communications with experts to help you migrate and stay compliant. Here is a link to a podcast I did with Ateral.

True IP Solutions
Unified communications solutions integrated
with access and camera solutions.


Training
Rick Frey
mikrotik training and certification as well
as consulting and integrations solutions

LinkTechs
Training on Mikrotik and distributor of related products

More info on training for the xISP 


Supporting Services
TowerCoverage
RF Mapping and Modeling for tower sites and customer pre-qualification

Wireless Mapping
Radio Mapping, two-way radio, mark study information, and Municipal broadband.

IntelPath
Microwave and Millimeter Wavechannel procurement.


Organizations, web-sites, and groups
WISPA
Trade Organization supporting Wireless Internet Service Providers=

WISP Talk on Facebook

Cambium Users group on Facebook


YouTube Channels 
TheBrothersWISP
Networking, ISP, and related topics

MSFixit


Did I forget you? Would you like to sponsor this blog and your name listed? Contact me for more information.

Quick and Dirty Baicells eNODEB Mikrotik Rules

If you have a Baicells eNodeB you wish to restrict access to these Mikrotik rules will help. There are some assumptions made. The following rules are meant to be a base for incorporating into your network.

/ip firewall filter
add action=drop chain=forward src-address=10.0.0.2 src-port=443 protocol=tcp \
   dst-address-list=!baicells_cloud
add action=drop chain=forward src-address=10.0.0.2 src-port=8082 protocol=\
   tcp dst-address-list=!baicells_cloud
add action=drop chain=forward src-address=10.0.0.2 src-port=48080 protocol=\
   tcp dst-address-list=!baicells_cloud
add action=drop chain=forward src-address=10.0.0.2 src-port=4500,500 \
   protocol=udp dst-address-list=!baicells_cloud
add action=drop chain=forward src-address=10.0.0.2 dst-port=80,443 \
   protocol=tcp dst-address-list=!WHITELIST


/ip firewall address-list
add address=baiomc.cloudapp.net list=baicells_cloud
add address=baicells-westepc-03.cloudapp.net list=baicells_cloud
add address=baicells-eastepc04.eastus.cloudapp.azure.com list=baicells_cloud
add address=1.2.3.4/24 list=baicells_cloud
add address=1.2.3.4/24 list=WHITELIST

10.0.0.2 is your eNodeB

The 1.2.3.4 above is your management Subnet.

You can tighten these rules up by combining them, or create a new chain. This is quick and easy and anyone can understand. What it does is allows the eNodeb to only communicate with the Baicells cloud and your management network. It also only allows you to access your eNodeB from your management network. These are not a complete ruleset but something to build upon.