xISP

xISP related Facebook groups for WISPs and other ISPs

The following is a list of some of the more popular Facebook groups for Wireless Internet Service Providers (WISPs), Fiber Service Providers (FISPs), and other related groups. This list is not complete by any means. If you find this list helpful please consider donating a few dollars via Paypal or joining my Patreon. If you are not listed here please consider donating and I will get you added.
Last Updated: 13 May 2021

General Groups
WISP Talk
https://www.facebook.com/groups/wisptalk/

Everything WISP
https://www.facebook.com/groups/everythingwisp/

WISP Equipment
https://www.facebook.com/groups/wispequipment/

WISP Talk – Starlink & LEO
https://www.facebook.com/groups/392803248590263/

WISP Talk (Newbies and Startups)
https://www.facebook.com/groups/2167897793237945

WISPs turned FISPs – FTTH, FTTP, just Fiber stuff
https://www.facebook.com/groups/fispstuff/

WISP Installers
https://www.facebook.com/groups/570927479951649/


Organizations
WISPA
https://www.facebook.com/WirelessISPAssociation/


Vendor User Groups
Cambium Users (English Only)
https://www.facebook.com/groups/cambiumusers

Sonar Software User Group
https://www.facebook.com/groups/921593151304842

Baicells Unofficial English Users Group
https://www.facebook.com/groups/Baicells

Mimosa by Airspan Users Group
https://www.facebook.com/groups/1550178418577187


Services
Atheral
https://www.facebook.com/atheralrocks

VISP
https://www.facebook.com/visp.noc

Distributors

Equipment Manufacturers

Common ISP outage causes

Over the years I have been able to narrow the most common reasons a service provider goes down or has an outage. This is, by no means, an extensive list.   Let’s jump in.

Layer1 outages
Physical layer outages are the easiest and where you should always start. If you have had any kind of formal training you have ran across the OSI model.  Fiber cuts, equipment failure, and power are all physical layer issues.  I have seen too many engineers spend time looking at configs when they should see if the port is up or the device is on.

DNS related
DNS is what makes the transition from the man world to the machine world (queue matrix movie music). Without DNS we would not be able to translate www.j2sw.com into an IP address the we-servers and routers understand. DNS resolution problems are what you are checking when you do something like:

PING j2sw.com (199.168.131.29): 56 data bytes
64 bytes from 199.168.131.29: icmp_seq=0 ttl=52 time=33.243 ms
64 bytes from 199.168.131.29: icmp_seq=1 ttl=52 time=32.445 ms
--- j2sw.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 32.445/32.844/33.243/0.399 ms

Software bugs
Software bugs typically are always a reproducible thing.  The ability to reproduce these bugs is the challenge.  Sometimes a memory leak happens on a certain day.  Sometimes five different criteria have to be met for the bug to happen.

Version mismatches
When two or more routers talk to each other they talk best when they are on the same software version. A later version may fix an earlier bug.  Code may change enough between version numbers that certain calls and processes are speaking slightly differently.  This can cause incompatibilities between software versions.

Human mistakes
“Fat fingering” is what we typically call this. A 3 was typed instead of a 2. This is why good version control and backups with differential are a good thing. Things such as cables getting bumped because they were not secured properly are also an issue.

What can we do to mitigate these issues?
1.Have good documentation.  Know what is plugged in where what it looks like and as much detail as possible.  You want your documentation to stand on its own. A person should be able to pick it up and follow it without calling someone.
2.Proactive monitoring.  Knowing problems before customers call is a huge deal. Also, being able to identify trends over time is a good way to troubleshoot issues.  Monitoring systems also allow you to narrow down the problem right away.
3.When it comes to networking know the OSI model and start from the bottom and work your way up.

Books can and are written about troubleshooting,  This has just been a few of the common things I have seen.

WISP Tower leasing Resources

The following are leasing companies that I have worked with on securing vertical real-estate over the years. This is not a total list of tower companies. If you provide co-location services to Wireless ISPs and want to be included please reach out to me. Donations motivate me to update these lists. Ones with a Star next to them are WISP friendly from our dealings.


American Tower
https://www.americantower.com/

Clearview Tower
http://clearviewtower.squarespace.com/

Crown Castle
https://www.crowncastle.com/

Heartland Tower
http://www.heartlandtower.com/

Insite Wireless
https://insitewireless.com/

KGI
https://kgiwireless.com/

Melody Wireless
http://www.melodywireless.com/

MidAmerica Towers
https://midamericatowers.com/

Nexus Towers
https://nexustowers.com/

SBA
https://www.sbasite.com/

Subcarrier Communications
https://www.subcarrier.com/

Tillman Infrastructure
https://www.tillmaninfrastructure.com/

TowerCO
https://www.towerco.com/TowerSearch

Towersites.com
https://tower-sites.com/

Tower Ventures
https://towerventures.com/

Vertical Bridge
http://www.verticalbridge.com/

United States-based WISP distributors

The following is an extensive list of distributors who sell products related to the Wireless Internet Service Provider (WISP) space.  This not a total list, but an extensive list.  If you are not on this list or want to add your own description then donations are always welcome.  It takes time to make these lists and there is nothing more motivating than some Paypal donations (https://paypal.me/j2sw).

Last Updated: 10 January 2020

Justin’s List of xISP vendors and resources

I have been working on this list for a while. The following are vendors, manufacturers, and various companies I have dealt with in my career as an ISP owner and consultant. This is not a complete list by any means. These are companies I have dealt with personally and/or are sponsors of this site. Companies with the are ones that support this blog and I personally recommend.  I don’t recommend them just because they support this blog, but because they provide a good product or service. If you would like to be included on this list please contact me as I am working on more detailed lists per category.  This is a starting point for those looking to narrow down some focus of their research.

Distributors
ISP Supplies
Texas-based distributor carrying a big number of product lines such as Cambium, Mikrotik, Airspan, and many others

Baltic Networks
Chicagoland based distributor carrying product lines such as Mikrotik, Cambium, and others.

CTIconnect
Distributor of fixed wireless and telecommunications infrastructure for Internet Service Providers (ISP’s), Cable Operators, Telephone Companies

Double Radius


Billing
Azotel
Mature billing solution which can
manage all aspects of your ISP.

Sonar
Modern Billing software with many backend automation

VISP
Automation and control of your WISP customers

More Billing providers can be found at xISP billing platforms


Manufacturers
Baicells
LTE and CBRS based solutions

Cambium Networks
Manufacturer of fixed wireless products such as EMP, 450, and cnPilot wireless.

Mikrotik
Manufacturer of Mikrotik routers and RouterOS routing and switching products

Ubiquiti
Manufacturer of WISP and WIFI products. Product lines include AirFiber and Unifi.


Tower Related
TowerOne
Training and equipment to keep climbers and companies compliant and safe. Large selection of needed items such as Harnesses and rope related items for tower work.


Voice
Atheral
Unified communications with experts to help you migrate and stay compliant. Here is a link to a podcast I did with Ateral.

True IP Solutions
Unified communications solutions integrated
with access and camera solutions.


Training
Rick Frey
mikrotik training and certification as well
as consulting and integrations solutions

LinkTechs
Training on Mikrotik and distributor of related products

More info on training for the xISP 


Supporting Services
TowerCoverage
RF Mapping and Modeling for tower sites and customer pre-qualification

Wireless Mapping
Radio Mapping, two-way radio, mark study information, and Municipal broadband.

IntelPath
Microwave and Millimeter Wavechannel procurement.


Organizations, web-sites, and groups
WISPA
Trade Organization supporting Wireless Internet Service Providers=

WISP Talk on Facebook

Cambium Users group on Facebook


YouTube Channels 
TheBrothersWISP
Networking, ISP, and related topics

MSFixit


Did I forget you? Would you like to sponsor this blog and your name listed? Contact me for more information.

Denial of Service and the xISP Part 1

Most service providers have been the victim of a Denial of Service (DoS) attack at one point or another. Sometimes you may not realize you are under an attack. A few months ago, I posted a simple screenshot at https://blog.j2sw.com/networking/anatomy-of-a-ddos/ of what an active DDoS looks like.

Types of Attacks
In order to know what to look for you have to understand the four basic types of attacks. I will outline this and talk about how modern attacks are affecting Internet Service Providers (ISPs). In my next article, we will talk about identifying these types of attacks and some mitigation techniques you can employ.

Throw everything at you attack aka Buffer overflow
This type of attack is throwing enormous amounts of traffic at you to fill up your switch and router buffers, causing the device to exceed its capabilities. Your devices become crushed by an overwhelming volume of data throw at them. This attack isn’t always sheer bandwidth. Sometimes it is tens of thousands of remote connections.

Attacking vulnerable protocols
Attackers go after exposed services like ICMP to do amplification attacks. Fragmented packets, which keep the router tied up are also a common method of attacking a host.

Application attacks
These are the ones most consumers hear about. Vulnerabilities in operating systems, applications, and packages are exploited and used in attacks.

Hacks
The fourth kind is not lumped in with Application attacks, but I wanted to separate it for a few reasons. The first reason is that someone compromising a system is not always sophisticated. If a bad actor guessed the password on your router and erased the configuration, they have performed a Denial of Service against you. If you don’t keep your software up-to-date and someone exploits a backdoor and “hacks” your system, they have performed of DoS attack.

Modern Attacks against networks
Modern DoS attacks are always evolving. As network administrators find ways to mitigate these attacks, the bad actors find ways to tweak them and get around mitigation techniques employed by providers. Most of the exploits above involve sheer volumes of traffic or connections being directed at a host to take it offline. This attack is especially detrimental for service providers because it takes your customers offline if the attack is significant enough.

One of the most common techniques these days is the Distributed Denial of Service attack (DDoS). These are usually botnets involving thousands of compromised machines or devices acting against a host(s). These can be anywhere in the world. They could even be users inside your network with compromised machines or other devices. Distributed attacks are hard to mitigate because they can be legitimate traffic pointed at a web-server as an example. The traffic is not malicious from a technical perspective. You have thousands and thousands of machines sending legitimate requests to a web-server or other host on your network. This traffic looks legitimate but is overwhelming for your hardware and Internet pipe.

Image courtesy of https://www.imperva.com/blog/how-to-identify-a-mirai-style-ddos-attack/

So what does a DDoS look like and what are your options when it comes to Denial of Service Attacks? In my next article in this series, I will talk about some best practices you can do so you are not as vulnerable to these types of attacks.

BGP, a single /24 and two diverse non-connected exit points

I am starting to see the following scenario more and more as IPv4 space is hard to get, but isn’t.

With ARIN it is still possible to get an IPv4 allotment. Many smaller ISPs qualify for a /24 and can get one if they wait long enough on the ARIN waiting list. a /24 of IPv4 space is the smallest block that 99% of the Internet allows to be advertised on the Capital I Internet. There are filter rules in place that drop smaller prefixes because that is the agreed upon norm.

So what happens if you are an ISP and you have a shiny new /24 but you have two networks which are not connected. Let’s look at our scenario.

The above network have no connectivity between the two of them on the internal side. These could be half way across the world or next door. If they were half way across the world it would make sense to try and get another /24. Maybe they are either side of a big mountain or one is down in a valley and there is no way to get a decent link between the two networks.

So what is a way you can use this /24 and still be able to assign IP addresses to both sides of the network? One way is to use a tunnel between your two edge routers.

Without the tunnel the scenario is traffic could come into network1, but if the IP is assigned on network 2 it will come back as unreachable. BGP is all about networks finding the shortest path to other networks. You don’t have much control over how networks find your public IP space if you have two providers advertising the same information. Some of the Internet will come in Network2 and some will come in Network1.

By running a tunnel between the two you can now subnet out that /24 into two eqal /25s and assign one /25 Network1 and one /25 to Network2 or however you want to. You can make the tunnel a GRE, EOIP, or other tunnel type. If I am using Mikrotik I prefer to use EOIP. If it’s another vendor I tend to use GRE.

Once the tunnel is established you can use static routing, OSPF, or your favorite IGP (interior Gateway Protocol) to “tell” one side about the routes on the other side. Let’s look at a fictional use.

In the above example our fictional ISP has an IPv4 block of 1.2.3.0/24. They have two networks separated by a tall mountain range in the center. It’s too cost prohibitive to run fiber or a wireless backhaul between the two networks so they have two different upstream providers. The ISP is advertising this /24 via BGP to Upstream1 from the Network 1 router. Network 2 router is also advertising the same /24 via BGP to Upstream 2.

We now create a Tunnel between the Mikrotiks. As mentioned before this can be EOIP, GRE, etc. We won’t go into the details of the tunnel but let’s assume the ISP is using Mikrotik. We create an EOIP tunnel (tons of tutorials out there) between Network 1 router and Network 2 router. Once this is established we will use 172.16.200.0/30 as our “Glue” on our tunnel interfaces at each side. Network 1 router gets 172.16.200.1/30. Network 2 router gets 172.16.200.2/30

To keep it simple we have a static route statement on the Network 1 Mikrotik router that looks like this:

/ip route add dst-address=1.2.3.129/25 gateway=172.16.200.2

This statement routes any traffic that comes in for 1.2.3.128/25 via ISP 1 to network1 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

Conversely, we have a similar statement in the Network 2 Mikrotik router

/ip route add dst-address=1.2.3.0/25 gateway=172.16.200.1

This statement routes any traffic that comes in for 1.2.3.0/25 via ISP 2 to network2 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.

It’s as simple as that. You can apply this to any other vendor such as Cisco, Juniper, PFSense, etc. You also do not have to split the network into even /25’s like I did. You can choose to have os of the ips available on one side and route a /29 or something to the other side.

The major drawback of this scenario is you will takef a speed hit because if the traffic comes in one side and has to route across the tunnel it will have to go back out to the public internet and over to the other ISP.

#packetsdownrange