The following is a list of some of the more popular Facebook groups for Wireless Internet Service Providers (WISPs), Fiber Service Providers (FISPs), and other related groups. This list is not complete by any means. If you find this list helpful please consider donating a few dollars via Paypal or joining my Patreon. If you are not listed here please consider donating and I will get you added. Last Updated: 13 May 2021
This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
Over the years I have been able to narrow the most common reasons a service provider goes down or has an outage. This is, by no means, an extensive list. Let’s jump in.
Physical layer outages are the easiest and where you should always start. If you have had any kind of formal training you have ran across the OSI model. Fiber cuts, equipment failure, and power are all physical layer issues. I have seen too many engineers spend time looking at configs when they should see if the port is up or the device is on.
DNS is what makes the transition from the man world to the machine world (queue matrix movie music). Without DNS we would not be able to translate www.j2sw.com into an IP address the we-servers and routers understand. DNS resolution problems are what you are checking when you do something like:
PING j2sw.com (126.96.36.199): 56 data bytes
64 bytes from 188.8.131.52: icmp_seq=0 ttl=52 time=33.243 ms
64 bytes from 184.108.40.206: icmp_seq=1 ttl=52 time=32.445 ms
--- j2sw.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 32.445/32.844/33.243/0.399 ms
Software bugs typically are always a reproducible thing. The ability to reproduce these bugs is the challenge. Sometimes a memory leak happens on a certain day. Sometimes five different criteria have to be met for the bug to happen.
When two or more routers talk to each other they talk best when they are on the same software version. A later version may fix an earlier bug. Code may change enough between version numbers that certain calls and processes are speaking slightly differently. This can cause incompatibilities between software versions.
“Fat fingering” is what we typically call this. A 3 was typed instead of a 2. This is why good version control and backups with differential are a good thing. Things such as cables getting bumped because they were not secured properly are also an issue.
What can we do to mitigate these issues?
1.Have good documentation. Know what is plugged in where what it looks like and as much detail as possible. You want your documentation to stand on its own. A person should be able to pick it up and follow it without calling someone.
2.Proactive monitoring. Knowing problems before customers call is a huge deal. Also, being able to identify trends over time is a good way to troubleshoot issues. Monitoring systems also allow you to narrow down the problem right away.
3.When it comes to networking know the OSI model and start from the bottom and work your way up.
Books can and are written about troubleshooting, This has just been a few of the common things I have seen.
Another water tower install earlier this year for a client. To critique our own work the cables were moved under the mounts after these pictures were taken. this was done to minimize damage to the cables.
The following are leasing companies that I have worked with on securing vertical real-estate over the years. This is not a total list of tower companies. If you provide co-location services to Wireless ISPs and want to be included please reach out to me. Donations motivate me to update these lists. Ones with a Star next to them are WISP friendly from our dealings.
The following is an extensive list of distributors who sell products related to the Wireless Internet Service Provider (WISP) space. This not a total list, but an extensive list. If you are not on this list or want to add your own description then donations are always welcome. It takes time to make these lists and there is nothing more motivating than some Paypal donations (https://paypal.me/j2sw).
ISP Supplies Texas-based distributor carrying a big number of product lines such as Cambium, Mikrotik, Airspan, and many others
I have been working on this list for a while. The following are vendors, manufacturers, and various companies I have dealt with in my career as an ISP owner and consultant. This is not a complete list by any means. These are companies I have dealt with personally and/or are sponsors of this site. Companies with the are ones that support this blog and I personally recommend. I don’t recommend them just because they support this blog, but because they provide a good product or service. If you would like to be included on this list please contact me as I am working on more detailed lists per category. This is a starting point for those looking to narrow down some focus of their research.
Distributors ISP Supplies Texas-based distributor carrying a big number of product lines such as Cambium, Mikrotik, Airspan, and many others
Baltic Networks Chicagoland based distributor carrying product lines such as Mikrotik, Cambium, and others.
CTIconnect Distributor of fixed wireless and telecommunications infrastructure for Internet Service Providers (ISP’s), Cable Operators, Telephone Companies
Most service providers have been the victim of a Denial of Service (DoS) attack at one point or another. Sometimes you may not realize you are under an attack. A few months ago, I posted a simple screenshot at https://blog.j2sw.com/networking/anatomy-of-a-ddos/ of what an active DDoS looks like.
Types of Attacks In order to know what to look for you have to understand the four basic types of attacks. I will outline this and talk about how modern attacks are affecting Internet Service Providers (ISPs). In my next article, we will talk about identifying these types of attacks and some mitigation techniques you can employ.
Throw everything at you attack aka Buffer overflow This type of attack is throwing enormous amounts of traffic at you to fill up your switch and router buffers, causing the device to exceed its capabilities. Your devices become crushed by an overwhelming volume of data throw at them. This attack isn’t always sheer bandwidth. Sometimes it is tens of thousands of remote connections.
Attacking vulnerable protocols Attackers go after exposed services like ICMP to do amplification attacks. Fragmented packets, which keep the router tied up are also a common method of attacking a host.
Application attacks These are the ones most consumers hear about. Vulnerabilities in operating systems, applications, and packages are exploited and used in attacks.
Hacks The fourth kind is not lumped in with Application attacks, but I wanted to separate it for a few reasons. The first reason is that someone compromising a system is not always sophisticated. If a bad actor guessed the password on your router and erased the configuration, they have performed a Denial of Service against you. If you don’t keep your software up-to-date and someone exploits a backdoor and “hacks” your system, they have performed of DoS attack.
Modern Attacks against networks Modern DoS attacks are always evolving. As network administrators find ways to mitigate these attacks, the bad actors find ways to tweak them and get around mitigation techniques employed by providers. Most of the exploits above involve sheer volumes of traffic or connections being directed at a host to take it offline. This attack is especially detrimental for service providers because it takes your customers offline if the attack is significant enough.
One of the most common techniques these days is the Distributed Denial of Service attack (DDoS). These are usually botnets involving thousands of compromised machines or devices acting against a host(s). These can be anywhere in the world. They could even be users inside your network with compromised machines or other devices. Distributed attacks are hard to mitigate because they can be legitimate traffic pointed at a web-server as an example. The traffic is not malicious from a technical perspective. You have thousands and thousands of machines sending legitimate requests to a web-server or other host on your network. This traffic looks legitimate but is overwhelming for your hardware and Internet pipe.
So what does a DDoS look like and what are your options when it comes to Denial of Service Attacks? In my next article in this series, I will talk about some best practices you can do so you are not as vulnerable to these types of attacks.
I am starting to see the following scenario more and more as IPv4 space is hard to get, but isn’t.
With ARIN it is still possible to get an IPv4 allotment. Many smaller ISPs qualify for a /24 and can get one if they wait long enough on the ARIN waiting list. a /24 of IPv4 space is the smallest block that 99% of the Internet allows to be advertised on the Capital I Internet. There are filter rules in place that drop smaller prefixes because that is the agreed upon norm.
So what happens if you are an ISP and you have a shiny new /24 but you have two networks which are not connected. Let’s look at our scenario.
The above network have no connectivity between the two of them on the internal side. These could be half way across the world or next door. If they were half way across the world it would make sense to try and get another /24. Maybe they are either side of a big mountain or one is down in a valley and there is no way to get a decent link between the two networks.
So what is a way you can use this /24 and still be able to assign IP addresses to both sides of the network? One way is to use a tunnel between your two edge routers.
Without the tunnel the scenario is traffic could come into network1, but if the IP is assigned on network 2 it will come back as unreachable. BGP is all about networks finding the shortest path to other networks. You don’t have much control over how networks find your public IP space if you have two providers advertising the same information. Some of the Internet will come in Network2 and some will come in Network1.
By running a tunnel between the two you can now subnet out that /24 into two eqal /25s and assign one /25 Network1 and one /25 to Network2 or however you want to. You can make the tunnel a GRE, EOIP, or other tunnel type. If I am using Mikrotik I prefer to use EOIP. If it’s another vendor I tend to use GRE.
Once the tunnel is established you can use static routing, OSPF, or your favorite IGP (interior Gateway Protocol) to “tell” one side about the routes on the other side. Let’s look at a fictional use.
In the above example our fictional ISP has an IPv4 block of 220.127.116.11/24. They have two networks separated by a tall mountain range in the center. It’s too cost prohibitive to run fiber or a wireless backhaul between the two networks so they have two different upstream providers. The ISP is advertising this /24 via BGP to Upstream1 from the Network 1 router. Network 2 router is also advertising the same /24 via BGP to Upstream 2.
We now create a Tunnel between the Mikrotiks. As mentioned before this can be EOIP, GRE, etc. We won’t go into the details of the tunnel but let’s assume the ISP is using Mikrotik. We create an EOIP tunnel (tons of tutorials out there) between Network 1 router and Network 2 router. Once this is established we will use 172.16.200.0/30 as our “Glue” on our tunnel interfaces at each side. Network 1 router gets 172.16.200.1/30. Network 2 router gets 172.16.200.2/30
To keep it simple we have a static route statement on the Network 1 Mikrotik router that looks like this:
This statement routes any traffic that comes in for 18.104.22.168/25 via ISP 1 to network1 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.
Conversely, we have a similar statement in the Network 2 Mikrotik router
This statement routes any traffic that comes in for 22.214.171.124/25 via ISP 2 to network2 across the tunnel to the Network 2 router. The Network 2 router then send it to the destination inside that side of the network.
It’s as simple as that. You can apply this to any other vendor such as Cisco, Juniper, PFSense, etc. You also do not have to split the network into even /25’s like I did. You can choose to have os of the ips available on one side and route a /29 or something to the other side.
The major drawback of this scenario is you will takef a speed hit because if the traffic comes in one side and has to route across the tunnel it will have to go back out to the public internet and over to the other ISP.