Tag Archives: isp

Common ISP outage causes

Posted on by

Over the years I have been able to narrow the most common reasons a service provider goes down or has an outage. This is, by no means, an extensive list.   Let’s jump in.

Layer1 outages
Physical layer outages are the easiest and where you should always start. If you have had any kind of formal training you have ran across the OSI model.  Fiber cuts, equipment failure, and power are all physical layer issues.  I have seen too many engineers spend time looking at configs when they should see if the port is up or the device is on.

DNS related
DNS is what makes the transition from the man world to the machine world (queue matrix movie music). Without DNS we would not be able to translate www.j2sw.com into an IP address the we-servers and routers understand. DNS resolution problems are what you are checking when you do something like:

PING j2sw.com (199.168.131.29): 56 data bytes
64 bytes from 199.168.131.29: icmp_seq=0 ttl=52 time=33.243 ms
64 bytes from 199.168.131.29: icmp_seq=1 ttl=52 time=32.445 ms
--- j2sw.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 32.445/32.844/33.243/0.399 ms

Software bugs
Software bugs typically are always a reproducible thing.  The ability to reproduce these bugs is the challenge.  Sometimes a memory leak happens on a certain day.  Sometimes five different criteria have to be met for the bug to happen.

Version mismatches
When two or more routers talk to each other they talk best when they are on the same software version. A later version may fix an earlier bug.  Code may change enough between version numbers that certain calls and processes are speaking slightly differently.  This can cause incompatibilities between software versions.

Human mistakes
“Fat fingering” is what we typically call this. A 3 was typed instead of a 2. This is why good version control and backups with differential are a good thing. Things such as cables getting bumped because they were not secured properly are also an issue.

What can we do to mitigate these issues?
1.Have good documentation.  Know what is plugged in where what it looks like and as much detail as possible.  You want your documentation to stand on its own. A person should be able to pick it up and follow it without calling someone.
2.Proactive monitoring.  Knowing problems before customers call is a huge deal. Also, being able to identify trends over time is a good way to troubleshoot issues.  Monitoring systems also allow you to narrow down the problem right away.
3.When it comes to networking know the OSI model and start from the bottom and work your way up.

Books can and are written about troubleshooting,  This has just been a few of the common things I have seen.

Justin’s List of xISP vendors and resources

Posted on by

I have been working on this list for a while. The following are vendors, manufacturers, and various companies I have dealt with in my career as an ISP owner and consultant. This is not a complete list by any means. These are companies I have dealt with personally and/or are sponsors of this site. Companies with the are ones that support this blog and I personally recommend.  I don’t recommend them just because they support this blog, but because they provide a good product or service. If you would like to be included on this list please contact me as I am working on more detailed lists per category.  This is a starting point for those looking to narrow down some focus of their research.

Distributors
ISP Supplies
Texas-based distributor carrying a big number of product lines such as Cambium, Mikrotik, Airspan, and many others

Baltic Networks
Chicagoland based distributor carrying product lines such as Mikrotik, Cambium, and others.

CTIconnect
Distributor of fixed wireless and telecommunications infrastructure for Internet Service Providers (ISP’s), Cable Operators, Telephone Companies

Double Radius

Billing
Azotel
Mature billing solution which can
manage all aspects of your ISP.

Sonar
Modern Billing software with many backend automation

VISP
Automation and control of your WISP customers

More Billing providers can be found at xISP billing platforms

Manufacturers
Baicells
LTE and CBRS based solutions

Cambium Networks
Manufacturer of fixed wireless products such as EMP, 450, and cnPilot wireless.

Mikrotik
Manufacturer of Mikrotik routers and RouterOS routing and switching products

Ubiquiti
Manufacturer of WISP and WIFI products. Product lines include AirFiber and Unifi.

Tower Related
TowerOne
Training and equipment to keep climbers and companies compliant and safe. Large selection of needed items such as Harnesses and rope related items for tower work.

Voice
Atheral
Unified communications with experts to help you migrate and stay compliant. Here is a link to a podcast I did with Ateral.

True IP Solutions
Unified communications solutions integrated
with access and camera solutions.

Training
Rick Frey
mikrotik training and certification as well
as consulting and integrations solutions

LinkTechs
Training on Mikrotik and distributor of related products

More info on training for the xISP 

Supporting Services
TowerCoverage
RF Mapping and Modeling for tower sites and customer pre-qualification

Wireless Mapping
Radio Mapping, two-way radio, mark study information, and Municipal broadband.

IntelPath
Microwave and Millimeter Wavechannel procurement.

Organizations, web-sites, and groups
WISPA
Trade Organization supporting Wireless Internet Service Providers=

WISP Talk on Facebook

Cambium Users group on Facebook

YouTube Channels 
TheBrothersWISP
Networking, ISP, and related topics

MSFixit


Did I forget you? Would you like to sponsor this blog and your name listed? Contact me for more information.

Preseem now supports IPv6

Posted on by

https://docs.preseem.com/changes

Features

IPv6

Preseem now supports IPv6 for all use cases. This includes the ability to assign subscribers a prefix of arbitrary length.

IPv4 with Prefixes of Arbitrary Length

Previously Preseem modelled subnet assignments to customers as a number of /32 assignments. For example a subscriber who was assigned a /30 would result in four internal /32 mappings. Preseem now supports assigning any prefix length to a subscriber without expanding these into /32 entries internally.

Netflix, IPV6, and affects of queing

Posted on by

While trying to get my Playstation to download the latest “No Man’s Sky” download quicker I figured I would share a little torch action.  This is showing my wife’s Ipad talking to Netflix while she is watching a streaming TV show. Keep in mind this is just an Ipad, not some 4k TV.

Some things to note as you watch this (no sound).

1.Uncapped the connection bursts to 50-60+ megs.
2.The slower your que the connection the more time it spends downloading data.  At slower ques the bursts last longer.
3.If you are handing out IPv6 to customers you should be queing them as well.

Just something to quick and dirty to keep in mind.

Atheral recommended 499 fillng help

Posted on by

Our friends over at Atheral have some companies to help you with your 499 filings.

What is FCC Form 499‑A?
FCC form 499‑A must be filed by interstate or international telecommunications providers in the US to register for the Universal Service Fund and report their revenue. You’ll have the form 499‑A immediately inside your registered agent account with our DC agent information pre-populated on it immediately after signing up for our DC registered agent service.

Inteserra Consulting Group:  https://www.inteserra.com/tom-forte
Lerman Senter:  https://www.lermansenter.com/attorneys/stephen-e-coran/
Compliance Solutions: https://www.csilongwood.com/
Marashlian & Donahue, PLLC: https://commlawgroup.com/

Quick home VPN using Mikrotik and an existing router

Posted on by

I had a situation today where we had an office worker needing to work from home.  This user had a Housefull of devices and a router managed by the Fiber to the home provider. This user had devices attached to the wifi on the provider router and such.  Normally I would want to replace this router, but it would be an undertaking.

For this setup, we used a Mikrotik MAP lite.
https://www.ispsupplies.com/MikroTik-RBmAPL-2nD

My quick solution was to have the user install the Mikrotik mAP as an ethernet device off of the provider’s router.  We then established a VPN tunnel from this device to the ISP’s network they work for.

 

We then added routes in the Mikrotik to the 3 networks they needed to access across the L2tp tunnel.  This user runs the Dude and Winbox. Once the tunnel was established we had two issues to overcome.

1. You have to add a nat rule in order for traffic behind the Mikrotik to reach the devices on the other side of the tunnel.  I simply added a nat rule that looks like this:

add action=masquerade chain=srcnat out-interface=all-ppp src-address=\
192.168.88.0/24

We could have done this in a few different ways, but remember this was a quick setup.

2. I needed the laptop they were working on the be able to route the three prefixes to the Mikrotik, thus going out the VPN.  In our setup, the laptop only has 2 default gateways.  It does not know any other routing info.

I created a bash script with the following in it. In short, you add the text below into a notepad file and save it with the extension of .bat.

route ADD 10.2.0.0 MASK 255.255.0.0 192.168.88.1
route ADD 10.3.0.0 MASK 255.255.0.0 192.168.88.1
route ADD 10.4.0.0 MASK 255.255.0.0 192.168.88.1

If you need help on creating a bash script
https://www.howtogeek.com/263177/how-to-write-a-batch-script-on-windows/

Once I had the file, which I simply saved into the Dude folder on the desktop, I created a shortcut on the desktop.  You will want to right-click on the shortcut and do the following.

It is important to note you are only able to do this on a shortcut in Windows, not the actual file itself.  No idea why. The script is important because this user brings the laptop back and forth.  I did not want to create persistent routes on the computer because the office network is different.  If you do not do persistent routes they will be after a reboot.  This way the user double clicks on the script shortcut when they login to the computer and before firing up the dude.

There are many other ways to accomplish this.  This was one of the quickest and less-impacting to the user and fewer things to support. One of the downsides to this setup is the user maintains two physical connections to two physical routers.  In this instance, the user could hardwire into the Mikrotik and maintain a wireless connection to the FIOS router.

If given more time you could have the laptop wired into the Mikoritk as your desk and have the wireless on the Mikrotik become a wireless client back to the FIOS router. This would make the setup a little more mobile.

#teleworker @packetsdownrange #j2 #vpn

Philosophies as a consultant Vendors, distributors

Posted on by

Over the years my views and philosophies on being a consultant have changed and are constantly evolving.  There are certain things that consultants can incorporate into their businesses in order to maintain a high level of service to clients.

Being Neutral
One of the things I have tried to do is be neutral when it comes to vendors and technology. While this is an admirable goal to have, you will find yourself gravitating toward technology you and your clients find useful and proven. It’s okay to be a certified consultant for a specific vendor. This brings up a whole new set of issues I will talk about later. There are two keys to take away from this. The first is to understand the underlying technology as a whole. If you think a particular product is superior enough for you to become certified in it, know why. Know how it is better than the competitors and where it lacks compared to the competitors.

The second key is to not be influenced by becoming a reseller/distributor for particular products. If you want to become a distributor, then focus on that. If you offer consulting services, become an integrator for that product. This way, you are not influenced by the latest promotion for a particular product and try to make it fit for a customer when something else might be better.

Vendor Expertise
As a consultant, you will probably find yourself working with specific products more than others. This is natural. I have found myself working with Cambium ePMP products more often than some others. I believe in the product, so I recommend it to my customers when it fits their situation. However, becoming an expert on a product line has pitfalls.

The first pitfall is you are an expert not paid by the vendor. If you are doing an excellent job on Social media and SEO your name should be popping up in google searches for that product. For example, if you do a search for “Cambium Consultant,” the first page that pops up is a page with my info on it. In a way, you are representing the brand without knowing it. This can lead to you answering questions about a product without any direct compensation for your time. I have always strived to answer questions on topics I am an expert on. There is a fine line between answering questions to a client who has not paid you money and one who has. Every potential contact is a potential client. You have to decide how to handle that grey area. This is an area I struggle with regularly. I am a Cambium ePMP expert and get many questions on this and that from folks who are not clients. I try and answer as many as I can, but at the end of the day, the paying clients do take priority.

Distributors and ordering
I mentioned earlier about me personally, not wanting to be a distributor or reseller. I don’t want to have to meet quotas and absolute minimums to keep stock of products. Some companies are better at this than I ever could be. Having a good relationship with a few good distributors is a good idea. Over the years, I have developed good relationships with several of these WISP distributors. There are some I shy away from due to they have competing services. There are a few vendors and distributors I have referred folks to, and the next thing I know they are offering them consulting services or saying, “I can fix that real quick for you”. They may not even realize they are hurting my business. These are distributors and vendors I personally stop referring business to. If it’s the right product, I will still include them in options for clients, but I make sure I keep on top of the relationship between myself, the vendor, and the client.

There are distributors out there who do very well offering consulting services. The question to ask is are they selling you products because the product makes them money or is it the right product for you? There is much room for either way.

Just some random 3am thoughts

Guest Article:Routers can catch viruses

Posted on by

Our friends over at TechWarn have their take on routers vulnerable to virus attacks

https://www.expressvpn.com/blog/can-my-router-catch-a-virus/

Big price differences between routers are often confusing to consumers as, unlike with personal computers, the quality difference is not always obvious. As routers are normally tied to a physical location, it is also rather difficult to test their reliability in different environments, unlike with highly mobile laptops or smartphones.

Routers often do not receive updates, or updates have to be manually downloaded and applied — a cumbersome process that is not an attractive option to many non-tech-savvy users.

Routers are desirable targets for attackers as they sit at a very sensitive spot on a network — right at the edge. They are a centralized point and connected to every single device in the network. Routers read all of the data that each device sends to the Internet, and if these connections are unencrypted, the router could easily inject malicious scripts and links.