Interesting topic on discontinued gear

So an interesting topic came up on Facebook tonight that got me to thinking. As WISPs grow and evolve, what are your thoughts on hoarding gear you have been using for years when it becomes discontinued? We will examine some ideas as to why this isn’t necessarily all a technical problem. It’s also a philosophical thing with the WISP owner/management.

First off let us examine the whys you would hoard equipment. One big reason is that you have a significant investment in the gear you are using.  This gear has been proven to work, and you have deployed large amounts of it. As a company grows, the ability to introduce new gear into things facing the customer becomes a slower process. To use the analogy, the larger the company grows, the slower the ship turns.

Another reason is the amount of capital needed to migrate to new gear.  Many times when a product line gets discontinued, there is no clear replacement for it. The Facebook post which brought up this post involved the Mikrotik NetMetal 9s.  These are now discontinued by Mikrotik and have no replacement.  If a WISP were to migrate to something else there would be a significant cost in new access points, but more costly, would-be customer CPE. “But just put up the new gear alongside the old and migrate customers over,” you say. This brings us to the next point.

Frequency plays a big role in any migration path. In a perfect world, everyone has open channels and there is no interference. However, that is hardly the case in many scenarios.  This scenario is especially true of 900mhz.  You only have 902-928 MHz to deal with in the US FCC realm.  At 20 MHz wide this is only one non-overlapping channel.   If you put up another access point on 900mhz on top of your existing you will be interfering with yourself. Besides, the frequency may be the reason you are able to reach customers.

Finally, the pros of hoarding equipment are the soft costs of upgrading. Training, engineering, customer service, and possible re-work of some installs can add to the overall cost.  Anyone who has had to change the pins on a reverse polarity Subscriber Module knows the pain I am talking about.

The Cons

The biggest trap I see operators fall into is they horde equipment and then forget about i.  They have spares on the shelves, and enough to service customers. They fool themselves into a false sense of security and kind of wait for something to fall into their laps.  Then, it seems all of a sudden, something happens, and they are scrambling for a solution.  Sometimes this is a software update current equipment gets, but the older stuff does not. This could be some critical security vulnerability or new code to interface with a new system.  Either way, this equipment is stranded on a software island.

Next up is hardware failure.  As equipment gets old it, is more prone to failure.  A WISP may find their reserves depleted after a weekend of storms or bad luck. What may have been plentiful supplies a month ago is now an issue.

Lastly, the performance of the equipment is a big issue.  In today’s bandwidth-hungry consumer ISP radios are needing to perform better and deliver more bandwidth to the customer. Sometimes a manufacturer discontinues a product because they see the limitations of the band or the equipment. Sometimes the manufacturer sees operators are moving on to other ways of doing things. This could be newer frequencies or data algorithms. Usually, it boils down to the equipment was too expensive to make or wasn’t selling well enough.

So whats a WISP to do?

The number one thing a WISP needs to do is not fall into a rut of doing the same old same old for too long when it comes to equipment.  What worked five years ago, may work okay today, but will it work two years from now? Always have a strategy to dump your equipment if need be for something better.  Whether that strategy makes business sense is a different question. Sometimes the approach is to have money in the bank for when the right equipment comes along. Until then, it’s business as usual. Don’t let yourself keep saying you will figure it out tomorrow.

I believe that WISPs should have three lines of thinking.

  1. What am I doing in the immediate future to run my business?
  2. What am I doing in the next 18 months to keep my business competitive?
  3. What am I doing in the next 24-36 months to grow and keep up with customer demand?

If you have strategies for each of these then hoarding equipment is no big deal.  You have plans in place. Just don’t let yourself fall into a false sense of security. Always be learning about new rules, technologies, equipment, and methods.  As your business grows you can delegate this to others, so you don’t have to be in the thick of it and can concentrate on your business.  If you are that “techie” who is doing all of this, keep an open mind.  Don’t be the typical I.T. guy stuck in your ways. None of this is saying hoarding discontinued gear is wrong, just have a strategy.

#packetsdownrange

 

WISPs and Tower assets – LLC or what?

As a Wireless Internet Service Provider (WISP) grows, they tend to acquire some tower assets of their own.  Sometimes these are ones they build and sometimes these are towers they acquire from various sources. What is a WISP to do with these assets and, how should they treat them? In this article, I give you some options to consider. *DISCLAIMER* This article is not to be construed or viewed as actual legal advice.

One of the first business questions is how will owning a tower affect my current business? Will it cause your insurance rates to go up? Do I have liability concerns, especially if the tower is required to have lights according to FAA rules? Will I rent out space to other entities? All of these are valid questions and can affect how you structure these assets.

One of the everyday things a WISP can do when they build or acquire assets is to form an LLC for the tower side of things.  The question is do you form an LLC per tower or an overall LLC for your tower assets.  Let’s look at what forming an LLC will help you with before we get into the above question.

If you are a WISP, forming an LLC for your tower assets will most likely save your WISP operations from insurance penalties.  You may not have to carry extra business insurance or even have to switch carriers because your current insurer does not cover towers. Sure, you have to carry insurance for the LLC, but it can be a cut-down policy and causes things like workers compensation for the WISP side to rise. You are mainly concerned about the tower causing damage or damage to the tower.

Secondly, by having an LLC own and manage your assets, you have put yourself in a better position if you decide to sell your WISP operations.  On paper, your WISP is renting space and whatever from the Tower LLC.  If you have other forms of income from the tower you can decide whether to sell the tower or keep it. If the towers are separated from the WISP operations a sale of either side is much cleaner come sale time.  This is not to say you can’t sell the tower assets with the WISP operations, but it gives you more flexibility.

The last question is, should you form an LLC for each tower.  Some operators like this approach for several reasons.  The first is it allows one tower issue not to affect the other LLCs.  Should you come into legal problems with a tower, a properly set up and documented LLC will shield the members from potential personal liability.  With the ease of online LLC formation, it is easy enough to form an LLC for each tower asset. As mentioned earlier the separate LLCs give you more flexibility. Should you own some carrier-grade assets you have the potential to sell these to a larger tower aggregator without affecting your WISP operations. This is one way several of my clients have infused cash into their operations.

The downside is you now have more paperwork and filings to keep track of.  You have to keep up on LLC renewals, filings, and other paperwork. If you have processes or teams in place, this may not be an issue. Legally, you need to make sure you are not mixing the LLCs, which could cause you to lose the protection of the company.  It just depends on how savvy you are or if you have proper legal representation which is affordable.

At the very least I think it is very advantageous for the WISP to move any tower assets into at least one LLC. You are doing your business a favor.

Water tower install with mounting frame

We recently headed up a job for a client of installing some RF elements horns, Cambium ePMP, and Baicells LTE for a client.  One of the gems of this job was the frame the client designed for the job.  We can’t take credit for this. We just think it’s cool. Some of these pictures were taken during construction, thus post clean-up.

The frame is truly an example of how WISPs are stepping up their installs to become more standardized and carrier-grade. It costs some money but is worth it in the end.

 

Wisps getting it done

Several times a year, Paul Conlin barrels up a steep, wooded hillside in his mud-green Rhino SUV to the top of Rattlesnake Mountain above Hume, to perform maintenance on his equipment, which puts homes scattered in the valley below on the internet superhighway.

https://www.fauquier.com/news/ingenuity-and-an-atv-brings-broadband-to-parts-of-fauquier/article_514563ca-aece-11e9-9698-53bbcec1f408.html

j2 Podcast:5g hype, more bandwidth, preseem

Hi this is Justin, it’s Tuesday, July 30th, 2019 and this is the ISP news you need to know.  It’s been quite hot in my office so I haven’t been doing many recordings due to fans and such making it noisy in here.  here are some of the things you need to know.

Think 5G is all hype?
The 5G providers are touting how 5G will bridge the digital divide and folks are paying attention.
https://www.digitaltrends.com/web/31-percent-us-households-no-broadband-internet/

Another reason your customers need more bandwidth
Google play store is now selling some Disney movies in 4k.
https://9to5google.com/2019/07/25/google-play-disney-4k/

FCC asks for more c-Band input
The FCC seeks more comments on C-band proposals for flexible use of the 3.7-4.2 GHz Band. Comments on all the further studies are due August 7 to Docket 18-122.   
https://insidetowers.com/cell-tower-news-fcc-asks-for-more-input-on-proposals-to-repurpose-c-band/

Preseem Netflix Video
The folks over at Preseem have a pretty informative video on Netflix.

Common Question: Wisps, Public IPs, and assigning

So a question I get on a fairly regular basis is how do I assign IPs to my customers out of this /29 block my ISP gave me. As many of you know, IPV4 public addresses, commonly known as globally routable IPV4 addresses, are in short supply and obtaining more is usually very expensive. You have three options to obtain IP addresses.

Option #1 is getting them from whoever is supplying your bandwidth.  These IPs are known as Provider Assigned IP Space or PI space for short.  Typically these IP addresses are leased to you for a price or just allocated to you because you are a customer. The key here they are tied to the provider.  If you leave for someone else, your IPs are typically not portable.

Option#2 is going to the open market to buy or lease IP addresses.  There are several companies out there which sell or rent IP addresses. For business continuity buying your addresses is the best way to go.  As of this writing, IPV4 addresses in the ARIN region at going for $17-21 an Ip address.  This pricing means the smallest block you can get costs you around $5,600.

Option #3 is asking ARIN or whatever Regional Internet Registry (RIR) you fall under and getting put on a waiting list.  This wait can take quite some time and you may be waiting for a very long time.

Most startup ISPs get a small block of addresses from their provider.  The block can be a /29 ,or sometimes they can get a /28 or larger. The upstream ISP routes the block(s) to the startup. Here is where the problem comes up.  If an ISP gets a /29 this means they have 1 block of 6 IPs to use or two blocks of two IPs they can use in the form of two /30’s.  For those of you not familiar with subnetting, each time you “break down” a block of IP addresses you always lose 2 to the network and broadcast addresses.

if you want to learn about subnetting the following Youtube Videos will be helpful. Be warned there is a lot of numbers and math involved, but it’s necessary to learn about subnetting and proper IP planning.

Imagine an ISP has two customers on different towers that want a public IP address.  These addresses are for cameras, VPNs, or any number of uses. In order to assign IPs to each of those customers out of your /29 you have to either break it apart into two /30’s and route that to the appropriate router of yours the customer is attached to.  With small blocks assigned to you, this becomes very inefficient.  Even with large blocks, it can be inefficient. The below diagram shows how we can properly subnet this out given the limited IP block you have.

subnet example

We broke out 2.2.2.0/29 into the following subnets.
2.2.2.0/30
2.2.2.4/30

We still have the same number of IP addresses we had before, but due to the way subnetting works we lost a total of 4 addresses to the network and broadcast addresses.  Think of these as the beginning and end of the subnet.  We can’t use them to assign to devices, but we need them as part of the subnet.  We can note these as the following

2.2.2.0/30 – Network Address
2.2.2.1/30 – FIrst assignable address
2.2.2.2/30 – Second Assignable address
2.2.2.3/30 – Broadcast address

The next subnet would be 4,5,6,7 in the same order.  The rules of subnetting say we can not assign the network and broadcast addresses to devices or else we start breaking things.

Now if both of these customers were on the same tower, behind the same ISP router we would not necessarily have to subnetted the /29 out. We could have routed the /29 to the router and customer 1 has the .2 IP address and customer 2 has the .3 IP address.  the subnet would have IPs 4-6 available, but only for customers on the same tower.

The other way to save on public IP addresses is to do a 1:1 nat.  Natting involves doing port forwarding from the public IP to a private IP address at the customer location. If you want to learn about 1:1 Nat here are some links:
https://ciscoskills.net/2011/01/21/configuring-nat-one-to-one-mapping/
https://wiki.mikrotik.com/wiki/How_to_link_Public_addresses_to_Local_ones

So what is the real problem? If you have a /29 you can only break that down twice into the two /30s for customers going through different routers. Your network is routed instead of bridged right?

So how does the startup WISP overcome this without spending lots of money on IP addresses? You can do the 1:1 nat for each customer as explained above.  In my experience, this works 98% of the time. It is also something most I.T. folks are not familiar with.  I have had to spend hours on the phone with customers over the years explaining how a 1:1 nat works and they do have a public address to connect to, even though they have a private IP address on their router. Customers who do not get the 1:1 nat concept seem to be very leary of it and often blame the 1:1 nat when things go wrong.  Now if you are an ISP that provides a managed router, the customer does not see this layer.

Your other option is an MPLS network with VPLS tunnels.  MPLS with VPLS allows you to bridge your sites together to share an IP address pool.  However,  bridging is bad is what you hear.  MPLS has things like split-horizon built into the protocol to make bridging things together not so bad of a thing.

Another option is PPPoE.  PPPoE creates a tunnel utilizing a /32 (one ip address).  Many providers are using a shared pool to hand out IP addresses to customers who are given Public Ip addresses.

But my provider also gave me a /30 that is between my router and their’s. Can I use this?
The short answer is this is the “Glue” between you and the upstream provider.  The only real use for this is to Nat traffic out the Ip assigned to your side.  Doing this is not scalable though.

I have briefly touched on things like PPPoE, MPLS, and other things in this article.  These are such large topics there was no way this single post could do them justice.  I wanted to present the problem, a solution, and other methods to research to solve it. I will go into these in later blog posts.