Follow me on Instagram for technology, and sometimes not technology, related photos.
Follow me on Instagram for technology, and sometimes not technology, related photos.
Let’s face it, if you are an owner, sysadmin, network admin, or whatever in a small to medium Internet Service Provider vacations can be a troublesome thing to accomplish. Getting away for a conference or training can be a huge deal as well. In this post, I want to help you layout some steps to help you have some rest and relaxation. Let’s start with why this vacation stuff can be a mystical thing. Then we will go into some strategies to implement so you can getaway.
Many small service providers, especially WISPs, are very entrepreneurial in nature. This means they are very driven, type-a personality folks. They pay attention to detail, and every detail counts. You have basically two categories for these folks. The first I like to call the boss. This is the person who has built the business and wants to make sure the well-oiled machine is running at 100% all the time. He or she is calling the office if they have not heard from anyone in a certain amount of time. They start getting nervous and want to make sure the place hasn’t caught fire and that’s why they are getting no calls.
The second person is the geek. The geek tends to treat their network like a pet. They care for the network it like it’s a living organism. They know all the ins and outs of every aspect of the network. They are so in-tune with the network they can walk into the server room and notice a slight difference in noise indicating a slowing fan or a server with ten disks spinning instead of eleven.
So, what are you to do if you want to take some time off? You need a plan. I see too many people at conferences in the hallways with laptops balanced on their knees trying to solve an issue. Sometimes this is a critical issue that warrants an emergency response. Sometimes its something that can wait until the evening or wait a few days. The thing is I see these folks over and over in the hallways at conferences. They are always outside on their phone or laptop. They are not getting the benefit of being there because they are still tied to the office like they never left
Step one – Admit you need help
Realize you need to getaway. This not only benefits you physically and mentally but benefits the company as a whole. It allows other employees the chance to step up and take leadership roles and exercise their problem-solving skills.
Due to various personality traits, it is sometimes hard to let go of the reigns. If you follow my steps you will actually be more in control because you will have systems and tools telling you what is going on with the network.
Step two – Have procedures
You need to have clear escalation procedures for issues that come up. Procedures for installs, server reboots, tower reboots, and anything that may happen need to be documented. I always imagine taking someone who has never seen any I.T. equipment and handing them a piece of paper on what to do in such and such a /scenario. This is where documentation comes into play as well. However, procedures are the steps to take. They are directions that anyone should be able to follow. If you have ever watched a movie where the control tower is trying to talk someone in a plane trying to land you know what I mean. They almost always have a shot of the control tower person breaking out this big book and starting to walk through procedures. You should have the same thing.
If you have an access point failure your procedure should involve how to identify which access point has failed, how to do basic troubleshooting and steps to try and remedy the situation. After these steps have done there should be an escalation procedure of what to do next.
If an access point has to be replaced your procedure should outline what model to grab from spare inventory, where the backup file is stored, how to restore the backup file, how to test the new unit. the more you have in procedures, the less time technicians should spend on the phone with you. They can walk through these procedures, send you a quick text or call if something needs clarification, and proceed with the procedure. Have others review your procedures. This can ensure you are clear and things make sense to someone who might be reading them for the first time.
Step three – Scheduling
If you are the type of person who has to be involved with every customer install for whatever reason (we will talk about reasons shortly) then do not schedule installs for the time you are away. If you must schedule installs try and schedule them for times you know you will be available or can make an easy time to be available.
Step four – Remote access
Setup remote access. IPsec VPNs or easy to setup. I did an article on having an mAP lite to carry around with you. https://blog.j2sw.com/equipment-2/mikrotik/mikrotik-map-for-the-wisp-installer/. Go to your local Mcdonalds and hop on their wifi and see if you can get to everything you need to before you fly halfway around the world.
Step 5 – Documentation
Make sure documentation is up-to-date before you leave. If it is not, and you can’t, instead of scheduling installs while you are gone, make your crew(s) update all documentation. Have them take pictures of tower sites, record model numbers, serial numbers, etc. If you are a company that already has good documentation then great. You are already one step closer to that vacation.
Documentation goes hand in hand with step two above. Having a documented network is different because it is more inclusive. Things like directions to a site, gate codes, I recommend having a physical file on every broadcast location technicians will need access to. This could be a PDF they print out or in a file cabinet somewhere. Don’t always count on your techs being able to access online systems.
Modern billing systems keep track of this information as well but having a sheet on file outside of this can save time. The tech can just print it out and start following along. Maybe it’s your brother or neighbor who needs the information. Or the landowner the tower is on. Updating documentation and audits are perfect things for employees to do when you are away.
Step 6 – Prioritize what needs to be escalated to you
Before you leave set expectations for your availability. Your staff should be able to tell customers you will be unavailable until the evenings or just plain unavailable. You should also prioritize what is an emergency and what can wait.
Should you be disturbed because of a single customer is offline?
Can it wait?
Step 7 – Have someone minding the store
If you are a one-person shop or small company consider an answering service for the time you are away. Have someone that can tell your customers something. You will find most customers are willing to accept an answer they will have to wait as long as they are told that. Sure, you will have customers who are unruly no matter what you tell them
Step 8 – Invest in automation
There are a plethora of monitoring and ticketing solutions out there. These are invaluable for cutting down on troubleshooting and just plain letting you know what’s going on when you are 3,000 miles away. Systems such as Zabbix, LibreNMS, or even the dude are all free tools to alert you to trouble. Billing software is another way you can automate. For wisps, I recommend this article https://blog.j2sw.com/xisp/xisp-billing-platforms/
Step 9 – Set aside time each morning or evening to check-in
if you must check-in, set a schedule. Something like every morning at 9-10AM you are going to be sitting by the pool, under an umbrella, with your laptop and will be available to talk about any issues. During this time you can
Step 10 – Don’t’ answer all e-mails and tickets
It is easy to get “on a roll” when you have your laptop or phone in front of you. It’s easy to lose a couple of hours sorting and replying to e-mails. Don’t get caught by this trap. Go through your inbox and only deal with the e-mails that you deem a high priority. If it is a priority item deal with it right then. If it can wait, mark it and deal with it later. Some folks with use message flags, others will move the message to a special folder to look at later. Do not use your inbox as a to-do list. At the very least, go through your inbox and deal with only the items you set in step 6. Even if you have extra time, don’t deal with lower than necessary items.
Having a clean inbox can make reading and respond to e-mail on a mobile device easier as well. Studies show you are more apt to delete unimportant emails in an inbox with fewer items than more. Your mind wants to keep it clean.
Step 11 – Don’t want until the last minute
A week before you are to leave put a freeze on any service impacting upgrades on your network. Don’t fall into the mindset where you have to do all the things you have been meaning to do before you leave. If it is not a security upgrade then it can wait until you get back. I have seen too many operators spend days leading up to when the leave updating software and doing all the things they have been wanting to do for a while. The problem with this is it usually takes 2-3 days for customers to start calling in with non-outage complaints. If the upgrade slows down customer speeds it may take a few days to start getting the calls. This can be problematic if you are already traveling.
Many Wireless Internet Service Providers (WISPs), especially newer startups, struggle with nat issues and having enough public Ip addresses to go around. Invariably, you start running into double nat issues pretty quickly. Then you get the dreaded gamer call:
Many times they don’t know why they are even calling. They just know the magic box is saying this is bad. This is related to how many layers of nat between your edge and them. Many times you are natting at the edge, then you are natting at the customer router. If you have multiple customers behind the same nat at the edge this compounds it even more.
So what is the fix? Give the customer public addresses. But IPv4 is hard to get! I didn’t say IPV4 I said public addresses. IPv6 is a public address. When given the choice between v4 and v6 most modern streaming and gaming platforms will prefer v6. Xbox has supported a protocol called Teredo for a long time. You can learn all about Teredo in this PDF. Basically, it is a tunnel in which the Xbox speaks ipv6 over the tunnel. The ISP does not have to support v6, which does away with the above-mentioned nat issues.
Great! I don’t have to worry about IPv6, Microsoft has it taken care of for me. There are two problems with this statement. Problem number one. There are more companies out there than Microsoft. Sony Playstation Online, Apple gaming, and Steam are just a few. Second, you have overhead of tunnels. In the world of who can pull the joystick quicker, milliseconds count. You don’t want them wasted in tunnel overhead. Plus, v6 is beneficial for other service such as Netflix.
Any other service that runs into port issues behind nat can be solved with Ipv6, This can be voip, cameras, and other type services. This is providing the product or service supports v6 addresses.
So what is an ISP to do?
Awhile back I put together a resource guide for ISPs. You can find it at https://blog.j2sw.com/networking/ipv6/ipv6-planning-and-implementation-resources-for-the-xisp/
I have been wanting to do some photos and thoughts on the Mikrotik SXTR-LTEs and other Mikrotik LTE products. I recently fired one up using dual sims. One is from Tmobile and one is from At&T. Verizon is pretty nonexistent in my area. I am about 2.5 miles away from a Tmobile tower and about a mile from a fiber-fed AT&T monopole.
As you notice in the following photo I am pretty buried in trees.
Some initial notes. Setup of LTE is a very easy process as far as the mikrotik is concerned. I literally had to put in some information in the APN and that was it as far as LTE goes. I did set up standard Mikrotik stuff (DHCP server, security, etc.).
Adding the second sim card can be a huge pain due to the location of the sim card slot. Luckily I had some tweezers that were angled to be able to slide the card in the slot. These were part of a dental kit I picked up off Amazon for releasing stuck SFPs and the like.
Look for a more in-depth series on Mikrotik LTE coming soon.
Recently I sat down and did a podcast with Daniel White of Atheral (https://atheral.co/). We talked about how WISPS can use VOIP to prevent overbuilding, Unified communications, and some of the forms and procedures for getting started with VOIP.
If you want to contact Atheral about their products you can do so at the following:
What is FCC Form 499‑A?
FCC form 499‑A must be filed by interstate or international telecommunications providers in the US to register for the Universal Service Fund and report their revenue. You’ll have the form 499‑A immediately inside your registered agent account with our DC agent information pre-populated on it immediately after signing up for our DC registered agent service.
Inteserra Consulting Group: https://www.inteserra.com/tom-forte
Lerman Senter: https://www.lermansenter.com/attorneys/stephen-e-coran/
Compliance Solutions: https://www.csilongwood.com/
Marashlian & Donahue, PLLC: https://commlawgroup.com/
Viewsheds are a feature present in the Online Map (using V5.3.0 or later), which indicate all the locations in the area surrounding a chosen place that are considered to be Line-of-Sight for a given tower height and subscriber height.
Customer Ethernet plugs in inside with the poe injector. No outlets needed outside. This is just stuff I had laying around. A HEX poe would be the new model of mikrotik to make this work. Will power cambium CPE okay.
In an effort to alleviate the ramifications of COVID-19, Horvath Towers V will be offering free tower co-location to rural broadband service providers for a period of six months.
“With so many families working and learning from home,” company President Jackie Horvath told Inside Towers, “the demand for wireless internet access has sky-rocketed. As such, we would like to partner with rural internet service providers to allow co-location on our tower assets on a temporary basis.”
Applications will be accepted between now and May 1. All inquiries are to be sent to firstname.lastname@example.org. As part of this program, the broadband provider will be responsible for the cost of installation and the equipment. The installation team must provide proper insurance before climbing the tower.
https://www.horvathcommunications.com/ has a map with a site list
Over the years my views and philosophies on being a consultant have changed and are constantly evolving. There are certain things that consultants can incorporate into their businesses in order to maintain a high level of service to clients.
One of the things I have tried to do is be neutral when it comes to vendors and technology. While this is an admirable goal to have, you will find yourself gravitating toward technology you and your clients find useful and proven. It’s okay to be a certified consultant for a specific vendor. This brings up a whole new set of issues I will talk about later. There are two keys to take away from this. The first is to understand the underlying technology as a whole. If you think a particular product is superior enough for you to become certified in it, know why. Know how it is better than the competitors and where it lacks compared to the competitors.
The second key is to not be influenced by becoming a reseller/distributor for particular products. If you want to become a distributor, then focus on that. If you offer consulting services, become an integrator for that product. This way, you are not influenced by the latest promotion for a particular product and try to make it fit for a customer when something else might be better.
As a consultant, you will probably find yourself working with specific products more than others. This is natural. I have found myself working with Cambium ePMP products more often than some others. I believe in the product, so I recommend it to my customers when it fits their situation. However, becoming an expert on a product line has pitfalls.
The first pitfall is you are an expert not paid by the vendor. If you are doing an excellent job on Social media and SEO your name should be popping up in google searches for that product. For example, if you do a search for “Cambium Consultant,” the first page that pops up is a page with my info on it. In a way, you are representing the brand without knowing it. This can lead to you answering questions about a product without any direct compensation for your time. I have always strived to answer questions on topics I am an expert on. There is a fine line between answering questions to a client who has not paid you money and one who has. Every potential contact is a potential client. You have to decide how to handle that grey area. This is an area I struggle with regularly. I am a Cambium ePMP expert and get many questions on this and that from folks who are not clients. I try and answer as many as I can, but at the end of the day, the paying clients do take priority.
Distributors and ordering
I mentioned earlier about me personally, not wanting to be a distributor or reseller. I don’t want to have to meet quotas and absolute minimums to keep stock of products. Some companies are better at this than I ever could be. Having a good relationship with a few good distributors is a good idea. Over the years, I have developed good relationships with several of these WISP distributors. There are some I shy away from due to they have competing services. There are a few vendors and distributors I have referred folks to, and the next thing I know they are offering them consulting services or saying, “I can fix that real quick for you”. They may not even realize they are hurting my business. These are distributors and vendors I personally stop referring business to. If it’s the right product, I will still include them in options for clients, but I make sure I keep on top of the relationship between myself, the vendor, and the client.
There are distributors out there who do very well offering consulting services. The question to ask is are they selling you products because the product makes them money or is it the right product for you? There is much room for either way.
Just some random 3am thoughts