RPKI and misconceptions

After my blog post about Hurricane Electric and RPKi support, I was seeing some comments by folks that warrant some clarification. I put together a short midnight podcast on this.  To summarize
1. route original validation is not the same as having ROA’S with your RIR
2. If you have an ASN you should have a peering DB entry
3.ROAs have nothing to do with your router supporting RPKI

Unimus version 2.0 released

A short list of highlights from this release:
– “Zones” feature for support of remote networks, including Unimus Core (remote agent / remote probe)
– Full Config Change Notifications (including a diff) over Slack
– Config Push Scheduling (MCP presets can now be scheduled)
– Push backups into Unimus over our API push endpoint
– Support for binary backups (for now only over the API push endpoint)
– Observium and PRTG support for NMS Sync
– Support for 22 new device types
– MANY other improvements and MANY fixes for various bugs/issues

Forum Thread

Overview Article

Atheral recommended 499 fillng help

Our friends over at Atheral have some companies to help you with your 499 filings.

What is FCC Form 499‑A?
FCC form 499‑A must be filed by interstate or international telecommunications providers in the US to register for the Universal Service Fund and report their revenue. You’ll have the form 499‑A immediately inside your registered agent account with our DC agent information pre-populated on it immediately after signing up for our DC registered agent service.

Inteserra Consulting Group:  https://www.inteserra.com/tom-forte
Lerman Senter:  https://www.lermansenter.com/attorneys/stephen-e-coran/
Compliance Solutions: https://www.csilongwood.com/
Marashlian & Donahue, PLLC: https://commlawgroup.com/

Philosophies as a consultant Vendors, distributors

Over the years my views and philosophies on being a consultant have changed and are constantly evolving.  There are certain things that consultants can incorporate into their businesses in order to maintain a high level of service to clients.

Being Neutral
One of the things I have tried to do is be neutral when it comes to vendors and technology. While this is an admirable goal to have, you will find yourself gravitating toward technology you and your clients find useful and proven. It’s okay to be a certified consultant for a specific vendor. This brings up a whole new set of issues I will talk about later. There are two keys to take away from this. The first is to understand the underlying technology as a whole. If you think a particular product is superior enough for you to become certified in it, know why. Know how it is better than the competitors and where it lacks compared to the competitors.

The second key is to not be influenced by becoming a reseller/distributor for particular products. If you want to become a distributor, then focus on that. If you offer consulting services, become an integrator for that product. This way, you are not influenced by the latest promotion for a particular product and try to make it fit for a customer when something else might be better.

Vendor Expertise
As a consultant, you will probably find yourself working with specific products more than others. This is natural. I have found myself working with Cambium ePMP products more often than some others. I believe in the product, so I recommend it to my customers when it fits their situation. However, becoming an expert on a product line has pitfalls.

The first pitfall is you are an expert not paid by the vendor. If you are doing an excellent job on Social media and SEO your name should be popping up in google searches for that product. For example, if you do a search for “Cambium Consultant,” the first page that pops up is a page with my info on it. In a way, you are representing the brand without knowing it. This can lead to you answering questions about a product without any direct compensation for your time. I have always strived to answer questions on topics I am an expert on. There is a fine line between answering questions to a client who has not paid you money and one who has. Every potential contact is a potential client. You have to decide how to handle that grey area. This is an area I struggle with regularly. I am a Cambium ePMP expert and get many questions on this and that from folks who are not clients. I try and answer as many as I can, but at the end of the day, the paying clients do take priority.

Distributors and ordering
I mentioned earlier about me personally, not wanting to be a distributor or reseller. I don’t want to have to meet quotas and absolute minimums to keep stock of products. Some companies are better at this than I ever could be. Having a good relationship with a few good distributors is a good idea. Over the years, I have developed good relationships with several of these WISP distributors. There are some I shy away from due to they have competing services. There are a few vendors and distributors I have referred folks to, and the next thing I know they are offering them consulting services or saying, “I can fix that real quick for you”. They may not even realize they are hurting my business. These are distributors and vendors I personally stop referring business to. If it’s the right product, I will still include them in options for clients, but I make sure I keep on top of the relationship between myself, the vendor, and the client.

There are distributors out there who do very well offering consulting services. The question to ask is are they selling you products because the product makes them money or is it the right product for you? There is much room for either way.

Just some random 3am thoughts

My introduction to the world wide web

Tonight I was reflecting on my career in the Internet Service provider Industry. I have been doing “ISP stuff” since 1992. Back in those days we only had BBSes. For the younger generation, these were Bulletin Board Systems.  Think of Facebook without the pictures, the like buttons, the algorithms, and all the stuff we take for granted.  These were places you could post messages, download files, and read other messages people have posted. Much like the bulletin board at the local grocery store.

Over the years Internet usage has changed. When I first started with BBSes and my commodore 64’s 300 baud modem, many of the BBSes were long distance.  This meant you quickly learned how to navigate the menus and get the information you need. From these BBS days, the Internet Service Provider (ISP) was born. With the invention of HTTP, we now had a standard way of viewing content on this new worldwide web.

I remember visiting a computer lab at Purdue University shortly after I received my driver’s license in 1993.I looked old enough to be attending Purdue so I would wander around campus looking for interesting places to visit. After a few times, I happened across a computer ab in the basement of one of the big buildings on campus.  It was a dimly lit place with both PC and Macintosh computers. At that time the Macintosh computers were not as heavily used for anything other than word processing.  This meant there were empty computers.  I sat down and on the desktop was an icon for Mosaic. I clicked on this and life changed. This was way before the concept of logins and passwords.

For the next several months I would go to Purdue at least one to two times a week and spend several hours reading and printing things off the web, Usenet, and other resources. I managed

1990s Usenet

to gain a shell account on expert.cc.purdue.edu and was soon learning about mail, ytalk, and Unix shell. I would print off reams of paper and read them at home and school.  Anything I could find.  The Usenet groups were full of F.A.Q.s about anything from guns to cars to TV shows.  One of the ones I remember printing off was a F.A.Q. about Star Trek,  It talked about everything from the Warp drives, to how phasers worked, etc.  Of course, this was all about a TV show but was fascinating.  I had an entire topic on a subject at the touch of a button.  The folks maintaining these newsgroups were the first real content creators.

Sponsored Post: Atheral voice solutions

Founded by two telecommunications veterans in 2018, Atheral is building customer-centric white- label and wholesale cloud solutions for Internet Service Providers that decrease end-user churn and increase profitability while being geo-redundant, highly available, and scalable. We focus on redefining technology in the cloud to minimize capital expenses while providing a predictable operating cost.

Atheral’s core white-label Voice over Internet Protocol (VoIP) platform is simple, flexible, and feature- rich with unique US-based support resources. Atheral is the only white-label VoIP wholesaler that focuses on WISPs while providing a customized branded experience for their end-users. Pricing, just like our platform, is feature-rich and straightforward:

  • Unlimited local and long-distance calling in the United States, Mexico, and Canada
  • One telephone number per user and e911 registration
  • Branded customer-facing documentation
  • 140+ Softswitch features
  • Our companion Android and IOS softphone app

Why should a WISP care about VoIP?

  • Government Funded Competition – Offering VoIP with your awesome broadband experience protects your ISP from being overbuilt by government-funded competitors or enables you to apply for government funding to expand your coverage area.
  • Customer Stickiness – Customers that purchase value-added services are more likely to stay customers, especially when they are satisfied with those services.
  • High Margin – While phone service in the home may be declining, VoIP is growing by leaps and bounds in the business community with the VoIP industry seeing ~21% annual growth through 2025. All those added users equal high margin for you – the average VoIP reseller sees margins in the 40%-65% range.

Visit https://atheral.com or e-mail info@atheral.com

VXLAN and why you should care as a service provider

As some of you may have heard Mikrotik has added in some VXLAN support in the latest RouterOS7 beta.  What is VXLAN and how would service providers use it? Let’s start out with some broad information about VXLAN

Where does TRILL and VXLAN fit in to your network strategy?

The always interesting RFC read

This document describes Virtual eXtensible Local Area Network
   (VXLAN), which is used to address the need for overlay networks
   within virtualized data centers accommodating multiple tenants.  The
   scheme and the related protocols can be used in networks for cloud
   service providers and enterprise data centers

Boil it down for me. What is vxlan?
In short, VXLAN allows you to create a layer2 network on top of a layer3 network. It allows you to bind separate layer2 domains and make them look like one. If you are thinking this looks like a GRE tunnel, you are correct except the layer2 domains are still separate with tunnels. VXLAN is mainly touted as a way to interconnect data centers. If you are having to use spanning-tree then VLXLAN is an answer.

Okay, but why not use tunnels or MPLS?
VXLAN allows you to accomplish what GRE does without having to change the network design. By using VXLAN you are also able to have standalone layer2 domains that talk to each other. With the tunnel approach, you have to do a lot of manual configuration.

Is this just a data center thing?
VXLAN was designed to solve many of the edge computing and hyper-scale computing issues. Imagine having compute nodes in different parts of a data center or even in different data centers.  You want all of those nodes on the same VLAN.  With GRE you could extend that VLAN, but with VXLAN you can have two standalone layer2 VLANs that are merged together. VXLAN also solves the 4096 VLAN issue.  This is important in hyper-scale cloud computing.

VXLAN benefits in a nutshell

  • increases layer2 segments to 16 million
  • Centralize control
  • Standards-based
  • Scalable

VXLAN downsides in a nutshell

  • Multicast must be available
  • more overhead to layer2 packet
  • no built-in encryption
  • Slow adoption of ipv6 support by open source

What about the service provider? How can I use this?
In a service-provider network, you have things like broadcast issues. Basically, bridging is bad. Your layer2 networks need to be contained. Imagine you are a service provider who is providing LTE services. You may have an LTE VLAN on your network.  Historically you would have to extend your VLAN across the network in order to do management and access your LTE core. Now you have this large broadcast domain across your entire network.  Or worse yet, you have tunnels to other cities or locations you don’t have physically connected to your network.  Now you have tunnels a part of your LTE VLAN.  MTU issues and other things are now a part of your life.

With VXLAN each LTE node can have its own layer2 VLAN but still talk to the others. This prevents the broadcast storms which can occur.

Another use for VXLAN is a way to allow managed service providers to deploy large scale networks over the 4000 limits of VLANs.  You could literally deploy thousands of layer2 segments to tenants

Why I should or should not care about VXLAN as a service provider?
If you just have a couple of layer2 networks to extend across your network VXLAN is not for you. However, VXLAN does allow for multipath routing and other protocols to be extended to remote networks.

VXLAN adds 50+ bytes of overhead to the layer2 frame. In many service provider networks, this is not an issue due to MTU being raised for MPLS, etc.   IP multicast must be extended across the entire network. Mac addresses are used in creating a distribution network across all of the routed layer2 domains.

Large service providers have started looking at segment routing to solve many of the issues I talk about. This causing them to gravitate toward EVPN. EVPN allows for BGP for the control plane and MPLS for the data plane. More on this coming soon.

In closing, VXLAN is an ultra-cool technology and has use cases for service providers.  Other methods also exist to solve these issues in the service provider world. For those of you looking to learn all you can, I will be posting a list of links for my Patreon folks.

Bad habits of a network admin/tech

Recently there was a thread on the Facebook group WISP talk.  The author outlined the following bad habits of network folks.  Below is the original list. I just copied it and it’s not mine.

  1. Install and forget;
  2. If it ain’t broke, don’t fix it;
  3. Make every switch port a layer 3 port;
  4. Go to the races as a cli jockey;
  5. Solve any wireless issue by adding more power;
  6. It is not my problem until you prove to me its my problem (arrogance in general)
  7. Who needs to RTFM?

So let’s expand on each of these and make it my own.

  1. Install it and forget it.
    This is what happens when folks get in a rush and don’t do much prep time. I often hear the excuse “I will document it later.”  If this is a new build 95%+ of the documentation should be done ahead of time.  When you go to actually install it you will have a blueprint to go on instead of making it up on the fly. This makes for a much smoother install.  As a bonus, you don’t have to do as much documentation at the end.  This goes a long way to ensuring it gets done.
  2.  If it ain’t broke, don’t fix it
    A network is as close to a living organism as you can get into the computer world.  Much like your body, issues can creep into the works and cause problems.  Bugs and vulnerabilities in software can be a huge Achilles heal if they are not dealt with. Equipment is replaced with newer models, which have their own unique personalities so to speak. Plus, in ISP networks customers are demanding faster and less latency.  This means you have to be able to upgrade your networks to meet consumer demand.
  3. Make every switch port a layer3 port.
    An ISP network has two major components. These are the transport network and the access network. The transport network gets bits between POPS, tower sites, cities, whatever.  This is the backbone of your network. The access point is what the customers attach to.  This is where customer authentication and provisioning is done. You access network needs to be as low latency as possible.  Layer3 routing adds latency. When you are talking about the customer access network, layer2 ports are handy for wiretap warrants and CALEA requests to name just a few advantages. I have a future blog post in the works on this topic
  4. Go to the races as a cli jockey
    I call these folks CLI snobs. They refuse to touch a GUI. Modern carrier and enterprise devices are very powerful and scriptable.  This is where the CLI can be automated and be very powerful.  However, a web-based GUI can be helpful for lower-level technicians.  This can alleviate some of the load off the higher up admins.  A web-interface can also be a quicker way of doing one-off tasks.  It’s helpful to know both, especially in an environment where you may have to walk an intern over the phone how to set an IP address on a device when they don’t have a serial console.
  5. Solve issues by adding more power
    In the wireless world adding more power can be a bad idea for your company. this can put your company in legal trouble if you are exceeding your governing body’s regulations. Fines and other things can happen.  More power can hurt your network as well as surrounding networks.  Self-interference is a very real thing.
    In the enterprise world you can buy too much of a router that, by the time you grow into it, something better and faster is already out.
  6. & 7Arrogance
    I am combining numbers 6 and 7 above into an overall Arrogance category. In fact, many of the items on this list are a result of arrogance. Over the years many of you have heard me talk about “The typical I.T. person” and folks like Saturday Night Live have done skits on such things. There are many reasons for an arrogant I.T. person (guy or gal).
    Sometimes this person has had to deal with arrogant bosses who do not understand I.T. and that has affected their mood.  Some have issues inside their heads which causes them to think others are inferior to them. There are tons of reasons and whole Psychology books are written on. Whatever the reason arrogance causes these folks to be the way they are.
  7. Believing there is just one right answer
    There are network Architects for a reason in large companies.  These folks set the tone for how the network is laid out.  This is so things do not get bogged down in endless discussions about this vendor and this technology. There are many ways to solve the problem.  You just have to make the best-informed decision and go with it.  Otherwise, you will be caught in an endless amount of what-ifs.
  8. Believing technology is more important than the business
    Technology is what allows a business to make money.  In an ISP the technology is part of what sets you apart.  Without paying customers all the tech in the world would do no good. Smart business decisions have to be made about the tech, but the business leaders also need to understand the tech is responsible for keeping customers happy.
  9. Making it overly complex
    Whether it be arrogance creeping in, wanting to use 100 percent of the features, or just wanting to get the most bang for the buck is where networks can come into play.  Usually, this happens at 2am and you are trying to diagnose the multiple services and layers involved. There is a fine line between providing cutting edge services and the KISS (Keep It Simple Stupid) principle.

These are just a few of the things.  Leave your comments on what you think network and I.T. folks do as bad habits.