OpenGear Resilience gateway for ISPs

Some quick notes and screenshots from the OpenGear Resilience Gateway . The model I am working with is the ACM7004-2-L. It has 4 serial Cisco Straight pinout, Dual 1 GbE Ethernet, Global 4G LTE-A Pro cellular, 2 DIO, and 2 output ports.

So what does this thing do and what can it do for you as an ISP? At the basic level, this is a console server with multi wan capability. What this means is when the crap hits the fan you should be able to login to this device across the internet and see what your switches and routers are doing across a console connection. In most ISP scenarios they are bringing in their internet connections from another provider and landing it on a switch or a router. As most followers of this blog know I am a fan of switch-centric based setups. this means your transport and internet connections are landed on a switch or switches and then a router on a stick attaches to these switches.

So why would you need this setup? Not every POP site justifies, or has available multiple transport or internet connections. Imagine you have a switch plugged in and that switch doesn’t come back from a reboot or power event? Without a console server such as this you are driving to the site and plugging in a console cable to see what is going on. With this you can access the device over on of the multiple wan connections, including a cellular connection to gain console access.

Even in redundant setups, a console server can give you insight into what is going on with a router or switch. You can access the console port without ever having to drive. Is the switch booting? Is it getting stuck on a bootloader somewhere? This is all information you can gain from the console port.

Some Screenshots of the Gui. One of the things I like is the dashboard. I am a sucker for dashboards. One reason I am is on any new piece of gear I am reviewing or learning a well thought out dashboard will give me much of the information I need to know. Are my interfaces up? Have VPN connections established? These can help me learn as well as save time troubleshooting

Some interesting notes about the features of this device. It does have environmental status indicators. If you have a device that you can plug into one of the console ports either via USB or rj45 console you can use the gateway to monitor this. Couple this with the Nagios and/or SNMP integration you now have a temperature, door alarm, or other sensors for your remote sites.

View of the back of the unit.

Other notable features include Digital Input and output, remote syslog monitoring, IPSec and OpenVPN, and many other features. If you are deploying lots of these Opengear has a Lighthouse Server for centralized management.

One of the best things I like about this is you are able to access the console server via the web interface. And the best thing? No Java required. This saves from remembering complicated port numbers, for when you ssh and want to access a specific device.

So how am I using this in a network? this device is going at a data center. The client has two cisco switches and two mikrotik routers which will plug into this. It will have an in-band wan connection on a management vlan directly into both routers. If both of these routers are down the gateway has a cellular backup with a IPSEC VPN to a router in a remote data center. You could always switch this up by connecting your second ethernet port into a secondary ISP in the data center. Some networks have a management router where management devices such as this plug into. I have done this with Mikrotik 4011s and it works just fine. I can plug an in-band connection into the mikrotik and a secondary ISP such as a cable or other ISP in the data center.

The cost may discourage some folks. On Amazon, these are just under a thousand dollars. If you need more console ports the price goes up from there. To them, I say what are the costs of downtime and your time. For this client, the closest tech is an hour away. I am two hours away. If a simple firmware or bootloader command fixes a switch not booting and turns 2 hours of minimum downtime into 5 minutes that is a huge win.

Look for a video overview soon.

OpenVPN, rooter project, and Mikrotik

Over the past couple of weeks, I have been fighting with getting an LTE device running The Rooter Project to establish an OpenVPN connection with a Mikrotik router. Apparently, OPENVPN is the only option when it comes to VPNs on The Rooter Project. For the purpose of this article, I am going to refer to the software as “the rooter”. This is just to denote the device running The Rooter Project software. In my case, this is a GL.iNET GL-X750 LTE device.

There are two parts to this setup. The OpenVPN setup on the Mikrotik and the setup on the rooter.

Mikrotik Setup

The Mikrotik setup is pretty straight forward. There are some great tutorials out there for a more in-depth setup. The RouterOS version I used for this setup is 6.47.

Creating Certificates
You will need to create 3 certificates on the Mikrotik.
1. cert_export_ca-certificate.crt

add name=ca-template days-valid=3650 key-size=2048 key-usage=crl-sign,key-cert-sign
add name=server-template common-name=* days-valid=3650 key-size=2048 key-usage=digital-signature,key-encipherment,tls-server
add name=client-template days-valid=3650 key-size=2048 key-usage=tls-client

Signing Certificates
Once you have created the above certificates you will need to sign them with the following

sign ca-template name=ca-certificate
sign server-template name=server-certificate ca=ca-certificate
sign client-template name=client-certificate ca=ca-certificate

Exporting Certificates
Run the following commands to add a passphrase to your key certificate and export them to files

export-certificate ca-certificate export-passphrase=""
export-certificate client-certificate export-passphrase=j2sw123com

This will give you three files: cert_export_ca-certificate.crtcert_export_client-certificate.crt, and cert_export_client-certificate.key. Download these out of “files” from the Mikrotik to the same computer you have access to the rooter on. I like to rename them to ca.crtclient.crt, and client.key so I can keep track of what is what.

Rooter Client Setup

I could not find out how to make the operating system read a config file I would edit by hand. Even after a reboot, the config file would not be read. I am not sure if there is a command to read it into the running-config. If someone knows, let me know and that will make this process much easier.

dev tun
proto tcp
remote 1194
resolv-retry infinite
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
cipher AES-128-CBC
auth SHA1
redirect-gateway def1
verb 3

In my rooter, the config is in /var/etc. I would cat this occasionally to make sure I did not have any extra options turned on. Since I could not make my edits the file stick, I would make the below changes in the GUI and verify they matched up to my above file.

If your OpenVPN is using a username and password create a file named passowrd.txt and put the username on the first line and the password on the second.

You will need that file along with the three files you generate on the Mikrotik above.

Log in to the router and create you an open VPN instance. In my case, I named it Nexstream because this is who I was working for on this project. You can name it anything you want.

Click on edit and you will be brought to the following screens. Fill them out as shown.

When you get to the bottom this is where you upload your password.text and your cert and key files. If you see anything missing go to the bottom and select the field and click add.

Make sure to hit save and apply before proceeding. Click on “switch to advanced configuration”. Match up your configuration with the following screenshots, which match up with the above config file. You are just basically making the proper checkboxes to match the plain text config I posted above. Again, if anyone knows how to get OpenVPN. on the rooter to read the config in let me know.

Once you have the GUI part done and the certs uploaded to the rooter you will need to deal with the keyphrase via the command line. Simply SSH to the rooter. The below code is a generic code for changing the client.key to not ask for a passphrase anymore.

cd /etc/luci-uploads/
openssl.exe rsa -in client.key -out client.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

Couple of things to note about the process.
1. Your location may vary. You must either be inside the directory with your keys or provide the path to the keys in the OpenSSL command

2.when I uploaded the keys it changed them to cbid.openvpn.FRIENDLYNAME.key.

what my actual code looked like to change the passphrase

cd /etc/luci-uploads/
openssl.exe rsa -in cbid.openvpn.vpnout.key -out cbid.openvpn.vpnout.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

If everything goes well you will be rewarded with the following screen on your OpenVPN main page. If, for some reason, it does not start the system log is actually pretty informative on what is going on.

Some Mikrotik SXT photos and first thoughts.

I have been wanting to do some photos and thoughts on the Mikrotik SXTR-LTEs and other Mikrotik LTE products. I recently fired one up using dual sims. One is from Tmobile and one is from At&T.  Verizon is pretty nonexistent in my area. I am about 2.5 miles away from a Tmobile tower and about a mile from a fiber-fed AT&T monopole.

As you notice in the following photo I am pretty buried in trees.

My view of the tower. Notice the high-tech holder.

Some initial notes.  Setup of LTE is a very easy process as far as the mikrotik is concerned.  I literally had to put in some information in the APN and that was it as far as LTE goes.  I did set up standard Mikrotik stuff (DHCP server, security, etc.).

Adding the second sim card can be a huge pain due to the location of the sim card slot.  Luckily I had some tweezers that were angled to be able to slide the card in the slot.  These were part of a dental kit I picked up off Amazon for releasing stuck SFPs and the like.

Look for a more in-depth series on Mikrotik LTE coming soon.

LTE components

What is what in the following diagram when it comes to LTE components
eNodeB -Evolved Node B
This is the main component that allows users to connect to the network.

MME – Mobility Management Entity.
The MME is responsible for initiating paging and authentication of the mobile device. MME retains location information at the tracking area level for each user and then selects the appropriate gateway during the initial registration process.

S-GW-Serving Gateway
The S-GW is responsible for keeping track of devices when they move between eNobeB’s. This is typically not an extra piece of hardware just a function of the EPC

This is what connects the LTE network to the Capital I Internet. This also is typically not an extra piece of hardware just a function of the EPC


Other terms
The S1 interface is described in the 3GPP TS 36.410 specification.

The X2 interface provides connectivity between two or more eNodeBs.


VXLAN and why you should care as a service provider

As some of you may have heard Mikrotik has added in some VXLAN support in the latest RouterOS7 beta.  What is VXLAN and how would service providers use it? Let’s start out with some broad information about VXLAN

Where does TRILL and VXLAN fit in to your network strategy?

The always interesting RFC read

This document describes Virtual eXtensible Local Area Network
   (VXLAN), which is used to address the need for overlay networks
   within virtualized data centers accommodating multiple tenants.  The
   scheme and the related protocols can be used in networks for cloud
   service providers and enterprise data centers

Boil it down for me. What is vxlan?
In short, VXLAN allows you to create a layer2 network on top of a layer3 network. It allows you to bind separate layer2 domains and make them look like one. If you are thinking this looks like a GRE tunnel, you are correct except the layer2 domains are still separate with tunnels. VXLAN is mainly touted as a way to interconnect data centers. If you are having to use spanning-tree then VLXLAN is an answer.

Okay, but why not use tunnels or MPLS?
VXLAN allows you to accomplish what GRE does without having to change the network design. By using VXLAN you are also able to have standalone layer2 domains that talk to each other. With the tunnel approach, you have to do a lot of manual configuration.

Is this just a data center thing?
VXLAN was designed to solve many of the edge computing and hyper-scale computing issues. Imagine having compute nodes in different parts of a data center or even in different data centers.  You want all of those nodes on the same VLAN.  With GRE you could extend that VLAN, but with VXLAN you can have two standalone layer2 VLANs that are merged together. VXLAN also solves the 4096 VLAN issue.  This is important in hyper-scale cloud computing.

VXLAN benefits in a nutshell

  • increases layer2 segments to 16 million
  • Centralize control
  • Standards-based
  • Scalable

VXLAN downsides in a nutshell

  • Multicast must be available
  • more overhead to layer2 packet
  • no built-in encryption
  • Slow adoption of ipv6 support by open source

What about the service provider? How can I use this?
In a service-provider network, you have things like broadcast issues. Basically, bridging is bad. Your layer2 networks need to be contained. Imagine you are a service provider who is providing LTE services. You may have an LTE VLAN on your network.  Historically you would have to extend your VLAN across the network in order to do management and access your LTE core. Now you have this large broadcast domain across your entire network.  Or worse yet, you have tunnels to other cities or locations you don’t have physically connected to your network.  Now you have tunnels a part of your LTE VLAN.  MTU issues and other things are now a part of your life.

With VXLAN each LTE node can have its own layer2 VLAN but still talk to the others. This prevents the broadcast storms which can occur.

Another use for VXLAN is a way to allow managed service providers to deploy large scale networks over the 4000 limits of VLANs.  You could literally deploy thousands of layer2 segments to tenants

Why I should or should not care about VXLAN as a service provider?
If you just have a couple of layer2 networks to extend across your network VXLAN is not for you. However, VXLAN does allow for multipath routing and other protocols to be extended to remote networks.

VXLAN adds 50+ bytes of overhead to the layer2 frame. In many service provider networks, this is not an issue due to MTU being raised for MPLS, etc.   IP multicast must be extended across the entire network. Mac addresses are used in creating a distribution network across all of the routed layer2 domains.

Large service providers have started looking at segment routing to solve many of the issues I talk about. This causing them to gravitate toward EVPN. EVPN allows for BGP for the control plane and MPLS for the data plane. More on this coming soon.

In closing, VXLAN is an ultra-cool technology and has use cases for service providers.  Other methods also exist to solve these issues in the service provider world. For those of you looking to learn all you can, I will be posting a list of links for my Patreon folks.

Importance of PIM in LTE

As the number of WISP LTE deployments increase, there are many things WISPs will need to be mindful of.  One such item is properly supporting antenna cables. LTE systems are more sensitive to cable issues.  In a previous blog post, I talked about pim and low-pim cables.   One of the things that can cause low pim is improperly mated cables.  If cables are not supported they can become loose over time.  Vibration from equipment or even the wind can loosen connections.

How do we support cables?
We can take a cue from the cellular industry. The following are some examples of proper cable support.  Thanks to Joshua Powell for these pics.

Where can you get these?
A good place to start are sites like sitepro1 or Tessco has a selection.

So the next time you are planning your LTE deployment think about cable support.