OpenGear Resilience gateway for ISPs

Some quick notes and screenshots from the OpenGear Resilience Gateway https://opengear.com/products/acm7000-resilience-gateway . The model I am working with is the ACM7004-2-L. It has 4 serial Cisco Straight pinout, Dual 1 GbE Ethernet, Global 4G LTE-A Pro cellular, 2 DIO, and 2 output ports.

So what does this thing do and what can it do for you as an ISP? At the basic level, this is a console server with multi wan capability. What this means is when the crap hits the fan you should be able to login to this device across the internet and see what your switches and routers are doing across a console connection. In most ISP scenarios they are bringing in their internet connections from another provider and landing it on a switch or a router. As most followers of this blog know I am a fan of switch-centric based setups. this means your transport and internet connections are landed on a switch or switches and then a router on a stick attaches to these switches.

So why would you need this setup? Not every POP site justifies, or has available multiple transport or internet connections. Imagine you have a switch plugged in and that switch doesn’t come back from a reboot or power event? Without a console server such as this you are driving to the site and plugging in a console cable to see what is going on. With this you can access the device over on of the multiple wan connections, including a cellular connection to gain console access.

Even in redundant setups, a console server can give you insight into what is going on with a router or switch. You can access the console port without ever having to drive. Is the switch booting? Is it getting stuck on a bootloader somewhere? This is all information you can gain from the console port.

Some Screenshots of the Gui. One of the things I like is the dashboard. I am a sucker for dashboards. One reason I am is on any new piece of gear I am reviewing or learning a well thought out dashboard will give me much of the information I need to know. Are my interfaces up? Have VPN connections established? These can help me learn as well as save time troubleshooting

Some interesting notes about the features of this device. It does have environmental status indicators. If you have a device that you can plug into one of the console ports either via USB or rj45 console you can use the gateway to monitor this. Couple this with the Nagios and/or SNMP integration you now have a temperature, door alarm, or other sensors for your remote sites.

View of the back of the unit.

Other notable features include Digital Input and output, remote syslog monitoring, IPSec and OpenVPN, and many other features. If you are deploying lots of these Opengear has a Lighthouse Server for centralized management.

One of the best things I like about this is you are able to access the console server via the web interface. And the best thing? No Java required. This saves from remembering complicated port numbers, for when you ssh and want to access a specific device.

So how am I using this in a network? this device is going at a data center. The client has two cisco switches and two mikrotik routers which will plug into this. It will have an in-band wan connection on a management vlan directly into both routers. If both of these routers are down the gateway has a cellular backup with a IPSEC VPN to a router in a remote data center. You could always switch this up by connecting your second ethernet port into a secondary ISP in the data center. Some networks have a management router where management devices such as this plug into. I have done this with Mikrotik 4011s and it works just fine. I can plug an in-band connection into the mikrotik and a secondary ISP such as a cable or other ISP in the data center.

The cost may discourage some folks. On Amazon, these are just under a thousand dollars. If you need more console ports the price goes up from there. To them, I say what are the costs of downtime and your time. For this client, the closest tech is an hour away. I am two hours away. If a simple firmware or bootloader command fixes a switch not booting and turns 2 hours of minimum downtime into 5 minutes that is a huge win.

Look for a video overview soon.

OpenVPN, rooter project, and Mikrotik

Over the past couple of weeks, I have been fighting with getting an LTE device running The Rooter Project to establish an OpenVPN connection with a Mikrotik router. Apparently, OPENVPN is the only option when it comes to VPNs on The Rooter Project. For the purpose of this article, I am going to refer to the software as “the rooter”. This is just to denote the device running The Rooter Project software. In my case, this is a GL.iNET GL-X750 LTE device.

There are two parts to this setup. The OpenVPN setup on the Mikrotik and the setup on the rooter.

Mikrotik Setup

The Mikrotik setup is pretty straight forward. There are some great tutorials out there for a more in-depth setup. The RouterOS version I used for this setup is 6.47.

Creating Certificates
You will need to create 3 certificates on the Mikrotik.
1. cert_export_ca-certificate.crt
2.cert_export_client-certificate.crt
3.cert_export_client-certificate.key

/certificate
add name=ca-template common-name=example.com days-valid=3650 key-size=2048 key-usage=crl-sign,key-cert-sign
add name=server-template common-name=*.example.com days-valid=3650 key-size=2048 key-usage=digital-signature,key-encipherment,tls-server
add name=client-template common-name=client.example.com days-valid=3650 key-size=2048 key-usage=tls-client

Signing Certificates
Once you have created the above certificates you will need to sign them with the following

/certificate
sign ca-template name=ca-certificate
sign server-template name=server-certificate ca=ca-certificate
sign client-template name=client-certificate ca=ca-certificate

Exporting Certificates
Run the following commands to add a passphrase to your key certificate and export them to files

/certificate
export-certificate ca-certificate export-passphrase=""
export-certificate client-certificate export-passphrase=j2sw123com

This will give you three files: cert_export_ca-certificate.crtcert_export_client-certificate.crt, and cert_export_client-certificate.key. Download these out of “files” from the Mikrotik to the same computer you have access to the rooter on. I like to rename them to ca.crtclient.crt, and client.key so I can keep track of what is what.



Rooter Client Setup

Caveats
I could not find out how to make the operating system read a config file I would edit by hand. Even after a reboot, the config file would not be read. I am not sure if there is a command to read it into the running-config. If someone knows, let me know and that will make this process much easier.

client
dev tun
proto tcp
remote example.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
cipher AES-128-CBC
auth SHA1
auth-user-pass
redirect-gateway def1
verb 3

In my rooter, the config is in /var/etc. I would cat this occasionally to make sure I did not have any extra options turned on. Since I could not make my edits the file stick, I would make the below changes in the GUI and verify they matched up to my above file.

If your OpenVPN is using a username and password create a file named passowrd.txt and put the username on the first line and the password on the second.

You will need that file along with the three files you generate on the Mikrotik above.

Log in to the router and create you an open VPN instance. In my case, I named it Nexstream because this is who I was working for on this project. You can name it anything you want.

Click on edit and you will be brought to the following screens. Fill them out as shown.

When you get to the bottom this is where you upload your password.text and your cert and key files. If you see anything missing go to the bottom and select the field and click add.

Make sure to hit save and apply before proceeding. Click on “switch to advanced configuration”. Match up your configuration with the following screenshots, which match up with the above config file. You are just basically making the proper checkboxes to match the plain text config I posted above. Again, if anyone knows how to get OpenVPN. on the rooter to read the config in let me know.

Once you have the GUI part done and the certs uploaded to the rooter you will need to deal with the keyphrase via the command line. Simply SSH to the rooter. The below code is a generic code for changing the client.key to not ask for a passphrase anymore.

cd /etc/luci-uploads/
openssl.exe rsa -in client.key -out client.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

Couple of things to note about the process.
1. Your location may vary. You must either be inside the directory with your keys or provide the path to the keys in the OpenSSL command

2.when I uploaded the keys it changed them to cbid.openvpn.FRIENDLYNAME.key.

what my actual code looked like to change the passphrase

cd /etc/luci-uploads/
openssl.exe rsa -in cbid.openvpn.vpnout.key -out cbid.openvpn.vpnout.key
Enter pass phrase for client.key: j2sw123com
writing RSA key

If everything goes well you will be rewarded with the following screen on your OpenVPN main page. If, for some reason, it does not start the system log is actually pretty informative on what is going on.

Everything you wanted to know about NTP

Network Time Protocol (NTP) is a service that can be used to synchronize time on network connected devices.   Before we dive into what NTP is, we need to understand why we need accurate time.

The obvious thing is network devices need an accurate clock.  Things like log files with the proper time stamp are important in troubleshooting.  Accurate timing also helps with security prevention measures.  Some attacks use vulnerabilities in time stamps to add in bad payloads or manipulate data. Some companies require accurate time stamps on files and transactions as well for compliance purposes.

So what are these Stratum levels I hear about?
NTP has several levels divided into stratum. All this is the distance from the reference clock source.  A clock which relays UTC (Coordinated Universal Time) that has little to no delay (we are talking nanoseconds) are Stratum-0 servers. These are not used on the network. These are usually atomic and GPS clocks.  A Stratum-0 server is connected to time servers or stratum-1 via GPS or a national time and frequency transmission.  A Stratum 1 device is a very accurate device and is not connected to a Stratum-0 clock over a network.  A Stratum-2 clock receives NTP packets from a Stratum-1 server, a Stratum-3 receives packets from a Stratum-2 server, and so on.  It’s all relative of where the NTP is in relationship to Stratum-1 servers.

Why are there levels?
The further you get away from Stratum-0 the more delay there is.  Things like jitter and network delays affect accuracy.  Most of us network engineers are concerned with milliseconds (ms) of latency.  Time servers are concerned with nanoseconds (ns). Even a server directly connected to a Stratum-0 reference will add 8-10 nanoseconds to UTC time.

My Mikrotik has an NTP server built in? Is that good enough?
This depends on what level of accuracy you want. Do you just need to make sure all of your routers have the same time? then synchronizing with an upstream time server is probably good enough. Having 5000 devices with the same time, AND not having to manually set them or keep them in sync manually is a huge deal.

Do you run a VOIP switch or need to be compliant when it comes to transactions on servers or need to be compliant with various things like Sox compliance you may need a more accurate time source.

What can I do for more accurate time?
Usually, a dedicated appliance is what many networks use.  These are purpose built hardware that receives a signal from GPS. the more accurate you need the time, the more expensive it will become.  Devices that need to be accurate to the nanosecond are usually more expensive than ones accurate to a microsecond.

If you google NTP Appliance you will get a bunch of results.  If you want to setp up from what you are doing currently you can look into these links:

http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html

How to Build a Stratum 1 NTP Server Using A Raspberry Pi

 

Building a Stratum 1 NTP Server with a Raspberry Pi

 

The importance of Network Monitoring Systems (NMS)

One of our open tickets on MidWest-IX is a member reporting slow speeds on their exchange port. After having them send us some data and a few e-mails back and forth we began looking at their switch port on the fabric.  Right away we noticed errors on the port. After a counter reset the errors were still incrementing

 19 runts  0 giants  1210 CRC  0 no buffer
 1329 input error  0 short frame  0 overrun  0 underrun  0 ignored

This led us to look at our LibreNMS data for this port.  A quick look shows on October 31st the port started seeing input errors.

By drilling down we are able to see exactly when this started happening

We now have responded to the customer to see if anything changed that day. Maybe a new switch, new optic, or software upgrade.  By having this data available in an NMS we were able to cut down on troubleshooting by a huge margin.  We now know when the issue started and are closer to the root cause of this.  Without this data, we would be spending more time trying to diagnose and track down issues.

Route Server Diagram for an IX

Normally on a peering exchange, all connected parties will establish bilateral peering relationships with each other customer connected to the exchange. As the number of connected parties increases, it becomes increasingly more difficult to manage peering relationships with customers of the exchange.

However, by using route servers for peering relationships, the number of BGP sessions per router stays at two, if the IX has deployed redundant servers.

My 3rd WordPress speedup tip

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

Ubiquiti launches Speedtest Server/network

https://blog.ui.com/2019/08/13/ubiquiti-launches-a-speed-test-network/

Ubiquiti launches the Ubiquiti Speedtest, the first public test network integrated with enterprise network equipment. Ubiquiti Speedtest comprises a network of test servers and built-in speed test capabilities. Reports include uplink/downlink throughput and latency. Sharing the results is easy via email or social media.

It appears you can run this on a Ubuntu server or VM. They have an installer and a docker image.   You can do browser-based speed tests or their WiFiman App.

Tests may run over LAN, Wi-Fi, or mobile networks. Ubiquiti Speedtest uses Ubiquiti test endpoints and provides automated and manual test target selection. The automated selection uses a combination of geolocation and latency measurements for determining the best servers. The algorithm may use several parallel endpoints for the best measurement accuracy.