Mikrotik CCR1072 for an ISP network running BGP.
Preseem now supports IPv6 for all use cases. This includes the ability to assign subscribers a prefix of arbitrary length.
IPv4 with Prefixes of Arbitrary Length
Previously Preseem modelled subnet assignments to customers as a number of /32 assignments. For example a subscriber who was assigned a /30 would result in four internal /32 mappings. Preseem now supports assigning any prefix length to a subscriber without expanding these into /32 entries internally.
Some quick notes and screenshots from the OpenGear Resilience Gateway https://opengear.com/products/acm7000-resilience-gateway . The model I am working with is the ACM7004-2-L. It has 4 serial Cisco Straight pinout, Dual 1 GbE Ethernet, Global 4G LTE-A Pro cellular, 2 DIO, and 2 output ports.
So what does this thing do and what can it do for you as an ISP? At the basic level, this is a console server with multi wan capability. What this means is when the crap hits the fan you should be able to login to this device across the internet and see what your switches and routers are doing across a console connection. In most ISP scenarios they are bringing in their internet connections from another provider and landing it on a switch or a router. As most followers of this blog know I am a fan of switch-centric based setups. this means your transport and internet connections are landed on a switch or switches and then a router on a stick attaches to these switches.
So why would you need this setup? Not every POP site justifies, or has available multiple transport or internet connections. Imagine you have a switch plugged in and that switch doesn’t come back from a reboot or power event? Without a console server such as this you are driving to the site and plugging in a console cable to see what is going on. With this you can access the device over on of the multiple wan connections, including a cellular connection to gain console access.
Even in redundant setups, a console server can give you insight into what is going on with a router or switch. You can access the console port without ever having to drive. Is the switch booting? Is it getting stuck on a bootloader somewhere? This is all information you can gain from the console port.
Some Screenshots of the Gui. One of the things I like is the dashboard. I am a sucker for dashboards. One reason I am is on any new piece of gear I am reviewing or learning a well thought out dashboard will give me much of the information I need to know. Are my interfaces up? Have VPN connections established? These can help me learn as well as save time troubleshooting
Some interesting notes about the features of this device. It does have environmental status indicators. If you have a device that you can plug into one of the console ports either via USB or rj45 console you can use the gateway to monitor this. Couple this with the Nagios and/or SNMP integration you now have a temperature, door alarm, or other sensors for your remote sites.
Other notable features include Digital Input and output, remote syslog monitoring, IPSec and OpenVPN, and many other features. If you are deploying lots of these Opengear has a Lighthouse Server for centralized management.
One of the best things I like about this is you are able to access the console server via the web interface. And the best thing? No Java required. This saves from remembering complicated port numbers, for when you ssh and want to access a specific device.
So how am I using this in a network? this device is going at a data center. The client has two cisco switches and two mikrotik routers which will plug into this. It will have an in-band wan connection on a management vlan directly into both routers. If both of these routers are down the gateway has a cellular backup with a IPSEC VPN to a router in a remote data center. You could always switch this up by connecting your second ethernet port into a secondary ISP in the data center. Some networks have a management router where management devices such as this plug into. I have done this with Mikrotik 4011s and it works just fine. I can plug an in-band connection into the mikrotik and a secondary ISP such as a cable or other ISP in the data center.
The cost may discourage some folks. On Amazon, these are just under a thousand dollars. If you need more console ports the price goes up from there. To them, I say what are the costs of downtime and your time. For this client, the closest tech is an hour away. I am two hours away. If a simple firmware or bootloader command fixes a switch not booting and turns 2 hours of minimum downtime into 5 minutes that is a huge win.
Look for a video overview soon.
If you are a Hurricane Electric customer you may be receiving e-mails like the following:
Dear ASXXX, Routing Security Report for ASXXX Hurricane Electric cares about your routing security. We filter all BGP sessions using prefix filters based on IRR and RPKI. This report is being sent to help you identify prefixes which may need either their IRR or RPKI information created or updated and to also help you identify possibly hijacked routes you may be accepting and reannouncing. Routes with RPKI status INVALID_ASN strongly indicate a serious problem. IPv4 SUMMARY Routes accepted: 3 Routes rejected: 3 Routes with RPKI status VALID: 0 Routes with RPKI status INVALID: 0 IPv6 SUMMARY Routes accepted: 1 Routes rejected: 0 Routes with RPKI status VALID: 0 Routes with RPKI status INVALID: 0 We currently do not have a valid as-set name for your network. Please add an export line to your aut-num ASXXXX that references your as-set name. For example, export: to AS-ANY announce your-as-set-name If you do not currently have an as-set, we recommend you create one named ASXXXX:AS-ALL Your as-set should contain just your ASN and your customers' ASNs and/or as-sets (not your peers or upstream providers).
What does this mean for you as a service provider? If you use Hurricane Electric as transit or peer with them on an exchange you will need to have ROAs for your blocksand have routing registry objects. I did a tutorial based upon Arin which can be found at: https://blog.j2sw.com/networking/routing-registries-and-you/
In short you need to do the following:
- Create a
mntnerobject (equivalent of a user account) to give you the ability to create IRR objects in your selected IRR database
- Create an
aut-numto represent your autonomous system and describe its contact information (admin and technical) and your routing policy
- Create an
as-setto describe which autonous system numbers your peers should expect to see from you (namely your own and your transit customers)
- Create a
route6object for every prefix originated from your network
- Update your peeringdb profile to include your IRR peering policy
- Generate RPKI https://www.arin.net/resources/manage/rpki/roa_request/#creating-a-roa-in-arin-online
Some folks are confusing having valid ROAs with your router supporting RPKI with route origin validation in real-time. These two are separate things. You create ROA records with your RIR, such as ARIN, which has nothing to do with route validation on your router.
Also, HE is filtering any RPKI INVALID routes. Does this mean they are requiring RPKI? You be the judge.
Gonna try these out for mounting equipment in a rack with square holes. www.rackstuds.com
A carrier-neutral data center, also called a carrier hotel, is a data center facility that is entirely independent of network providers.
This was an older article I had on my blog a few years ago. Much of this applies still.
Bandwidth is a big hurdle most aspiring WISPs face. The reason is if high-speed alternatives were already in place, the need for a WISP would not be as great. Sure there are business models in which the WISP can compete with other high-speed solutions. However, the bread and butter of a WISP is going into underserved areas.
You have several options for bringing a connection into your area to re-distribute to your customers. I will outline these and then go into further detail
-Leased Lines (Fractional, T-1, T3, etc.)
Leased Lines are the most easily accessible across the United States. However, as more and more providers build fiber it is taking over as the preferred method of connectivity. Fiber is more “future proof” than a T-Carrier circuit such as a T1 or T3. Most phone companies can provide t1 service to almost anywhere. This is because T1 service uses the existing copper already at 99% of locations. If you have a phone line you can almost always get t1 service. Once you go beyond T1 things get a little more complicated. However, T1 has the ability to do bonding if the carrier and telco support it. You essentially buy multiple T1s and combine them into a single “pipe”. This requires the provider to support bonding as well as some special configuration on your routers.
Some questions you should ask your provider/telco.
1.Where is my circuit “homed out of”? This means where does the circuit terminate on the facility end. You do not want this to be too far. If it is too far your reliability will suffer because you have more distance and equipment to go through. This raises the likelihood of an equipment failure, backhoe digging something up, & utility poles falling. The longer the distance also means the “loop charge” will most likely increase. We will get to that in a moment.
2.There are several types of T1s for our purposes. Some terms to familiarize oneself with are PRI, channelized, transport, and port fee.
3. Ask your provider to spell out what type of t1 this is. If you are buying the T1 from a backbone provider such as Qwest, Level3, and others they will typically bundle everything into one package. Ask them to break this down if they don’t. You want to know what the Local loop charge is, what the port fee is, and what the bandwidth costs. The local loop is typically what the telephone company charges to deliver the circuit from Point A (their equipment) to Point B (you). If you are going with a 3rd party, and not the local telephone company, the provider typically becomes the central point of contact for the entire circuit. This can add a level of complexity when issues arise.
The port fee is a charge normally passed on for connecting to the provider’s equipment. Say you have a 48 port switch sitting in a CO-Location facility. For each Ethernet cable you plug in from the telephone company they charge a fee either one-time and/or monthly. This is just the way it is typically. One of those “Because they can” charges. The 3rd charge is the cost of the Internet bandwidth. A T1 can handle 1.5 Megabits of bandwidth so the cost per Megabit is not as big of an issue because you are not buying in bulk.
4.Ask to see the Service Level Agreement (SLA). If you are unfamiliar with the terms have a consultant look this over.
5.Know where your DMARC location is. This is the spot where the provider’s responsibility ends and yours begins.
6.Ask if the provider can verify with the telco how long the next circuit would take to install. You don’t want to go to order a second circuit and find out the local telephone equipment does not have enough capacity. This has happened to our clients on many occasions. This can be a quick process or the telco can take months and months to get around to installing the needed equipment.
We like to refer to Indianapolis, Indiana as an “NFL City” when explaining the connectivity and peering landscape. It is not a large network presence like Chicago or Ashburn but has enough networks to make it a place for great interconnects.
At the heart of Indianapolis is the Indy Telcom complex. www.indytelcom.com (currently down as of this writing). This is also referred to as the “Henry Street” complex because West Henry Street runs past several of the buildings. This is a large complex with many buildings on it.
One of the things many of our clients ask about is getting connectivity from building to building on the Indy Telcom campus. Lifeline Data Centers ( www.lifelinedatacenters.com ) operates a carrier hotel at 733 Henry. With at least 30 on-net carriers and access to many more 733 is the place to go for cross-connect connectivity in Indianapolis. We have been told by Indy Telcom the conduits between the buildings on the campus are 100% full. This makes connectivity challenging at best when going between buildings. The campus has lots of space, but the buildings are on islands if you wish to establish dark fiber cross-connects between buildings. Many carriers have lit services, but due to the ways many carriers provision things getting a strand, or even a wave is not possible. We do have some options from companies like Zayo or Lightedge for getting connectivity between buildings, but it is not like Chicago or other big Date centers. However, there is a solution for those looking for to establish interconnections. Lifeline also operates a facility at 401 North Shadeland, which is referred to as the EastGate facility. This facility is built on 41 acres, is FEDRAMP certified, and has a bunch of features. There is a dark fiber ring going between 733 and 401. This is ideal for folks looking for both co-location and connectivity. Servers and other infrastructure can be housed at Eastgate and connectivity can be pulled from 733. This solves the 100% full conduit issue with Indy Telcom. MidWest Internet Exchange ( www.midwest-ix.com ) is also on-net at both 401 and 733.
Another location where MidWest-IX is at is 365 Data Centers (http://www.365datacenters.com ) at 701 West Henry. 365 has a national footprint and thus draws some different clients than some of the other facilities. 365 operates Data centers in Tennessee, Michigan, New York, and others. MidWest has dark fiber over to 365 in order to bring them on their Indy fabric.
Another large presence at Henry Street is Lightbound ( www.lightbound.com ). They have a couple of large facilities. According to PeeringDB, only three carriers are in their 731 facility. However, their web-site lists 18+ carriers in their facilities. The web-site does not list these carriers.
I am a big fan of peeringdb for knowing who is at what facilities, where peering points are, and other geeky information. Many of the facilities in Indianapolis are not listed on peering DB. Some other Data Centers which we know about:
On the north side of Indianapolis, you have Expedient ( www.expedient.com ) in Carmel. Expedient says they have “dozens of on net carriers among all markets”. There are some other data centers in the Indianapolis Metro area. Data Cave in Columbus is within decent driving distance.
Great example of in-floor cabling