Free peering has a cost

Lately, there has been some discussion about the pricing of Internet Exchanges (IXes) and the reasons behind free peering ports and paid peering ports. In this article, I want to go over some of the benefits and pitfalls of both. 

Free peering
Everyone loves free. Sure, we all do. While most free peering has costs associated, those costs are usually in the form of cross-connect charges. In some data centers, this can be a one-time fee to many hundreds of dollars a month for a typical cross-connect. Herein lies the rub. Any entity providing services to others needs to have money for equipment upgrades, labor, accounting expenses, and costs of operation. This money can come in the form of donations from members or outside sources. A non-profit can be the vehicle for this, but you still need funding to support the entity. If the labor is volunteer, what happens if the most active volunteers get pulled away due to personal or business-related obligations? Do things still run? Is there a plan in place to fix any issues? I am not saying it can’t be done, but life happens. When it comes down to it, most people will choose their family’s welfare over a volunteer gig.

Paid Peering
On the flip side, we have IXes, which charge port fees and other fees for peering. These fees go equipment and operational costs and sometimes salaries. Paid peering costs….well money.

The biggest question to ask yourself is how stable the company is. It does not matter if they are giving away ports or charging. What matters is are they going to be able to provide you service when things get busy, outages happen, or things get busy. When peering becomes more and more critical to a health network and your company’s bottom line, these questions also become essential. It does not matter if it is free or paid if the IX isn’t able to provide service. This scenario can happen on an IX, which charges no money or charges money for a port.

Hurricane Electric 10 gig for $1000 a month

This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
To view this content, you must be a member of Justin Wilson's Patreon
Already a qualifying Patreon member? Refresh to access this content.

Remote Peering

Martin J. Levy from Cloudflare did a presentation about remote peering possibly being a bad thing. In this presentation, he brings up several valid points.

https://www.globalpeeringforum.org/pastEvents/gpf14/presentations/Wed_2_MartinLevy_remote_peering_is_bad_for.pdf

Some thoughts of my own.

Yes, remote peering is happening.  One thing touched upon is the layer3 vs layer2 traffic.  We at MidWest-IX only allow remote peering at a layer2 level unless it is groups like routeviews.org or other non-customer traffic situations.

Many providers are overselling their backbone and transit links.  This oversubscription means access to content networks in places that do not have an exchange or places that do have the content locally can suffer through no fault of the ISP or the content provider.  We have situations with content folks like Netflix who do not join for-profit IXes at the moment, keeping the content further away from customers.  These customers are reaching Netflix through the same transit connections many other providers are.  The can result in congested ports and poor quality for the customer.  The ISP is left trying to find creative ways to offload that traffic.  An Internet Exchange is ideal for these companies because cross-connect charges within data centers are on the rise.

When we first turned up MidWest-IX, now known as FD-IX, in Indianapolis we used a layer2 connection to Chicago to bring some of the most needed peers down to our members.  This connection allowed us to kick-start our IX.  We had one member, who after peering with their top talkers, actually saw an increase in bandwidth.  The data gained told the member that their upstream providers were having a bottleneck issue. They had suspected this for a while, but this confirmed it. Either the upstream provider had a congested link, or their peering ports were getting full.

As content makes it way closer remote peering becomes less and less of an issue.  There are many rural broadband companies just now getting layer2 transport back to carrier hotels. These links may stretch a hundred miles or more to reach the data center.  The rural broadband provider will probably never get a carrier hotel close to them.  As they grow, they might be able to afford to host caching boxes. The additional cost and pipe size to fill the caches is also a determining factor. The tradeoff of hosting and filling multiple cache boxes outweighs the latency of a layer2 circuit back to a carrier hotel.

I think remote peering is necessary to by-pass full links which give the ISP more control over their bandwidth.  In today’s race to cut corners to improve the bottom line having more control over your own network is a good thing. By doing a layer2 remote peer you might actually cut down on your latency, even if your upstream ISP is peered or has cache boxes.

The problem with peering from a logistics standpoint

Many ISPs run into this problem as part of their growing pains.  This scenario usually starts happening with their third or 4th peer.

Scenario.  ISP grows beyond the single connection they have.  This can be 10 meg, 100 meg, gig or whatever.  They start out looking for redundancy. The ISP brings in a second provider, usually at around the same bandwidth level.  This way the network has two pretty equal paths to go out.

A unique problem usually develops as the network grows to the point of peaking the capacity of both of these connections.  The ISP has to make a decision. Do they increase the capacity to just one provider? Most don’t have the budget to increase capacities to both providers. Now, if you increase one you are favoring one provider over another until the budget allows you to increase capacity on both. You are essentially in a state where you have to favor one provider in order to keep up capacity.  If you fail over to the smaller pipe things could be just as bad as being down.

This is where many ISPs learn the hard way that BGP is not load balancing. But what about padding, communities, local-pref, and all that jazz? We will get to that.  In the meantime, our ISP may have the opportunity to get to an Internet Exchange (IX) and offload things like streaming traffic.  Traffic returns to a little more balance because you essentially have a 3rd provider with the IX connection. But, they growing pains don’t stop there.

As ISP’s, especially WISPs, have more and more resources to deal with cutting down latency they start seeking out better-peered networks.  The next growing pain that becomes apparent is the networks with lots of high-end peers tend to charge more money.  For the ISP to buy bandwidth, they usually have to do it in smaller quantities from these types of providers. Buying this way introduces the probably of a mismatched pipe size again with a twist. The twist is the more, and better peers a network has the more traffic is going to want to travel to that peer. So, the more expensive peer, which you are probably buying less of, now wants to handle more of your traffic.

So, the network geeks will bring up things like padding, communities, local-pref, and all the tricks BGP has.  But, at the end of the day, BGP is not load balancing.  You can *influence* traffic, but BGP does not allow you to say “I want 100 megs of traffic here, and 500 megs here.”  Keep in mind BGP deals with traffic to and from IP blocks, not the traffic itself.

So, how does the ISP solve this? Knowing about your upstream peers is the first thing.  BGP looking glasses, peer reports such as those from Hurricane Electric, and general news help keep you on top of things.  Things such as new peering points, acquisitions, and new data centers can influence an ISPs traffic.  If your equipment supports things such as NetFlow, sflow, and other tools you can begin to build a picture of your traffic and what ASNs it is going to. This is your first major step. Get tools to know what ASNs the traffic is going to   You can then take this data, and look at how your own peers are connected with these ASNs.  You will start to see things like provider A is poorly peered with ASN 2906.

Once you know who your peers are and have a good feel on their peering then you can influence your traffic.  If you know you don’t want to send traffic destined for ASN 2906 in or out provider A you can then start to implement AS padding and all the tricks we mentioned before.  But, you need the greater picture before you can do that.

One last note. Peering is dynamic.  You have to keep on top of the ecosystem as a whole.