“Glue addresses” in networking

Imagine this scenario.  You have bought an IP or DIA circuit from someone that is going to provide your network with bandwidth.  Typically this company will make the connection, IP wise, over a /30 or even a /29 of IP space.  I have called this the “glue address” for many years.  This is the IP address that binds (the glue reference) you to the other provider’s network. They can route you IP blocks over that glue address or you can establish BGP across it, but it is the static address which binds the two networks together.

Some network folks call this a peering address.  This isn’t wrong but can infer you are doing BGP peering across the address.  You aren’t always doing BGP across the glue address.

#routinglight #packetsdownrange

The problem with peering from a logistics standpoint

Many ISPs run into this problem as part of their growing pains.  This scenario usually starts happening with their third or 4th peer.

Scenario.  ISP grows beyond the single connection they have.  This can be 10 meg, 100 meg, gig or whatever.  They start out looking for redundancy. The ISP brings in a second provider, usually at around the same bandwidth level.  This way the network has two pretty equal paths to go out.

A unique problem usually develops as the network grows to the point of peaking the capacity of both of these connections.  The ISP has to make a decision. Do they increase the capacity to just one provider? Most don’t have the budget to increase capacities to both providers. Now, if you increase one you are favoring one provider over another until the budget allows you to increase capacity on both. You are essentially in a state where you have to favor one provider in order to keep up capacity.  If you fail over to the smaller pipe things could be just as bad as being down.

This is where many ISPs learn the hard way that BGP is not load balancing. But what about padding, communities, local-pref, and all that jazz? We will get to that.  In the meantime, our ISP may have the opportunity to get to an Internet Exchange (IX) and offload things like streaming traffic.  Traffic returns to a little more balance because you essentially have a 3rd provider with the IX connection. But, they growing pains don’t stop there.

As ISP’s, especially WISPs, have more and more resources to deal with cutting down latency they start seeking out better-peered networks.  The next growing pain that becomes apparent is the networks with lots of high-end peers tend to charge more money.  For the ISP to buy bandwidth, they usually have to do it in smaller quantities from these types of providers. Buying this way introduces the probably of a mismatched pipe size again with a twist. The twist is the more, and better peers a network has the more traffic is going to want to travel to that peer. So, the more expensive peer, which you are probably buying less of, now wants to handle more of your traffic.

So, the network geeks will bring up things like padding, communities, local-pref, and all the tricks BGP has.  But, at the end of the day, BGP is not load balancing.  You can *influence* traffic, but BGP does not allow you to say “I want 100 megs of traffic here, and 500 megs here.”  Keep in mind BGP deals with traffic to and from IP blocks, not the traffic itself.

So, how does the ISP solve this? Knowing about your upstream peers is the first thing.  BGP looking glasses, peer reports such as those from Hurricane Electric, and general news help keep you on top of things.  Things such as new peering points, acquisitions, and new data centers can influence an ISPs traffic.  If your equipment supports things such as NetFlow, sflow, and other tools you can begin to build a picture of your traffic and what ASNs it is going to. This is your first major step. Get tools to know what ASNs the traffic is going to   You can then take this data, and look at how your own peers are connected with these ASNs.  You will start to see things like provider A is poorly peered with ASN 2906.

Once you know who your peers are and have a good feel on their peering then you can influence your traffic.  If you know you don’t want to send traffic destined for ASN 2906 in or out provider A you can then start to implement AS padding and all the tricks we mentioned before.  But, you need the greater picture before you can do that.

One last note. Peering is dynamic.  You have to keep on top of the ecosystem as a whole.

Transit, peer, upstream. What do they all mean?

As a service provider, you have a mountain of terms to deal with. As you dive into the realm of BGP, you will hear many terms in regards to peers.  Knowing their names AND your definition of them will serve you well.  I emphasized the and in the last sentence because many people have different definitions of what these terms means. This can be due to how long they have been dealing with networks, what they do with them, and other such things.  For example, many content providers use the term transit differently than an ISP.  So, let’s get on to it.

Transit or upstream
This is what you will hear most often.  A transit peer is someone who you go “through” in order to reach the internet.  You transit their network to reach other networks.  Many folks use the term “upstream provider” when talking about someone they buy their internet from.

Downstream
Someone who is “downstream” is someone  you are providing Internet to.  They are “transiting” your network to reach the Internet.  This is typically someone you are selling Internet to.

Peer
This is the term which probably needs the most clarification when communicating with others about how your BGP is setup.  A peer is most often used as a generic term, much like Soda (or pop depending on where you are from). For example someone could say:
“I have a peer setup with my upstream provider who is Cogent.” This is perfectly acceptable when used with the addition of “my upstream provider”.  Peers are often referred to as “neighbors” or “BGP neighbors”.

Local or Private Peer
So what is a local peer? A local peer is a network you are “peering” with and you are only exchanging routes which are their own or their downstream networks.  A local peer usually happens most often at an Internet Exchange (IX) but can happen in common points where networks meet. The most important thing that defines a local peer is you are not using them to reach IP space which is not being advertised form their ASN.   Your peering relationship is just between the two of you. This gets a little muddy when you are peering on an IX, but thats being picky.

I have trained myself to qualify what I mean by a peer when talking about them. I will often say a “transit peer” or a “local peer”. This helps to add a little bit of clarity to what you mean.

Why is this all important? For one, it helps with keeping everyone on the same page when talking about peering.  I had a case a few weeks ago where a Content provider and I wasted configuration time because our definition of transit was different.  Secondly, you want to be able to classify your peers so you can apply different filter rules to them. For example, with a downstream peer you only want to accept the IP space they have shown you which is their own.  That way you are not sending your own transit traffic over their network. This would be bad.  However, if you are accepting full routes from your transit provider, you want your filters to accept much more IP than a downstream provider. So if you have a team being able to be on the same page about peers will help when it comes to writing filters, and how your routers “treat” the peer in terms of access lists, route filters, etc.