These are unedited transcripts and may contain errors.
EIX Working Group, 3rd November 2011 at 2 p.m.. EIX Working Group, 3rd November 2011 at 2 p.m.
CHAIR: Welcome back everybody. Thanks for joining us at the second EIX session this afternoon. Another hour and a half of peering and Internet Exchange related content. There is a very, very slight change to the agenda again, the Working Group Chair's lunch is overrunning so Kurtis is still upstairs, so we'll move him ?? he has just arrived, but we have set the other slides up so we'll do Ondez who was going to do the extended route in community services first and then we'll do Kurtis to do the talk on peering edge, immediately after that.
If I could welcome Ondrez Filip to the stage to do his presentation.
At the end of this session, there is going to be an open?mike period for anybody who runs an Internet Exchange or has a 30?second peering related topic they want to talk about which might start some discussion, that would be very interesting. And I just want to offer one more plug to the BoF session which is running at 5:45, next door tonight on IPv6 privacy issues, which is an open forum, slideless, to discuss any topics of interest to to with IPv6 privacy.
ONDREZ FILIP: My name is Ondrez Filip. I am a board member of CZ.NIC Internet Exchange in Prague and l'd like to talk ?? I'll read a bit about what we have done with extended communities on our route servers, but before I will start, I have a, you know, a few slides about CZ.NIC itself about what are the hot topics and what news in the checks republic in Prague and one thing was really very surprising, nobody expected, but we are actually 15 so we would also like to have a small celebration which is going to be on November 21, in Prague. So if you are interested, you are really welcome, we would like to see you all there, let us know and we'll send you an invitation for that, so there will be, I hope, a nice programme, we'll see tan tear item and all the things Prague can offer to you.
That's the breaking news.
We have some other things to be proud of, we just reached 200 gigs. It happened I think at five, six days ago, so it's really very new thing. It's because we are still growing at a really nice pace, we got a lot of new customers this year. One was very special for us, and it has to do something with the history of the country, we just got radio for Europe and radio liberty, so that's something that was really symbolic for us because that was a company that helped us allot during our history. And other than that, we got a new POP opened, so currently we have currently five POPs, they're connecting with topology, which I have a picture on the next slide and we are continuing upgrading the switches. We are migrating from Cisco 6,500 to Cisco Nexus and first phase we used them as a core switch and now we also start to use Nexus as an access reg. Surprisingly it works to we are quite happy. No issues at all.
So this is our current topology. It's not visible but if you can see the numbers. 1 POP at NIC.CZ 2, 5. 4 is the largest with three extra switches and the third one. That's the current topology. No problems so far as I said, quite stable for a very long time.
But now back to the topic. What was actually the problem? You know, we have route server and some customers decided they do want to filter to whom the updates are advertised, so we introduced a route server policy which is based on standard BGP communities and as you probably know those BGP communities can carry only 16 bit messages, actually 32?bits, but you can just have, you know, two 16 bit numbers. And the problem, you know, emerged when some new members with ASN that are higher than you know 65,000 came. They could filter the old members, but the old members couldn't filter them, so some of the, you know, current members said oh that's a problem, you need to do something with that, you need to fix the problem.
So, this was our current policy to explain the situation in our registry. Usually, there is a way to signal whether you want to peer with somebody or not. If you want to advertise your prefix to some user, you need to type AS number of the route server plus the AS number of the second member, and if you do not want to advertise to anybody, you just do 0 and the AS number of the route server. So, that's probably the scheme that many of other exchangeses follow and as I said, the problem is that the standard communities just support 16 bit ASNs.
So, how we wanted to solve the issue. There are extended communities that are defined in RFCs 4,360. Those communities has a little bit more bits for signalling, they are 48 bits for ASNs to IP addresses and it depends what you want to put there. Plus they have some signalling bits, so it's a little bit more than 48 bits. There is more types of ?? there are more types ?? more messages maybe, you can either send origin and things like that and as I said, you can either send AS number or an IP address in that BGP extended community.
So, we look at it very carefully and we, you know, change all kind of route server policy and we change it to actually this table. So, the logic is really the same. Again, there is a priority, how we evaluate your signals you are sending to the, the member is sending to us. If he or she sends 0 and peer AS, we do not advertise or, he can send a source origination, 0 and peer AS. And again there is signal how to advertise to peer and how to completely, you know, not advertise anything except of course what was accepted in the role a number 2.
The fourth condition is I think redundant and it doesn't make sense to make it because it's by default but just to have a complete picture.
So, that's the new policy. All conditions are evaluated in fixed order from row number 1 to row number 4, and you can mix both, so you can, you know, send a prefix with extended communities and standard communities. It works, we can't avoid that but we of course do not recommend that because that could make the stability of such routing a little bit problematic. But, it's possible. Or all clause are applied. So if this community or the extended community comes, first come, that's supplied and we take it.
And all communities and extended communities that are sent to a route server are removed when we are resending our routes to the peers of course to clean up this signalling thing.
So, so far quite easy. But we had some small problem and the problem is that some Cisco CRS boxes can not generate SO:0 and ASN 32. We checked with the RFC and my understanding is that it's not very clear if this is a valid extended community signal or not. So, you know, we discussed with Cisco and the issue is still open so we'll see what's going to be resolved. So those members propagate SO and, you know, zero IP address and 16 ASN instead. So the problem is they cannot block any peer. If you look at the logic of this slide before, they cannot send this one. But, of course, they still can create their own peering policy. They can do it this way, so they list the numbers, the members they want to peer and by default they do not send their routes to, and they do not send the routes to anybody who is not listed. So, that's way, how they work with that, you know, so far and we'll see if the situation improves or if we will see that there is no way to, you know, to work with that we maybe a little bit amend that policy so we'll see what's going to happen.
We have two route servers, one is BIRD and one is Quagga. It's implemented on both. We have some small problems with Quagga, we know how to clean up the communities, we are still working on, Quagga is quite an old piece of software and it's not very well maintained so we are a little bit fighting with that. As I am one of the authors of BIRD Internet routing, let me talk a little bit about about BIRD because extended communities are quite new, they were introduced since 1.3.3 and the current release is 1.3.4. This really didn't have any new features, it was just a bug fix release and it's one of the recommended, if you want a route server based on BIRD. It's very well tested and ran at the NIX.CZ and I should, I didn't ?? I didn't, but I should very soon publish some configuration at the moment plates if you are interested or you can talk to me, I think I can send it to you by e?mail. And in 1.37 there are some other new features like include, so you can include multiple layers of configuration file, BGP TTL security, OSPF NSSA and things like that. So that's what's in the current release 1.3.4.
So, that's all. So, the extended communities are currently using at NIX.CZ, we use it, the customers used it and it works. We support ASN 32, we have some small issue with Cisco as I said. And about BIRD works, it's mostly Quagga has some small problems, but it is working. And I am sorry we didn't test open BGPD yet so we will work on that but maybe someone knows ?? I understood from the evaluation that this is feature is supported but I am really not sure if it's working practically.
And that's all. Thank you very much.
AUDIENCE SPEAKER: First of all, thank you for your presentation, we are also thinking about introducing it in our site in Amsterdam. Do you have any statistics how many users started using extended communities?
ONDREZ FILIP: Honestly it's just one or two.
AUDIENCE SPEAKER: It's only one or two.
ONDREZ FILIP: Just two members actually, just a few.
AUDIENCE SPEAKER: And how, when it was introduced?
ONDREZ FILIP: It was introduced, it was before euro IX meeting, three months ago.
AUDIENCE SPEAKER: And do you have any issues with some vendors?
ONDREZ FILIP: Except that with Cisco, no, if those guys don't use it and we accept the extended communities, process them and then clean up, so, those guys really don't know that there is something inside, so they shouldn't have any problems actually.
AUDIENCE SPEAKER: Okay. Thank you.
AUDIENCE SPEAKER: Project manager at Cisco. I would like to go to you because I am the guy that defined priority inside Cisco. And I can help you here.
ONDREZ FILIP: Excellent. Perfect. Thank you.
AUDIENCE SPEAKER: That's the first thing. The second thing is there is a new draft about the wider community, which is to go behind that extended community. Is that something of interest for you or for the EIX community in general?
ONDREZ FILIP: You know, I am from the exchange point so I do what the customers want. Since we had the direct request for extend the communities we implemented them and if there was an interest for wide communities, I don't think it's a big deal to do it.
AUDIENCE SPEAKER: I think, the benefit of a wide community, that it's less restrictive. There is more open things you could do with the wide community when extended community has been done for very strict purpose.
ONDREZ FILIP: So, I will, you know ?? give the question to you, so is it supported in your hardware?
AUDIENCE SPEAKER: Not yet. There is a new draft.
ONDREZ FILIP: If that's coming, we are ready to implement that, yeah, no problem.
CHAIR: Thanks. That was a great question. I love it when a plan comes together. Is there anybody in the room that's tried extended communities with open BGP or not? Thank you for the information. That was really useful. Thank you.
(Applause)
CHAIR: Okay. Next Kurtis Lindqvist who is going to talk about originally impeering edge.
KURTIS LINDQVIST: Hi. I am Kurtis, I work for Netnod. We run an exchange point in the north. We are special. We put exchange point in bunkers and for Stockholm they are disconnected. We provide the connectivity from there to you. We have been, as we are growing, and as we are launching new services, we have been thinking and talking to members and internally of how this peering thing actually works. And the title has a question mark ?? I realised today when I reread the slides is that you might actually think or be led to believe that I am going to present some sort of solution here. I am not not. Sorry James. It's a question, during this presentation I am going to come with some statements, claims, and I have nothing to back them up with. They are all questions and I do want to have feedback from you and maybe you can help me become a wiser and more intelligent person.
There are 16 slides and I have 20 minutes and I know I can do it. The question is if you can follow it.
KURT LINDQVIST: Let's start from the beginning. This is the Internet. It's very simplified. I hope you can all follow. I am the acting SIP here, I have peers and I have an up stream. If the peering session failed or if the exchange point fails, I would most like to reach all the designation to say my upstreams. If the upstream fails, I want to restress to the Internet and given how today's peering looked like if might not even reach all the peers, because the traffic.
But there is some level of redundancy here. Okay. We are making it a bit more complex for you. This is closer to the truth. I am connected maybe to two exchange points and I see peers maybe even the same peers at both exchange points. If I lose one of the exchangeses, I could still use those unique peers over the transit and if the transit fails I might use those two exchanges to reach my peers and they might even be enough overlap that the symmetry of traffic might even work a bit better.
So, this is a very fundamental part. I hope you followed me so far. Some of you are glazing over already.
This of course is a very simplified picture of what the world looks like and how we do peering and I really truly wish I could have some grad students and I was asking Randy Bush do his grad students look at this and he said, no, we don't have any. Because I would actually ?? I am really curious to know how is this redundancy for your peers actually do work. But, again, I don't have any facts. I have nothing to back this up so I am just going to go by hearsay and rumours and claim that's the truth.
This is how I think it works.
Why do you guys peer? Probably for one or more of these reasons. You get stuff cheaper access to traffic. You get some lower round trip times, you can make marketing that you are at 1,000500 exchange points and you might even save a bit on capacity or get more capacity cheaper by sending traffic to the exchange point. Where it might be more expensive, but...
But at the end of the day, if you would lose all your peers and you would have to send all traffic to the transit, it wouldn't be cats traffic. It's probably going to be expensive but it will work, it will work so, so, there might be so the congestion or packet loss but I suspect all of you will survive and so will the Internet. It does have some consequences though especially on maybe round trip time. Again congestion might affect this, and based on the feedback I get ?? and again here is the hearsay ?? people seem to believe that swapping all traffic or even parts of it over to transit is actually to be avoided. You guys don't like that. So, most people have actually gone to quite some extent to protect this in one way or the other. You have connected to multiple exchange points. We have provided the redundancy for you, I don't mean just Netnod, we have provided some form of redundancy protection. Multiple exchange points in the same country, loads of things that protects your peerings. And you make an effort to make sure this is redundant. There is a few approaches to this and this slide is wrong by the way, and this is my own exchange point, I couldn't get a slide right anyway.
But, these two switches are independent, they are separated. I have one operator up here who has one router connected to both switches or you have one operator down here who has two routers connected to two switches. So one is redundant and the other one isn't and if you don't know which one is which, I'll leave that for to you think of.
Another approach, and I realise that this was ?? I stole this from another website, when I stole it some explanation would come with it, but it didn't. What I was saying in AMS?IX case they have done is they have provided some level of protection in their network for your traffic. If you buy one port at AMS?IX you get some level of protection words to the peering fabric. They are doing your job for you. We have the links approach which is some way could be seen to the Netnod model. ?? at Netnod we force to you connect to both switches. We will come to your home. Grab you out of bed and make sure you connect to both switches. In the links case they give you a slight incentive financially to connect to the both of the two peering lance, or exchanges and you can then have redundancy for a failure. The draw back is because they are not forced to, you will see less peers in one LAN than the other one. So you will have less redundant but still some form of redundancy.
You can look at this motorway. What is actually creating this level of redundancy for your peers, and again I am assuming that you actually care about peering and you want this to work, if you don't, then you can ignore me. Either the ISPs has multiple ports hopefully on multiple routers, if you more ports on one router, you have thrown a lot of money to the problem but you haven't solved it, or, the exchange point protects the single port and in like the AMS?IX case it profits some level of protection behind this so that if the failure of the exchange won't affect your traffic. Again if your router fails you are screwed. But there is some level of protection that is peering.
So or, you have some added layer of redundancy provided by the exchange point like in the case of Linx and Netnod on the you guys connect to multiple exchanges. This is the fact that I am truly interested in.
So, what's actually driving this? What are you guys doing? What's driving the cost here is that the number of ports on your routers of course is the number of ports on the exchange because you have to pay us normally per port because we have to pay the vendors by port. The more ports we are forced to buy of course the higher your price will be. And you guys are cheap so you want us to make this as cost effective as possible. And we try. By the way. So, a simple factor of this is that we as exchange point operators could say this is not our problem to solve. You guys worry about this. If you want truly want redundancy, you pay for it. We can't force you so this is not the Netnod model. This is the Linx model. We provide this for to you buy it, but as none of you are buying it we'll probably cancel it anyway.
Or, we can do the AMS?IX model where they provide protection for you, that is in one ways just shifting the single point of failure to you guys away from us and you can't blame us when things fails, which might be a good trade off. I don't know.
And the thing is that of course one model is cheaper than the other one and the other draw back is that you guys, if you look at Netnod we see the same Mac address on both switches, or quite a few providers. So you guys have been given a tool for redundancy that you are not using. And how you manage to get the same Mac addresses in two switches I am still a bit curious about, but anyway.
Very few of you will actually go to the effort and believe that a single exchange point is so valuable that you'll put in two routers against it. This is actually the harder the presentation because I don't actually understand that. If you really truly believe that this is very important traffic for you, and you keep telling me that, then what's the level of protection are you actually expecting here? I know all the exchanges do a really good job, we never fail, we never go down and we forward every single packet you send to us, but still, what's the true level of redundancy you guys want? And that's what I am after really with this presentation, because if for a while ignore this notion that you as providers should have the right to choose for yourself, and we assume that what you are trying to do is that in the perfect world, and this is what people have told me, in the perfect world we have multiple exchanges in Europe and redundancy is achieved by you guys going to all of and if it fails you'll move your traffic to another part of Europe. So, I don't understand them. Because you introduced round trip times. You'll probably get jitter and fail in a moment and let's face it, BGP convergence isn't exactly the fastest protection on earth, not the best one either by the way. People tell me that those end users actually don't notice this and this is actually what I asked Randy is do they have any data on this? Because, for me, who has worked a long time in the industry and actually followed some of the BGP development, I just find it mind boggling that a failure across Europe an BGP wouldn't be noticed by end users. I just don't believe that. People seem to tell me that they don't notice it and that might say more been end user than about the ISPs, but people are so used to seeing a reload anyway they just continue with. But I really would like someone to do data on this. Because is it really truly the case that we can replace redundancy locally with redundancy globally and people wouldn't notice? I don't know. This is a question. I am curious to get feedback on what people think.
So how much do you really care about this? About this redundancy? Because, the reason I bring this up, where we came from was that we had a discussion among the membership that we are going to drop this requirement for two switches and I don't know, but I thought that everybody would say yeah, just drop it. We'll save money. But half the membership came back and said no, are you crazy? That's the only redundancy they have. We only peer here. If you drop this, we have no other protection. So people do seem to think that peering is important. But, some people seem to believe that they can do redundancy across Europe. It doesn't matter if my peer is in Gibraltar fails I send him to loo loo instead. And you know there a few thousand kilometres in between and I just have a very hard time believing that that's an acceptable fallover scenario. So, what I am asking for is what is an acceptable fallback scenario? What do you believe is useful? And how do you regard peering protection as opposed to using transit for protection?
I actually think this is an interesting topic and I would like to do is get some feedback from you, anonymous, private, at the microphones, and I want to collect this if people think this is an interesting topic and I'll even try and take Randy into a dark corner around force him to provide some grad students for me to actually look at this, or anyone else, to try and get some real BGP data on this feedback finally.
So... please, James.
JAMES BLESSING: There is different networks will handle this in different ways, because as you say, some people only have one peering point that they are connected to. Other networks are connected to 20 or 30. So moving traffic for them is much easier. The other issue you have got, that is how long ASN Internet Exchange actually going to be down for? If it's going to be down for a couple of hours, moving it on to transit temporarily is not the end of the world. In fact, if you are building 95% I'll, it doesn't touch it.
CHAIR: But the actual outage events still effects the end user experience in the moment of convergence?
AUDIENCE SPEAKER: Yes.
KURTIS LINDQVIST: And there is an old study from '99 that says the BGP convergence for iBGP was four minutes at best or something.
AUDIENCE SPEAKER: But you can have that some convergence issues because something else has happened. It doesn't have to be an exchange going down.
KURTIS LINDQVIST: Fair point.
AUDIENCE SPEAKER: One of the things about connecting to ARIN Internet Exchange is that, as a member, you have to have certain things to connect to and so many people now connect over pseudo wires and, you know, DWM systems and things like that. It's kind of a question for you: Do you think that hurts Netnod's position because you require people to get two ports? People less interested in connecting to Netnod because they would need to buy two sets of transmission capacity and if they didn't have a router in Stockholm?
KURTIS LINDQVIST: To be honest, I don't know. The weird thing is that we get the feedback that Netnod is more expensive and the reason ?? if you actually look at it per port we are one of the cheapest in Europe. Of course we are more expensive and then I do this survey and people say oh God no we really want those two ports. In a way I would agree with you. I would have thought that if you would have asked me I would have said absolutely we see so little people, but that doesn't seem to be the case and that surprised me a bit.
AUDIENCE SPEAKER: Is the problem here you are asking resisting members than the people who have your policy? This is somewhere smaller more developing exchange points they have a set of people and have a set of rules ??
KURTIS LINDQVIST: For the statistics you are right. But I did actually ask potential members, and got that similar feedback as well. Maybe it's not the same levels but.
AUDIENCE SPEAKER: Maxim, first, thank you for the advertisement of ?? I believe I cannot make it better.
KURTIS LINDQVIST: You don't ?? anyway ??
AUDIENCE SPEAKER: Second thing, it doesn't provide pick?up. We provide pick?up for only 10 G ports and despite this, we have still customers that connect on two sites, so, for example, we have 10x10 G port and customers have the same 10x10 on two separate sites. So I think it's just a good way to view the network, do not trust exchange.
KURTIS LINDQVIST: I would agree with you, but anyway...
DAVE: So, is the question really still, you know, peering versus transit for redundancy? You know, when we are talking about traffic volumes that we have today and you talk about, you know, the commits that people have to sign in order to get the transit compass that they may need in order to fail over from peering, is that really viable at the volumes that we're talking about nowadays?
KURTIS LINDQVIST: I wasn't ?? I didn't actually want to split necessarily peering versus transit. I was more thinking peering to peering. In general, how do you actually do your redundancy? What's the planning for doing this? I wasn't necessarily concerned about peering switching to transit.
AUDIENCE SPEAKER: Right. And I am not going to speak to that because I am not a member of your exchange, but I think the larger question when you ask the question is more about are people going to other exchanges for their redundancy? You know, do they split in London and Amsterdam versus going to Sweden because, you know, in theory they are on the same exchange versus being able to split not only their traffic but their business relationship
KURTIS LINDQVIST: So we actually had a lot of respondents who are connected to both Linx and to AMS?IX and to us and they still don't want one port on that node. People who have it probably will build' doesn't network and therefore want to trust one exchange. But, absolutely. People do go to multiple exchanges but again my question was: Is the failure between those two exchanges, is that really truly redundancy? Because between Stockholm and London, I think there is 20 or 30 milliseconds difference just in latency, I would think if I built a network, and I used to, that's a bit for my traffic volumes and you probably have more volume than most of us. I don't know the answer to this that's why I am asking people.
PATRIK: First of all, you asked does BGP convergence hurt user experience, the answer is yes ??
Patrick Gilmore: Yes, we have real data that shows this. So the question is if Netnod fails and we move over to AMS?IX that's 30 milliseconds and ?? we move over to Netnod B. Obviously the answer is yes. But at the beginning of this, you still have a failure even going between two sites in the same building. So, you are going to have some user experience problems. And the question is, how much is it worth to not pull back to 30 milliseconds later?
During the last session on jumbo frames, you stood at that microphone over there and you said we like to let our users decide. We don't want to tell them how to run their networks. We don't want to make the decisions for them. Yet somehow you have guaranteed me that I am going to pay a higher price if I need a 10?gig port I am going to buy two if I like it or not but you are going to tell me those two or cheaper than one and don't anybody mention Quickster.
KURTIS LINDQVIST: As I said, that's what I thought this would be a no?brainer question. Everybody said yes, yet half the people said no, we want to keep it this way. That's why I am standing here. Because I was shocked. I would be happy to drop the other requirement but a lot of people wasn't that's what surprised me a bit.
AUDIENCE SPEAKER: Let's be clear, the Netnod membership, they are unusual is the best way I can put it. There is no other exchange that actually posted individual member ports to the world. Not even in a member protected location, right?
KURTIS LINDQVIST: There is quite a few actually, but anyway.
AUDIENCE SPEAKER: There is no other major exchange, nothing with traffic that anybody here is really aware of, right. And they have many other interesting things about them. I mean, guys are in a bunker, right, you are not allowed to see the switch. You have to buy fibre to it which does actually change the price even though you have done a job about getting that down, etc., etc.. it's just an unusual place and maybe that's the way life is. But if you honestly believe that you should put the decision in the hands of the membership, then just have two switches and let them decide.
KURTIS LINDQVIST: That works for me.
CHAIR: Okay. Martin ??
AUDIENCE SPEAKER: Martin Leavy, Hurricane Electric. The analogy to jumbo frames is different in the same way that you can relate a budget airline to a high end airline. You choose to connect to Netnod with two ports because you know everybody else is connected with two ports. That's part of the feature set of what you are doing there. Netnod in fact has competitors, they very small, in Stockholm, which only run one port. So, there is a budget airline and there is a full service airline approach and that is a different conversation than the technical issue of what the packet size should be, which was the previous discussion. Sorry.
CHAIR: John and mike, and we have to move on.
AUDIENCE SPEAKER: John from ?? I'll try to be very brief. I hate the analogy to airlines by the way, I think we should go no further than that. I am a Ryanair fan and proud of it.
So, the two quick points I wanted to make is, first of all, inspired by what you said about only seeing one Mac address and I think actually will and maxim already made this point for me. What we see amongst UK members of Linx that they like to have diversity in the data centres because Linx is present in many data centres they get a choice and they like to split their traffic across those because and we probably could get you some data on that if you are interested. By the way I think it is a valuable thing to study so I applaud that.
And then my other point was there is actually a certain amount of data on what happens in the event of major outages and again I have to confess, we did have a few of those words to the turn of last year. So, if you wanted to look at that data, then that might tell you something and we still have the historical records of that that may contribute.
KURTIS LINDQVIST: Thank you.
AUDIENCE SPEAKER: Mike Hughes and I used to run an exchange point, I don't at the moment. Thanks very much for doing this Kurtis, it's something that I think amongst the operators we have been carping on about for a long time is what redundancy is right and what people want and we all have different models. You mentioned about seeing the same Mac address on both of your switches in Stockholm. The interesting thing is how many times the same Mac address appears on multiple exchanges in Europe. This is an indication of how much some providers care about redundancy, which is not a lot, as far as I can tell. Just because of the amount of times the same Mac address will appear, not just on exchanges in city A where this overworked router is, but how many times it will appear in city, B, C, D, E, F and somewhere in Australia if you are really lucky. That is an illustration of some people's attitude to redundancy which really sucks. Thanks very much. It's a very pertinent presentation.
PATRIK FALSTROM: Cisco. So one of the things that I also would like to know and I think we should continue with this discussion and one of the things that I also think we should try to talk about a little more explicitly is what kind of problem we are trying to solve and what are we afraid of? What breaks.
KURTIS LINDQVIST: Yeah, good point.
CHAIR: Okay. Thank you everybody. That was really interesting.
(Applause)
CHAIR: Next up we have Joao Damas who is going to talk about a routing project.
JOAO DAMAS: I am going to talk about a little bit shortly, about open source routing forum, which is a new ISC activity, and what this is all about. Well, it's about providing better support and development support for open source routing engines. We are going to concentrate not on the four writing plane but the ?? ASP F, and BGP and so on for v4 and v6 obviously. And we are going to concentrate on a specific piece of software that you are familiar with, which is Quagga, and well, there is something about zebras and Quagga and its processor, we couldn't pick which one so we coloured it.
What's the whole point of this? Well, some people have come to us and seen how we handle open source in other things and think that we have a good model to make things happen, to make things progress, and there is a clear desire out there to have an open source supported solution for having routing protocols in community hardware, and Quagga is good, but has suffered from a little bit of lack of coordination and continued support.
So, these people did ask us to put together something that would provide an infrastructure, a framework where Quagga could live. The idea is not that we take over Quagga, not at all. What we are going to do basically is concentrate in a few areas to begin with, in providing a stable and open testing platform so that real testing can be done on Quagga and all its multiple branches and stuff that's out there so that it can be known to work well.
We have been asked to work on release management, basically looking at the branches that everyone out there is contributing and all the patches and stuff and to turn it into a package that anyone can use rather than have to figure it out for themselves. So that would make things more stable as well as reduce in the engineering costs on the side of whoever wants to use this stuff.
An eventually provide some development resources for fixing bugs or develop and integrate other features.
Basically, that's it. It's a project that got started with seed funding from Google and we are going to take it from there. We have two people, at least two people, working on it. One from a project management point of view to coordinate the whole assembly and one person actually looking at open tickets that have not been addressed, bugs that ?? fixes that need to be integrated, that kind of thing. What I put up there is the website, so if you want to have more information about it. I invite you to go there. That's all I have to say.
CHAIR: Thank you. Any questions for Joao?
We now have two presentations which follow with slightly different comments about proxy up on Internet Exchange. I might propose that Wolfgang from DE?CIX begins and went they have maxim up on the stage and we'll take questions after both of your presentations. First of all, over to Wolfgang. Thank you.
WOLFGANG TREMELL: Thank you, I am Wolfgang from DE?CIX and at DE?CIX I run customer support and I would like to talk about ?? I would like to talk about an incident we had a few weeks ago which involved Proxy?Arp, what things ?? basically it went wrong, what can go wrong. It took us quite a time to figure out what actually happened, and I also show you what lessons we have learned to prevent that in the future.
I am naming this Proxy?Arp considered harmful for some historical reasons.
Yes, what happened? This is the DE?CIX slide. We have a /22 here, we have customers in all, if you see the /22 as a collection of /24s, we have customers in all parts of this, and suddenly someone announces on global BGP a more specific of our peering LAN. I'm not telling you who it was. The people who followed this incident know it and if you have a look in the historical BGP data can easily find it out.
Most of our customers actually had proper filtering and this is a thing you all should have if you are connected to an exchange. Make sure that nobody announces you the peering LANs you are connected to the global BGP filter out. It's quite easy have ?? if you have a filtering anyway, put the peering LANs into it but quite a few customers didn't have that.
So now what happens? These two guys you see here at the top, they accepted the more specific route and they suddenly have a route to the outside, which is a more specific than the peering LAN they are connected to. And one of them has Proxy?Arp enabled. This one doesn't have Proxy?Arp. No danger here. This one has Proxy?Arp and says oh, I have a better route to this network, send all traffic to me, and actually they were there were six customers with Proxy?Arp enabled who also accepted the more specific route. And this is why it took I say quite a time to figure out what actually happens if it was just one, okay, one bad guy, just shut him down, and we see, six bad guys, that can't be the case, we can't shut them all down but at the end we did, and that resolved the issue but it took us quite sometime. We had to shut down six customers, all who were Proxy?Arp enabled and all who accepted that more specific route. In the meantime we were trying to contact the operator who announced the network, who was in Asia, and who perhaps didn't have a clue what he actually did.
So, if you look back, Proxy?Arp, it's quite an old thing. It was introduced in the eighties to solve the problem of interconnecting sub networks, and my question so the audience: Do we still need this? Is there any practical application for Proxy?Arp nowadays? You say a little bit? But nowadays all hosts usually have networking, they have routing, they can route. Benedict is going to the microphone.
BENEDICT STOCKEBRAND: I am not going to talk about Proxy?Arp but proxy enabled discovery which is pretty much the same without v6 and for some ancillary reason, certain people use it for mobile IPv6, so we might actuallien can you not err it if mobile IPv6 ever really takes off. And people configure the routers wrongly and they have, how should I put it, slightly brain damage to resets for the router equipment. We might actually encounter that there as well just to make things worse.
WOLFGANG TREMELL: For v4 I don't see the reason to have it still proxy up. You can leave it in your code but please make it for default, turned off.
What lessons have we learned? Before this, we checked Proxy?Arp for new customers connecting to the exchange. If the customer had Proxy?Arp enabled we ask him switch it off and then we can connect you, but we didn't monitor for configuration changes for connected customers. Now, we do. We check for Proxy?Arp every ten minutes. If we detect someone with Proxy?Arp enabled, the first level support gets a message, customer gets contacted. If he doesn't reply, say in five to ten minutes, we shut him down and we send him hey, we have shut you down because you have Proxy?Arp enabled. As soon as he comes back and says yeah, I have disabled it, we reenable him again. It's much easier than to deal with such a massive scale of problem like we had a few weeks ago.
Thank you.
CHAIR: Thank you. If you can stay on stage. Take a seat, we'll take questions about both Proxy?Arp presentations at the same time to try and bring us back on time target again. This is Maksyu Tulyuk from AMS?IX who is going to discuss a similar topic.
MAKSYU TULYUK: I am going to speak more about Proxy?Arp, how it works. I'll speak about operational experience of what we had. How we fixed it. Why it happened and what does help us to make this less harmful for your customers.
So, firstly, I will speak about what happened, then why it happened. What we did to improve it and the last thing, what tools we use and what tools really help to us make it even worse.
So, outage:
Firstly, it was simple standard procedure. We moved customer from quarantine VLAN it peering VLAN. It happened many times so nothing special. However this time, we connected a new customers with enabled Proxy?Arp, so start replying for all requests and now you will see what engineers see on his screen. Here is our route servers, this is our out normal system. So, definitely, we started trouble shooting. There was four hypothesis. Something happened on the platform. Something happened with BGP, Cisco issue. And something happened with the route servers.
First, we quite quickly understood that everything is fine on platform, so no interfaces went down, no packet losses. You see there was only small, small drops on the graph. Only BGP sessions went down. So,...
And by our statistics, there was, we detected that 111 BGP peers went down. However, most of addresses was related to Cisco routers. So, 90% in 100 peers down were Cisco. I mean, you all vote in operations, well most of them and you understand when it happened with Cisco, it's real difficult to blame own actions, especially after RIPE experiment. However, also this action didn't happen at once, so, they come new, new, new addresses, Mac addresses, there was analysis and there was still some Cisco addresses. Suddenly there appears some addresses from another vendor, so there was start more.
The bugs, we looked on our route servers and because of our route servers based on PSD, we see message that somebody start hijacking IP addresses and all the same was from the same Mac. In the same time we received a call, with will not the same time, quite a bit later, the same call from a customer that said, hey, I see in my Mac table, this one. So all addresses are related to the same Mac addresses.
The most beautiful slide. We disconnected it and because outage was so big, we have so many calls, that it was disconnected three times, because three different engineers ran the same script to disconnection.
As I said, we have quite a lot of calls. There was 11 calls. Unfortunately because there was ?? we didn't know what is going on and if you all the time receive a call, you all the time switch contacts. So firstly you reply to the call, then you look at e?mail, then you look at platform and finally the duty engineer just forget to send network ticket.
However, what was positive from this call, customer confirmed that we disconnected the proper customer.
After this happened, because this address was famous for all platform and a lot of routers have it in his Arp table, they start sending packets ?? so switches just sending it for all ports to find out this Mac address. It was quickly detected and we just set out by Mac address from our monitor host and all traffic goes away. Nobody was affected during this moment.
However, there was still problems, because we recommended to set up Arp cache time?out for 4 hours, there was a possibility that session will be down still for 4 hours. There are two ways to fix it: Reactive approach. Which is just do nothing. Time?out will be normal after that. However it's not our way. We did it proactively. So we send spoofed Arp request and it was fixed quite quickly, let's say than 15 minutes.
So, in summary for a whole outage. If you look at this outage and look at the .1 and .5, you see that it happened for 40 minutes, it's quite a long time, and definitely customers ask why it was so long. Not only customers, also our CTO. We have a whole discussion about this.
So, now the analysis, what happened, and especially the most important how we did it to prevent it in future.
So, it happens because first, we didn't test for Proxy?Arp new customers and second, customers didn't have set up a permanent IP addresses. Also he has a default route, so, for example, if we even connect him in main VLAN and he has addresses it wouldn't be so harmful. In this case, yeah, he wasn't happy.
So, how we fixed the issue with Proxy?Arp in two ways: First way, now we test all customers, all new customers, does it have Proxy?Arp or not. And second way, we periodically check it if Proxy?Arp disabled on the current VLAN. We are not so so...ask we do it, but yes, we did it and we check it, system worked, recently we find out one customer is Proxy?Arp, he quickly fixed it, so system works.
Second: About permanent IP address. We still discussing it. Can we connect a customer from current VLAN to permanent network without setting up IP addresses because sometimes, for example, if you might upgrade of existing Linx is useful. It's still under Constitution.
And also, now a few words about why it was so long. There are three ?? four points about why it was so long. First, unfortunately a number of new customers of made a shift engineer from previous shift and engineer on the current shift didn't know that enabling was happening. Also, it was that bad that I think that they didn't send network ticket.
There was no roles between engineers, so in my opinion it wasn't good if one engineer sent a ticket and start discuss with the customer and another just deal in these tickets. And also it takes long time because we didn't see kernels.
Now, more details about that. So, what I said: Engineers from previous shift didn't inform current engineer that it happened. Yes, we fixed it procedure and now everybody talks to everybody.
On time information: I think you know that if some outage happened, you need two things: First inform customers and before inform customers, make effect at least and sometimes it take a really long time just to make this ?? because new customers came and came and came it was really difficult to understand, who is down, so inform who is current on the problem. So we did two options: First option in old notification system, we just made a change and if something happened, we can just send the tickets and say hey we have a problem. And second is, we are making new we object interface quantification system. It's under testing now.
Outage procedure: As I said, in my opinion, it's really good when one engineer do one job, another engineer do another job, knows how it happened when three engineers disconnect the customer. We now develop a more in stage discussion.
And logs: We have our route server is based on BSD and these route servers has notification that IP addresses start hijacking but it wasn't sent to our central syslog server, so the engineer didn't see this. It was quickly fixed. Easy.
So, that's one more root causes and why it happens so long. If you look one more, you think why it's only 40 minutes? Why it's so quick? Well ??
So now I will speak more about tool and procedures that we used to ?? that helped us. It's my favourite slide. This pending begin is from Madagascar cartoon, never seem alone, we have principle, never debug alone. So if you are debugging alone it looks like that, if you are in a team, it looks like that. Easy.
Central log server. Yes, we have it, we collect all statistics. We know how to analyse it. I reckon it's fine. It's helped really a lot.
Visualisation. We have visualisation tool. One of them is weather map, it's a map of our network and you will see what is loaded on each link between each router, yeah, it's quite complex, but it works, it works fine.
Realtime matrix. If you go on this website, you can see delay frame loss between any two sites of Amsterdam links and it also helps to detect package loss.
M RT G: You see that there was a packet drop, well packet drop, not so big but at least we see it's not so big directly or indirectly and we also have the same nor route servers, not so many sessions went down.
Traffic snooping: We have a monitor host that allowed us to see it all this an set up a Mac address to make a blackhole. It's also might be a really good solution. Arp responding. There is a few reasons why Arp responding work. General idea is when some host is down a lot of of routers send requests like: Hey, where is this Mac address? So immediately appears a lot of Arp requests. Arp responding just allow it. They have so when amount of question let's say 40 per second, they require this whole debt and start announcing old Mac address, so all this ?? so all host immediately receiving reply and Arp traffic went down.
Arp sponge helped us because Arp sponge has up?to?date cache. Arp cache so we have a data to send Arp spoofing requests. There is a more details about how ?? more technical detail how it works. It works ?? I put it on our website. So once more about tools that help us.
And before I comment on questions, just to make a general note. Yes, everybody makes a mistake. It's most important that we know where was our mistake? We fixed it and I believe that it will never happen again. Thank you.
CHAIR: Thanks to both of you for those presentations. I'd now like to open questions from the floor or remote participation that would be directed to either or both of the speakers.
GERT DÖRING: Gert Döring and connected to DE?CIX sand involved in debugging this.
I want ?? I don't want to ask a question but I want to remark something and that is that Proxy?Arp is actually a quite useful tool in selected circumstances. To the huge problem is that Cisco made the stupid idea is making it on by default. So if there is anything broken IOS it's Proxy?Arp on by default and that really needs to be communicated to Cisco.
CHAIR: Yes. And for everybody who is connected to an Internet Exchange point right now from Cisco, that's a really good time to remind everybody to look and make sure that no IP Proxy?Arp is in the interface standard just in case this didn't come across in the presentation or the question. Next question:
AUDIENCE SPEAKER: Alexander ?? media lines. Could you guess why such problem peers in the short period of time, at at least two exchanges? It seems we are not connected to any of you but we are connected to exchange which also seems to have such problem, but we couldn't see such report but problem has such.
WOLFGANG TREMELL: I would say it's completely unrelated.
MAKSYU TULYUK: I would say the same.
AUDIENCE SPEAKER: Why only now it problem only appears, it didn't appear two years ago.
WOLFGANG TREMELL: I don't know. Ours was remotingly triggered by somebody announcing more specific and yours was more homemade by moving the customer into the peering LAN ??
MAKSYU TULYUK: We know the problem, yes, exist. We are periodically check people for Proxy?Arp, but, yeah, this is in our case just bad sequence of events that caused such a big event. I mean if for example, we detected ?? if the duty engineer sees this request that yes IP address is ?? it will be fixed in five or ten minutes, it wasn't set up.
WOLFGANG TREMELL: Perhaps we both closed our eyes too long to Proxy?Arp and what real harm it can do.
MAKSYU TULYUK: Yeah, I agree.
AUDIENCE SPEAKER: Berth rant Viv yeah. I hear the message, I will take that back to home to the company. But bear in mind that any change in our default only come in the new release. That means it will take at least more than 12 months or 24 months, even if we ?? if we can change that, because there are maybe some router also sold in enterprise market where it may be made as well. I am not promising anything but I'm taking that back to Cisco. And at least we can do that INXR for sure.
CHAIR: Thank you. We'll make sure that's minuted.
MAKSYU TULYUK: Just to make things clear in our case it wasn't a Cisco router, it was Linux router, more, it was properly set up but it was bug in Cisco kernel. So, and it did proxy Arp on an interface where it should have been disabled, but because of this bug it wasn't.
WOLFGANG TREMELL: In our case it was six Cisco routers having the problem.
CHAIR: I love it when a plan comes together, round two. Maybe this could be solved by default in the future. Thank you both for that really interesting set of presentations.
(Applause)
Okay. Almost complete now. I think the next presentation is Bijal Shanghani who joins the community, actually been involved in the peering community for a long time, but you have a new role within the peering community and this is the first chance we have had the chance to welcome Bijal Shanghani as the second general of EURO?IX. It's good to have you with us and have an update again. Thank you.
BIJAL SHANGHANI: Hello. So, to start with with, we want to say bye?bye to Serge after ten long years at EURO?IX, he has now moved onto the RIPE NCC. We won't miss him too much because we'll still see him at the RIPE meetings.
Okay. So where are we with EURO?IX? At the moment we have got 60 affilitated members. 46 of those are in Europe. And operating 70 IXPs in 27 different countries. 14 IXPs for the rest of the world, that includes the US, Latin America and Asia.
We have 11 industry related patrons. So all IXPs are invited to join EURO?IX. We are recently introduced a remote membership scheme. What that gives the IXPs is access to the website, the databases, the mailing list, the only assumption is that they are won't be participating in the EURO?IX forums, and so therefore, the annual membership is considerably less.
So we had our latest ?? or last EURO?IX forum was in Lyon which was hosted by Lyon X and we had 42 IXPs present there. The next one is in Amsterdam and will be hosted by AMS?IX. So, if you are an IXP and you want to attend, please contact me
This is the traffic taken in the last 24 hours. So, currently, the combined EURO?IX member traffic is at 6.87 terabits.
This is the European IXP traffic and what we can see here is in the end of September 2010 to the end of October 2011 we have gone up to 7.237 terabits which is an increase of 49 percent in the last 12 months, so that's pretty significant.
Route servers. The EURO?IX started a project on route servers, it was in 2008. At the time it was I think Quagga was the only one that was available. But now we have a few more, and this is what our members are using.
We have also got tools that you can use and this is available for everyone, so, you can go to your EURO?IX.net and have a look at our tools. They are over there. What we have is the ASN database, which listed IXP participants from about 285 IXPs in all the regions. And there is also a search interface, so if you want to search for an AS number and see where they are present, you can do that. There is a filter tool. There is also a section of the most common AS numbers, so that is the, those that are at 10 or more exchanges. And also the latest entries in the ASN database can be available as an RSS feed, if you like.
So, where are we going? We have, in the ?? since 2005 we have had an increase of traffic of 12 fold. And the exponential trend line suggests that in 2015, the European IXP traffic projection will be 47 terabits, so that's something for everyone to look forward to.
What's the peering thing going to look like in 2015? The numbers in brackets are actually what's current, so, the number of IXPs is expected to be 145. There is 8,000 participants at these IXPs. 4,400 unique ASN numbers. At the moment we have got 3,329. The aggregated European IXP traffic is expected to grow to 47 terabits. Currently it's around 6.3. The average IXP peak traffic will be ?? is projected to be 360 gigs. Currently it's at 48. The average number of IXP participants is 55. And the average participant traffic is projected to 6.5 gig and that's currently at 1 gig. So that's quite a growth there as well.
And that's it. Are there any questions?
CHAIR: Thanks, any questions? Christian has one I think.
AUDIENCE SPEAKER: Christian any idea how much the percentage of IPv6 traffic will grow in these next four years?
BIJAL SHANGHANI: Good question.
AUDIENCE SPEAKER: What's the percent? Do you already have the figures of what's the percentage of the IPv6 traffic on the members is?
BIJAL SHANGHANI: That's something that we haven't got at the moment. But it's something that we are going to start collecting and looking at as well, definitely, yeah.
AUDIENCE SPEAKER: At the Vienna Internet Exchange it's below 1 per mil currently, so there is room for growth.
BIJAL SHANGHANI: Oh yes, lots of growth.
CHAIR: Okay. Any other questions? Thank you very much for the update.
(Applause)
CHAIR: By my reckoning, we now have 15 minutes remaining for Internet Exchange points who are in the room who want to come to the microphones or come to the front and share some news. Four people got in touch and said they'd absolutely love to do that. I think this is the order their e?mails came in. No? Okay. At least ?? two asked for sometime at the microphone to do this, so I'd recommend, is it easier to do the microphones here on the webcast. If you run an Internet Exchange point and you have any news to share whatsoever ?? or you can come to the front. If you want to start /ABG owe.
SPEAKER: Hi, my name is /ABG owe scare had a /TAUR house, I do this regarding NYX. I have two points to make here. First one is the we are detecting NYX to ?? NYX was not stable this year so we are studying the NYX firm from scratch including vendors directions. The second point I'd like to make is that we are expanding NYX to two new locations in New York city and I cannot give you detail but I would like to announce that. Thank you very much.
CHAIR: Thank you for the update. Please, if you have any ?? anybody else is welcome to walk up to the front. You can see ?? please come and share your news.
SPEAKER: I would like to take the ?? my name is Thomas managen and today I will have two EIX was creed in Leeds, called IX Leeds. We are now legally formed and incorporated. We have 14 members and are starting to sort all the usual first step of an IX and probably will have more report in the future.
CHAIR: Thank you. Ben, would you like to ?? if anybody has questions at any point about any of the speakers, run to the mikes in the room and ask the questions.
SPEAKER: Hi, I am Ben hedges from Linx, two quick bits of news. First as everybody is probably aware we have just finished our migration of the primary LAN over to Juniper so very pleased that's gone smoothly, everybody is very happy, so we are quite proud we managed to do that on time and on schedule. So, the other bit of news is now we have got the plan in place we are going to be launching or resale programme which we are calling connections. There is going to be a way that people can get connected to links. We have been able to do that for a long long time before through another programme but gives it a bit more commercialisation and obviously it's got the VLAN possibility as well. So if anybody is interested in finding out about that, just let me know.
CHAIR: Thank you, Ben. And we have and race from DE?CIX now.
SPEAKER: Hi everybody. Andres from DE?CIX. Just one announcement. We have a job opportunity the we are looking for a junior business developer with a strong technical background and hopefully some Russian language skills. If you know somebody or can recommend some of your colleagues, please let me know. Thanks.
CHAIR: Thank you. We now have a presentation from J P net.
SPEAKER: Hi, I am from JPNAT in Japan. I have two updates from JPNAT. One is about, in the spring we conducted an experiment about 100 giga ethernet with multiple vendors and multiple high speeds and was successful and we are preparing for a productisation.
And the other is, we are currently deployed a new route servers in Osaka, there is the secondary city in Japan so now all our customers in the JPNAT can also exchange routes with without peering associations. And the last thing, if you want this JPNAT jacket, JPNAT just talk to me later. Thank you.
CHAIR: Yes, if you have any freebies, then you are more than welcome to take the stage and give them out. Are there any more exchanges in the rooms or operators who are looking to start an exchange point in the room? Do we have anyone from Hispanics here, they got in touch asking for a brief slot. No, if there are any other exchanges or any peering networks who have an announcement, maybe the network that's going to start peering or a group of operators who are going to come together and form an Internet Exchange point and you want to raise the profile, then it's a great time to run to the mike or to the front and say something. If there is genuinely no other exchange points in the room ?? I don't believe that for a second.
Okay. Everyone is gone quiet or shy. So we can maybe bring the Working Group to a close if there are no further announcements.
A quick reminder that there is a BoF next door at a quarter to six about IPv6 privacy issues that would be very interesting. And we can finish slightly early and go to the break if you are ?? if everybody agrees. So thank you very much for coming along.
(Applause)
(Coffee break)