These are unedited transcripts and may contain errors.
Plenary Session at 2 p.m., 31st of October, 2011:
ROB BLOKZIJL: Good afternoon. I think it's about two o'clock and I think it's time to start, so if people can find seats...
Right. Welcome to the 63rd RIPE Meeting, the second one in Vienna. I think the previous one was in 1999, if I remember well. I think it's a pleasure to come back to Vienna for a RIPE Meeting, it's always a pleasure to come back to Vienna, but RIPE meetings in Vienna seems to be very, very nice.
I have a few things to say of a more technical nature, but I first want to say a few words about the programme this week. As usual, we have a very busy programme, that's nothing new, but I want to remind you that, this week, we have a couple of important items. In the first place, later this afternoon, after the break, we have a long session on digital certificates. I think it is extremely important not only for you to be present, but also to take part in the discussion there. It is something that has been presented for many years already at RIPE meetings, but this afternoon it's a little bit different, it is not going into all the technical details; it's more about the principles. And given some of the concerns raised at the last RIPE Meeting, and given the fact that it has had very healthy take up by people generating certificates, I think it is extremely important if you all give it a thought and voice your opinions this afternoon.
Secondly, this is now the almost, almost, almost last RIPE Meeting where we will still be discussing, I think, IPv4 Address Policy policies. I find this, personally, rather amusing. The smaller the amount of free pool, the heated the discussion about the last remaining few addresses, and I'm not deriding this; I think it's all very important, but, as a personal message, I would like to say: don't lose the big picture. The days of IPv4 are nearly over and gone, meaning the pool is almost empty. That goes for all of us, and whether you are a newcomer in the ISP fields in five years from now or you want to start an Internet exchange point in three years from now or you think people with green eyes have special privileges in two years from now, I think you should really give it a second thought whether we should have provisions of reserving ever smaller address blocks for ever more peculiar special reasons, and again, I'm not advising pro or con any of the more recent proposals, but I want to remind you: don't lose the big picture, and the big picture is IPv4 is over for everybody very soon now. And your future, and the future of the industry, lies with IPv6 and not with IPv4 with all sorts of fanciful clutches of magic boxes. So keep that in mind as a message for your discussions later this week, or ignore it, I don't care, but I truly believe in what I just said, that it's time to take a step back and think about the big picture.
Right, a few more technical remarks.
In the first place, we are very glad with our hosts for this meeting, the University of Vienna, and we have some of our friends from the University saying a few words after I shut up the University of Vienna and the Vienna Internet Exchange, to be more correct. Something new at this RIPE Meeting: we have a Plenary programme well, that is not new, but we have a programme committee for the Plenary programme, and that is has been new. At the last RIPE Meeting, we agreed that having a programme committee for the Plenary programme would be a good thing, and the first result is this week, later this afternoon, we will say a few more words about the programme committee, how it worked, and you, all of you, will have to do some activity this week because the Programme Committee represents you, meaning you appoint representatives of the Programme Committee. So that will come later.
We have a welcome reception tonight, which is always highly appreciated by our participants because it's the first informal meeting of the week, so tonight it starts at 7 o'clock on this level, somewhere here. It's not easy to miss it, usually. And tonight is sponsored by Google and [] next layer. If you need technical support, you have problems with your connectivity or any other technical support related to Internet issues I mean, you may breakdown, we can't give technical support for that, but all other stuff, there is a RIPE NCC technical team going around, they have blue name badges, so if you have a problem, you can find one and you can also find them via the reception desk.
Last but not least, you have received a folder with at your registration, with all sorts of information, but the most up to date information is on the meeting website, rosie.ripe.net, and there you'll find the updated versions of all the agenda and any other information regarding the meeting, social events, and so on.
And the final remark I want to make is again something of a first; this meeting, there is fussball tournament. If you don't know what fussball is, you may it as I know it, which is table football, and you find, on the first floor, near the breakfast or lunch room, a couple of these tables, you'll find people there, and, if you are interested, you can contact them and take part in the competition. There is there will be a winner and there will be a prize, the winner will get one of those tables shipped home.
That's all what is on my list. I'd like to thank you again for coming, wishing you all a very fruitful and productive meeting and a joyful meeting, we try to accommodate that last part in the evenings thanks to sponsors, and now I would like to give the floor to Christian Panigl, who heads the local host team this week, to say a few words.
Christian Panigl: Thank you, Rob. My name is Christian Panigl. I am working for the University of Vienna and it is really a pleasure, after twelve years, to have another RIPE Meeting here in Vienna. In May 1999, some of you will remember, we had a very nice meeting at the Palais here, not far from here in the Palais Auersberg. This Palais would have been much too small for this meeting, so we have more than doubled the number of participants since then, so it was a challenge to find a suitable meeting hotel to host this meeting, but I think, with the Hilton, it's a perfect place. And the people at the Hilton were really helpful to make all this happen, and I hope you enjoy the venue.
As most of you will know, the University of Vienna is has a specific role in providing services to the Internet community. One of their roles is ACOnet, which is the national research network connecting all the universities, colleges, research institutions in Austria, and ACOnet actually is providing the Internet connectivity for you for this meeting. Telekom Austria A1, Telekom Austria as they are called today, were kind enough to give us 1 gigabit ethernet link from here to the hotel to the University of Vienna where we connect the meeting directly to the ACOnet backbone, so I hope the Internet connectivity is good enough for you, and please come forward specifically if you find anomalies regarding IPv6, let us know. We never have had the chance to have so many challenging IPv6 users at one place like today, so we would be keen on learning what you find out which should be improved regarding our IPv6 connectivity. Don't be shy, just tell us.
And last but not least, another role of the University of Vienna is taking in the Internet scenery here in Austria is operating the Vienna Internet Exchange. We are doing this now since 15 years and it's a good opportunity to celebrate this with you and we have kind of hijacked the Wednesday party to let you participate in our celebration in 15 years of Vienna Internet Exchange. I hope I see many of you there, and those who can't make it, probably we have still the chance to clink glasses during the RIPE dinner on Thursday.
Well, so far, for me, and it was a real pleasure to be local host this time, not alone, but together with nic.at, we have a very good partnership with nic.at for many years. Nic.at is responsible for .at name space and other businesses and Robert Schiscka is the one who will tell you a little bit about that. Thank you and enjoy the meetings and whenever you have issues or problems, don't hesitate to either contact the RIPE opes team or my selves, me and my colleagues, who will be ready to help you out. Thank you.
(Applause).
ROBERT []: Thank you, Christian, for introducing us. I am the general manager of nic.at, not nic.80. It used to be a single product company in the past, so managing the Austrian name space is an interesting job but not fancy one, if it comes on terms of what kind of products you are offering, but things change. We have been very active in the ENUM community, the last eight years or nine years, probably you remember that we have been the first commercial ENUM service provider, but to be honest, commercial translates, we spend a lot of money and this and not so much on earning a lot of money, nevertheless I think it's an important thing, if DNS is one of your core things you have to be involved in those kind of developments and also deliver some services to some great facility to create the facility of new products.
Besides our core business, we also have been very active in other areas and we opened up our product to provide back end services for other registries, for other countries. For instance, we won a bid for running the Bahrain registry and we are currently in the phase of switching over to production system and getting live the Bahrain.ph later this year. And we have also decided to offer our services to the new TLDs which we will see the opening phase next year. Interesting times for us and things changed a lot and we also decided to run our own Anycast network and we are not only serving our own name space. Hungary is using this and a lot of the registrar has shown to be interested and the awareness in the commercial customers and the necessity to provide name services has changed over the last years, so asking people about things like Anycast five years ago outside this room, you would not get any kind of response besides very big eyes, but things changed and there is really kind of dynamic in this market. By the way, if you are interested in back end services for registry or Anycast net, please contact us. You can identify my stuff by us wearing this nice T shirts and, by the way, if Klaus would just stand up, he is a master and a genius of and if you don't have enough partying after the plenty RIPE events, please contact us, we can help along as being, we are local here in the city, we know a lot of things going on here.
That's already touched two main agenda items of RIPE meetings: T shirts and parties. Going back to T shirts, we will distribute this nice T shirt on Wednesday afternoon when the coffee break you can get one of these T shirts, and Christian already mentioned we are also happy to co host tonight's party, in a jazz club in Vienna on Wednesday evening, so, see you all there and hope you enjoy the meet in Vienna. Have a good time, and thanks.
(Applause)
ROB BLOKZIJL: Thank you, Christian and Robert, for your nice and kind words. This concludes the opening introduction. Now, I change my hat and I am chairing the first session. So there is a small change in the programme. If you look at your programme, you can see that one of the speakers in the first slot is Daniel Karrenberg with ten minutes wise words introduction on more general matters pertaining to RIPE and the RIPE NCC. Well, Daniel got caught in a broken aeroplane in Dusseldorf, but he is on his way and he will be in time for the coffee break, so his presentation will be shifted to the start of the next session straight after the coffee break. So four o'clock. And we will now continue with the first announced item, which is a report on the RIPE NCC membership survey. Is my speaker here?
DESIREE MILOSOVIC: Thank you, Rob. Good afternoon. The presentation as well as the full survey is already on the RIPE website. I believe I was here in May when the RIPE NCC services Working Group announced, as well as the RIPE NCC communication, a group that there will be a survey, and I am very pleased to say that the results and the analysis of the survey is ready and today, this is just two minutes to let you know that we have had the largest and the broadest response ever from all the RIPE NCC members survey done so far.
So, there is only a few things I'd like to share with you today. And that is that 2011 members' survey was not only a members' survey, it was also a survey that involved all the other stakeholders in the Internet community and that's why it's different, and it has been carried out by the Oxford Internet Institute, because it was felt that a third party should do analysis and that all the respondents have all the necessary privacy in all the and anonymity to send their requests. I already said we have 825 respondents, and that's a mix of RIPE NCC members and non members.
Really, the findings that have been published today and the full report is online. It does include the general findings, all the 12 questions and sub questions that have been said. However, we have included all the responses from all the survey respondents in the appendices, so you can read them, and if you do not find your response there, let me know, but they are all inclusive, so, if there is something like that.
So the key findings on the survey will be presented on Wednesday at 4 p.m. at the RIPE NCC Services Working Group and you can go online, download it and I would be available to answer all the key questions then on Wednesday as well. There will be an e mail address, I am happy to talk to anyone about the survey findings as well throughout the meeting.
That's all. Thank you very much.
(Applause)
ROB BLOKZIJL: Thank you, Desiree. I suppose people will now not now, but tonight, start reading the survey, and if they have questions or remarks, they can come to the NCC services Working Group.
Right, next on the agenda is the programme committee, and the presentation will be given by Joao Damas, the Chair of the programme committee.
JOAO DAMAS: It's just a quick update and some news so that you are aware. The programme committee has is something new, we are trying this after the last RIPE meeting and this is the first RIPE Meeting where we have been working to put together the programme. It says up there, we take care of filling the Plenary, the tutorial sessions and the BoFs. The tutorial session took place this morning. I hope everyone enjoyed that. The BoFs take place after the regular meeting hours, and the Plenary, well, we are right now in it.
The programme committee is there is one last item of content that I haven't mentioned there, which is the lightening talks. For this first one, perhaps we are not going to be using the real distribution, I think we were discussing we would have several smaller slots along the week giving opportunity for lightening talks. Unfortunately, this time, we had to place them all on the last session of the Plenary on Friday. We just opened the submission system so you can now submit lightening talks. Thank you very much for the support from the RIPE NCC and today under wood, a member of the RIPE programme committee who is more familiar with the software we are using here.
If you want to submit things well, let's start back track a bit. Lightning talks are small, 5 to 10 minute talks on specific topics that are that you think are interesting to share, but don't need like a full preparation like a normal talk would do. So it's tidbits, kind of thing. If you have things like that you would like to share, you can go here this URL, and you create an account in the system, basically that's mainly so that we have access to your e mail to get back to you and tell you whether that was acceptable or not and you you use this link in the menu up and submit your ideas. It would be better if you submitted your ideas to give them yourselves rather than put someone on the spot.
And finally, the RIPE programme committee has been made up of eight persons, four of which are appointed by the [admins]; for instance, one, myself in this case, is appointed by the RIPE Working Group Chairs and four other are elected directly by the community, that's you. We have we found four people, for victims, that couldn't run fast enough to go the first time. The four people that are in that are Daniella [Rena], who is I see back there; Todd Underwood; and Rob Evans there and Stefan and Sander Stefan, and I always will make this mistake, and so the idea is that we start rotating, or reappointing or put them out for election immediately with this first meeting, so that instead of having a set of victims, we have a set of volunteers.
So the first one will be rotating, the first position coming up, I believe is Rob's, and, of course, Rob is can run for it as well. There is a lightweight election process which is outlined there. Basically, from now, you can send mails to these address saying that you want to intend to run but send all the information that is requested, we should have a little bit of a notion that you want to say about yourself to other people, to this address, and we'll publish that on the main meeting website, and then on the Closing Plenary on Friday, everyone that's in the room gets to decide who takes Rob's position from amongst the people who want to do this job.
So a little bit of background of what you do and why this thing is put together the way it is on those URLs, so I won't bother with them. And basically, that's it, unless someone has any questions that we can try to address right now.
Okay. Then thank you very much.
(Applause)
ROB BLOKZIJL: Thank you, [Joao], and thank you, Programme Committee. Next on the programme is, last year we had the World IPv6 Day and the coming presentation on the programme is, what did we learn from that IPv6 day? And Emile Aben is going to give the presentation.
EMILE ABEN: Hello, good afternoon, I work at the RIPE NCC, and as Rob already mentioned, at RIPE 62, we presented some plans for measuring World IPv6 Day, and I am assuming everybody here knows what World IPv6 Day is was? Does everybody remember what they were doing at World IPv6 Day?
I know what I was doing: I was looking at these measurements really hard. So we presented our plans at the previous meeting, we had an IPv6 I charts and some 6to4 measurements, which are not in this talk, but I'll focus on active measurements that we have been doing. So we had a measurement network of 50 sources to 50 destinations, roughly, and, from that, we did various measurements, DNS, ping, traceroute and HTTP. So this is what our measurement network looked like. It was mainly composed of the RIPE NCC TTM measurements service and CAIDA archipelago measurement infrastructure. As you see it has a nice global spread. We have Africa, we have Dakar, actually their v6 connectivity was not good at that point but they have improved since, I have heard.
First thing: Control. So this whole World IPv6 Day thing was about making mainly we object servers dual stacks, or dual stack accessible. How do you that? You use this wonderful global infrastructure called DNS and you just announce a AAAA record and there you are, you have if you already had an A record, you are suddenly accessible over v4 and v6, assuming you did all the negotiating stuff right. One with probably with all the DNS thing, is, it is heavily cached, which is very nice, except for when it caches things you don't want to cache. Right before World IPv6 Day, we looked at the negative caching for the participants of World IPv6 Day, so this is somebody asks for a AAAA record for your host name, for how long will the answer be negatively cached? And as you can see, a lot of people had their had it at 1 hour. So people had 1 day, some even had 2 days. So, what this causes is that once you announce your AAAA records, you will not immediately see your v6 traffic flowing in but it will just trickle in, and, for some people, for instance, the two day case, it could mean that if people had just accessed your website, asked the AAAA record, the answer will get negatively cached, it was only the day after that the IPv6 would potentially have seen these records. Luckily, there is this binds all kind of things, where the default setting limits this to 3 hours, so it's not as bad as it looks here, but there is definitely a control switch there that some people didn't see or didn't use, which we actually can see in the measurements that we did on what percentage of our vantage points saw the AAAA records. What you see here is every horizontal line is a participant. The colours represent if what percentage saw AAAA, nobody saw AAAAs, it's blue; everybody saw AAAA is green, and in the middle you see World IPv6 Day, the X axis is time, and you can see in the middle lies people had their AAAAs up on World IPv6 Day, very nice, but you see a couple of issues with negative caching and, of course, also your usual TTL caching when people, or your usual resource records caching when people turn their AAAA records, when they removed it from their DNS.
So, know your on /off switch, control your DNS, and also for roll backs, these types of situations, it's good to have these low TTLs, because if things go wrong, they it will continue to go wrong.
Another thing is to test and monitor, because you don't want stuff like this to happen, so, on the day itself, on IPv4, this website was perfectly accessible, but if you happen to have a machine that was dual stacked, you actually saw the picture in the bottom left, so HTTP 404, which is like this is a layer 7 HTTP is layer 7 so there was no problem with IPv6 itself, because it can still deliver this message, but this was just somebody who didn't update their web server configuration, which is, you can see if you actually look at the right side, their website was perfectly available, so, this was something that was probably not properly tested.
Another example we encountered, this is the end of World IPv6 Day. The blue and the green are HTTP and ICMP, so this goes from having service to no service, which is indicated in this orange colour. But they still announced their AAAA records. So, somebody who has a dual stacked PC at home, goes to their website, tries the v6 first, has to wait for 20 seconds if they are not using any happy eyeballs type system and then they will start using IPv4, so this is a horrible user experience for everything, they'll have to wait 20 seconds. So, you also don't want stuff like that to go wrong.
So, lesson learned: humans are not perfect, we need to test and the more in life you test, the more likely you'll get it right. During events like this or any time, really, monitor your infrastructure and, what's also interesting that some of these people were not very reachable, as in not like network reachability but actually trying to talk to them to get them to fix the problem, which of course is avoidable. One thing you can do you is have your contact information in the RIR databases up to date of course, and you can also monitor the web, NANOG, Twitter and so on.
So, moving to a little bit more of a global view. We also looked at the performance of IPv4 and IPv6. So what we did for each source and destination pair, we looked at the measurements we had over IPv4 [](freezing here) and IPv6 during a 10 minute time frame, and just divided it into two. So, IPv4 we just compared them, like did IPv4 perform better during this time frame or IPv6? And so we stacked that up for all of the, all of our measurements during World IPv6 Day and you actually get this distribution. The good thing is it is centred around zero, which means that IPv4 and IPv6 performed have equal performance, it's the most common case that we found actually, but if you look closely, you will see that the left side of this is slightly fatter; there is more there. There is actually 62% of the histogram is to the left, so IPv4 performs better. The rest is IPv6 performs better. So there is a little difference here. The positive side of this is there is a right side, which means if you go dual stack, you have this 38 percent chance, if I can extrapolate like that of course you have this 38 percent chance that your v6 will perform better which people that don't have, don't have a dual stack, they don't have that chance. So, it is two chance for your best performance and I think this is something that realtime happens, like for instance, if you do Skype, wouldn't it be nice that they also checked on v6 if they might get better performance there.
So, I looked a little bit into where did this performance difference come from and one thing is, of course, tunnels. Unfortunately, we didn't do any tunnel probing measurements, so I used a method to estimate how many tunnels we saw, which is just counting the number of times we saw tunnel in a host name. I mean, it's not very sophisticated, or anything, but it gives you an idea, and it's about 9% of the path that we measured. And to give a little bit of a historic perspective, we actually within the TTM mesh, we do a tunnel discovery and we have a time series of that going back from 2000 to 2004, and, at that point, we saw about 50%, and now, 2011, we see less than 2% of paths. So this problem is disappearing, at least we have all the indications from the TTM mesh that this is disappearing, so that's good.
Another thing that we looked at is just looking at our trace route data and just look at the differences between v4 and v6 or if you have a similar path [](freezing here again), is there anything causing differences? And this is actually an example: I have a trace route of the University of Greece to the University of Los Angeles, where the path are really the same. If you look at the host names for these two paths, which I don't show here, it really goes from Greece to I think it actually goes through Vienna and it hits the US in the same points, and even the I should mention the graphs the different networks differently, of course. The green network, for everybody who is not colour blind, you see in HOPS 9 and 11, you see it has something that responds very slowly or something. It does it equally in v4 and v6. So from this, you see no difference if the path is the same and from this trace route data, this is the best example. There is a couple more examples of this and the opposite, when you see large differences, it's usually because there is a different AS path or a different the v4 packets travel through through different organisations than the v4 packets.
So, another thing that we saw was partial reachability. Like everybody knows, this Internet is a collection of of interconnecting networks and of course the interconnects can be different in v4 and v6, so which causes different path in IPv4 and IPv6, but something worse than that is that it can cause some of our vantage points not to reach some of the destinations at all. So, in v4, if we could go from A to B, in v6 we could not go from A to B, which is pretty bad, of course. So our first question is, are these vantage points representative, if we think they were pretty good locations, but we are working on improving this, and the RIPE Atlas project, which I hope you have all heard about and hope you are all excited about hearing more about it, which will happen during this week, that's something that will help with that.
But what we actually encountered is the partitioning of network in v6, and I have a couple of examples here. There is a Wikipedia article on this. I gave the same presentation in APNIC two months ago and not soon or quite soon after that, one of these got fixed, so I think having this information or stressing this information publicly is helping with fixing problems here. So...
So another thing we looked at is the Alexa top websites for it's interesting to see what the long term effect of the World IPv6 Day on the contents side of things. So, the red line is the complete Alexa list, the complete 1 million, and the green line is the 25,000, and what you can actually see is, on World IPv6 Day, we saw a nice peak of 3.8%, and the day itself that colour is not really readable, my apologies; before, we saw 0.4 percent and it jumped to 0.7 percent after the day. So the day itself had an effect of, if you do some creative accounting, of doubling, doubling the percentage of websites that had a AAAA records.
The strange thing is, roughly a month later, we saw another big jump of 0.3 big 0.3%, which is something I looked into this data, it is something in Germany, it involved two large hosting providers there and it involved lots of .de domains, but I haven't been able to pinpoint this and nobody has been able to pinpoint this for me, so I don't know what it is, if it was caused by v6 day or not, but apparent lots of people have confidence that this is the right time to enable it, so they did.
So, another thing we looked at is the long term effects on the autonomous systems: How many autonomous systems do announce an IPv6 prefix out of the pool of total ASs? And as you can see from April [](freezing here again) to World IPv6 Day, we actually saw a nice increase from 9.5% to over 11% and it was quite a steep growth. If you look at the top graph here. After that, there is still a nice growth, but it slowed down a little bit. So, what we can speculate what causes this difference. One thing could be that people people pushed their deployments forward, they wanted to be ready for World IPv6 Day and got the budget and everything right to do that. So, there is less networks left that can really deploy. It could also just be caused by simple things like a summer vacation that came right after it, or even the economy. We have this type of data available at this URL. It's at the bottom of this page, on a monthly basis on a per country basis, so if you are interested in comparing comparing your own country to your neighbouring country and telling your peers, your neighbouring country how well you are doing or then you can use that.
So, that's these are the main things that we found in the data that we analysed. So, in summary, v6 dual stack just worked fine, but you should make sure that you test and monitor it and that your network can reach all the other networks, but it's no different than v4, really. So, dual stack is two chances for your best performance, and, in general, events like this, like the World IPv6 Day itself, it works, it raises awareness, it gives people a target to work words to, and I think as a community and as a RIPE NCC, we are ready for a next World IPv6 Day, be it week, month, year or just leave it on.
More information: We have the website where we had our measurements, an interface of measurements up. This is still up. Analysis on RIPE labs about this, if you want to know more about this and we have the raw data available for this, so if you want to do some data digging yourself, it's available. So that's it from me. If people have questions, I'd be glad to take them.
AUDIENCE SPEAKER: I have a question on slide number 18. This percentage, did you more a comment: In ACOnet, we have some customers that have universities mainly, that have IPv4 addresses from the earlier registration space, so they do have their own address number and IPv6 space. In IPv6, we use our provider aggregated address space to will never show up in the global routing table with their own AS and their own IPv6 address space. Have you also thought about that and how do you expect this curve to grow?
EMILE ABEN: What we are looking at with this curve is for growth per se, and we realise this, this will never get to eventually, it will get, somehow, get to 100% if people switch off v4, of course. But, yes, we know there are networks like that. There is also, if you look at this data, there is v6 only networks. But this is all looking at the growth, which is, I think is the important thing here, and hopefully seeing it accelerate. And what I think this shows is that, before v6 day, there was quite a growth and after it slowed down a little bit, so we are hoping it's going to pick up again soon.
AUDIENCE SPEAKER: Geoff Huston from APNIC. I'll follow up on what you say about AS numbers in v6. We actually have around 4,000 autonomous system numbers in the v4 routing table, but only 5,000 are transit. You see them in the interior of AS paths offering services to others. Fascinatingly, about 40% of the transit networks in the v4 system to v6. So, that 12% is actually low in some ways, because what's going on is out there at the edge, the last mile networks is not much v6 at all. But in the core of the network where transit happens, almost half of the networks in v4 that offer service, we see also originating v6. I suppose underneath that, from the transit network point of view, the story is quite an encouraging one at this point.
EMILE ABEN: Also, if you would normalise this, based on, for instance, announced v4 address space, then you would sort of like, if you announce a big if you have a large IPv4 footprint, it would weigh heavier in some count like this. It would also be a much higher percentage.
RUDIGER VOLK: Rudiger Volk, Deutsche Telekom.
First of all, it's really nice to hear that at least [](freezing here again) in the Austrian academic scenario, better aggregation is happening in IPv6. Question: Can we actually track that and actually do a little bit of publicity about it?
EMILE ABEN: The better aggregation?
AUDIENCE SPEAKER: Yes, better aggregation, and also that, well, okay, actually the scope of of active IPv6 is kind of larger than represented in the numbers you are giving. This is two times good news.
AUDIENCE SPEAKER: Todd Underwood, Google. So I think that it's true that there is probably significantly less deaggregation in v6 and I think that's true because people are actually doing traffic engineering on v6 and I think that's because people are doing almost no traffic on v6. So what I would be interested in tracking is whether people start deaggregating v6 prefixes to do standard BGP traffic engineering, multihoming pieces of them in different ways when they actually need to start doing traffic engineering on v6, so that would be worth tracking in the future.
EMILE ABEN: Okay. Any more questions, comments?
AUDIENCE SPEAKER: Ronan Mullaney from Prolexic. Is there another IPv6 day planned, and when is it?
EMILE ABEN: I am probably the wrong person to ask, but I have heard, through the grapevine, there is stuff planned and I actually know there is in the Middle East region, there is also a specific event for the Middle East planned, if I remember correctly, in February, but maybe somebody from the IPv6 day cabal can mention current plans?
SPEAKER: I can comment on this. There are some plans to have another v6 day next year, and as far as I know, later, in America, they are going to have IPv6, in February next year, I am not sure about the exact date.
EMILE ABEN: Thank you.
(Applause)
ROB BLOKZIJL: Thank you, Emile. The next speaker in this slot is Brian Nesbitt, who will give a presentation on some Cisco related experiences.
BRIAN NESBITT: So, not that I ever thought I'd wish Daniel to have a broken plane, but that means I won't be eating into any of your coffee breaks.
There was a wish for part of RIPE Plenary to involve more operational or an ongoing amount of operational information and, we have some to share, so we thought we might talk to you a bit about HEAnet, the Irish National Research and Education Network's experience with the Cisco CRS platform and the IOS XR operating system.
So, I run the network operation centre in HEAnet, and, along with my colleague, Dave Wilson, was one of the primary people for deploying the CRS platform, and these days, apparently, it is all my fault something goes wrong on it. We deployed the platform in 2008. The platform was fairly new when we bought it, very, very new indeed, and there is lots there was lots of IOS operational discussion over the years. I have never heard anyone really talk about IOS XR apart from me, so I thought this is a talk worth giving to a few people, any of you you are using software, thinking of using the software, never going to use the software and just want to know a little bit more about it.
Since 2007, the software has got better, certainly; the hardware has got better, but not everything has improved and not everything has got to the point that we were promised that it would when we first installed it all.
The important point, before I go into this talk, is that this is highlights and low lights. It's the interesting stuff. A lot of time, packets just get sent, as I was reassuring one of our clients last night, who will be watching this talk. Day to day it all works well. When I gave this presentation for the first time a number of years ago, which was one year after we deployed the software, I was informed people had used it to sell Juniper kit. This is not the reason for this talk. I would very much prefer if I was never told that story again. And we run a mixed network at this point in time. The majority of our kit is Cisco. We are running quite a bit of Juniper and there is a hodge podge of other things thrown in as people buy random routers on occasion.
So, our layer 3 network is 65 clients. We are that small island on the edge of Europe, so we are not all that big. We have about 180 ,000 end users on it. The clients are connected between 10 meg and 10 gig at this point in time, and, as far as the CRSs are concerned, it's all BGP all the time. There are two CRSs, mirrored configuration, one primary and one secondary. It is by far and away, I suspect, not the least complicated network of all these people in the room, but it's certainly not a multinational complicated MPLS network or anything crazy like that.
There are two connectivity to both routers, as resilient as possible, all that good stuff.
And the IGP that runs across that network is still a mix of OSPF and ISIS. There is no rocket science in there, despite our occasional wish to tie a rocket to the bottom but that's a whole different conversation.
The Cisco CRS 1, for those of you who have not encountered it, it comes in a variety of sizes. The 8 slot is about a half rack, the 16 slot is a full rack box. Other than the fact that it's over 2 metres tall and it needs a special plinth for the data centre floor to avoid crushing everything beneath it, there is nothing terrible remarkable about the physical installation. You move the box in your cage, you throw cables into it. Packets start flowing.
Cabled all the ports, day one, which has made a lot of things easier, but again, nothing that you wouldn't do with a similar sized box. We did have to persuade the data centres in question that really this wouldn't use up all our power an cooling budget for the entire floor and eventually we showed them some data and they believed us.
So, that's the box as modelled by Cisco. It's very nice, clean, lovely. That's the box modelled by us, still fairly nice, clean and lovely. It's not very easy to move. Data centres have small doors apparently when the box is over 2 metres high. This is something to consider. We had to move it from a data centre because of a horrible Indian burial ground issue that kept on causing our routers so die.
The hardware has, overall, been extremely reliable. In four years, we have had three faults on it, hardware wise. Two of them have been the same line card and both of those cards failed in the resilient router. The MSC did fail on one of the boxes. Again, we have good hardware support. They were swapped out within four hours and everything kept ongoing. And the resilience means that even if it happened in the primary box, we wouldn't have lost more than a BGP fail over of traffic.
Flash cards, which we will get to a little later, are a huge hardware consideration, and have caused us much pain. The boxes are scaleable, the currency ones we have are up to 640 gig. They have now released new line cards up to 140 gig per slot and Cisco appear to be doing everything they can to get you to buy these new line cards and upgrade your CRS 1 to a CRS 3. The MSC As, when we bought them they were informed, shortly after the ink was drying on the contract, that they were going end of software support next year and end of life in 2014. This did not make us very happy. We have spent the last few months getting assurance after a assurance after assurance that while they will be going end of software support in the middle of next year, it doesn't actually mean they will stop being supported in software. Honest...
All sorts of complications with chip makers, and things like that, but the MSCBs appear to be functionally the same piece of kit, just with a different processor in it, and apparently there will be no bugs that will crop up that will do with that processor.
The new 140 gig back plain cards are coming out now and there is some fabric upgrade cards. That's a process we are looking at to upgrade that to a CRS 3 and start getting line cards that will cope with 140 gig. And yes, they are really easily impress insurance people. They walk into cages. People straight out of college have been told to assay these guys who are apparently wanting to insure this big hunk of metal for over a million euro. They look up and up and up. I could tell them it was anything. They have no idea.
So that's the hardware, and we'll come back to some of the parts about the hardware in a bit. But the software, announced in 2004, available is version 2 is the first production release. It's not new any more; it's seven years of operational history. It should be a lot more stable than it is. We first installed it at version 3.5.2 in December 2007 [](freezing again). It's now available for 12,000s in the ASR 9Ks. We are currently running 3.9.2. Planning for 4.X for a bunch of reasons; the primary one is that NetFlow starts doing horrible things after 63,000 odd prefixes on software below 4.X and we are trying to decide between 4.0.4 and 4.1.2. 4.1.2 is released on the 30th of November. I am currently wondering do my engineers want to upgrade it on the 5th December. They are currently wondering if I am mad. We will figure that out. But 4.1.2 will be the latest full release coming in the end of November /early December this year.
A flash card upgrade was required from the move from 3.6.X. We had 1 gig flash cards were put in the box. They are, you know, they are standard DCMCIA, formatted flash cards. I know some of you in the room have dealt with Cisco before. We were told we needed at least a 2 gig flash cards for 3.6 and onwards sorry, for 7.9.2. How much would you pay for a 2 gig flash card, ladies and gentlemen? 1,419 euro ex VAT. We are a comparity, we pay VAT, including the two maintenance window required to remove the route processors, because you have to physically remove the route processors to remove the flash cards. And obviously many hours of engineer time to upgrade these.
We did not pay 1,419 euro ex VAT for these flash cards. We seriously considered raiding them for a few of our cameras but we felt there was bad idea. Our support people got twitchy very quickly. We did upgrade them. We put them in. There was no down time because the route processors are resilient, but it was a pain in the ass, they have been working fine since installation. We were discussing recently the upgrade to 4.0 or 4.1 and I got looked at suddenly and asked: What size of flash card do you have? 2 gig. Oh, okay, you won't be able to put three versions of IOS XR on each card. Which was the right answer, because the answer of, you have to upgrade your flash cards, would have resulted in a CRS being thrown at them. So we should be good for a while on 2 gigs. The 8 gig cards were available when we went to buy them. I could have bought a house, even at Dublin prices at the time, for the kind of money they were talking about.
IOS XR, and I meant to mention this at the beginning, the fundamental point of this is: every OS sucks. If you haven't heard the song by the Canadian group 3 dead [] trawls in a baggy, which is about, admittedly, PC operating systems, you should. It applies equally, in my opinion, to Internet operating systems. I have many problems with IOS XR. I have been doing this job for long enough to know that those many problems with IOS XR might crop up in a similar or subtly different fashion in a variety of other operating systems around the world. Before I go through any more of this, this is a very important point to remember:
I think IOS XR is a great improvement over IOS. People have occasionally looked at me strangely when I have said this, but there is a lot in there that, well, one might claim they copy some of the design principles from another router manufacturer who released their core OS a few years before theirs came out but that would clearly have to be rumour and every Cisco I have ever mentioned it to has denied it and fled the building. V4 and v6 are treated largely the same, which was great, coming from having to dual stack a network in IOS and coming into an OS which kind of went yeah, cool, give me an Internet protocol, I am good, I can take the numbers, I don't need to be hit repeatedly with a hammer in order to take 128 bit address was lovely. Commit functions are a thing of beauty and we'll go into them in a bit more later. Editable lists where admins who also do things with routers can back up v I and hack their lists on that. Route policy language, we will touch on again and they have some sane and logical configure groupings. I find it much easier to look at these days with IOS XR conflict, there are core routers much longer than all the CPE. So it's much nicer there, it's all laid out. I can look at a particular piece with one command, etc., etc..
There is no notion of configuring from console or configuring from terminal or otherwise. You are configuring the box but Cisco were intelligent enough not to hope that millions of engineers around the world were able to stop hitting space and then 'T' after they had typed C O N F. Everything is in sections. The line in the login details are at the top, they are not buried at the bottom somewhere. Much more flexibility in defining user rights, but when I was talking to my operations engineers about this, and I was asking them for their good /bad things about it, one of them pointed out that defining user rights was in fact a bad thing because he kept on making a mistake in giving people not enough rights or too few or too many or otherwise, it was almost too much choice but I am happy with the ability to be particular about that, even if, in our operational environment, we are really quite relaxed about it. We get student interns in. On their first day, they are told they are enabled access on the entire network. The look on their face is always wonderful.
Access lists and route policies are before the protocols. This suits my taste; it may not suit yours, but they are separated out, and, again, the v6 ones are with the v4 ones rather than in a different random place in the configuration file.
Commit: You know, I would make wonderful comments about love and commitment and things like that, but commit is a fantastic operation. Once you get over the forgetting that you haven't entered commit and, therefore, why is the change you just made not working. Commit confirm is an alternative to reload an X. It's lovely. Commit comment admittedly we had grand plans of everyone using commit comment, that lasted approximately 30 seconds when everyone just committed, and, I mean, rancid sits there in the background doing its good job but we don't really commit comment any more, it just wasn't going to happen. Commit replace: Yeah, this replaces whatever bit of the configuration, well replaces the entire configuration, whatever bit of configuration you have just entered. This is tricky when what you have just entered is one interface stanza and what you have is an entire newly deployed router with a full configuration. I feel we all make mistakes when dealing with routers. This one was mine. No one in our company will ever use commit replace lightly again. I am an object lesson for this. Used properly, it's wonderful, if you think it's just actually going to replace what you think it's replacing. No, no, no everything.
So, commit brings dangers, it brings people, configuring things, hitting return, thinking that's worked and then walking away because they are used to IOS working the moment you hit 'return'. You get over that pretty quickly.
RPL, that's reasonably visible there. It is one of Dave Wilson's favourite things along with the appropriate brown paper packages etc. We don't have route maps. You have proper if, [] as if parameters, I suspect it's not very visible to people on that side of the room. It let's you, very logically, lay out what's going on. Engineers are far it's far more able to realise what's happening in the statement and indeed edit statements as required. So, it works very nicely and Dave suddenly realised I shouldn't have put something in this slide or something, no?
Apparently, we need to change this now that I have shown it to the public.
So, yeah, very logically laid out. You set your preferences, you tell it it's the end of the policy. It's very straightforward.
This is one of our customer routing. One of our client's BGP configures, so it's very straightforward. You specify the route policy for the customer. You have some, the bits highlighted in red are bits we defined elsewhere. You have a route policy for deny all out. Default originate. These things are fairly normal but what it allows us do there is then set the customer in policy, so the bits in red there are referring to the bits in red here, so you set your prefix set, which is the DIT v4 set. You set your preference, which is the 400 we list in the BGP configuration, and it all just works at that point in time. Or at least it should. So if destination is in the prefix set, then set the local preference to be the preference. Set community to be this. Job done. And even people who have, you know, if you know anything about coding languages or otherwise, this becomes very, very obvious as to what it's doing. The low MED is a policy we use to set a MED 5 on things. It is simple. And we apply it in a bunch of different places. Nothing terribly strange there but it's much nicer than our experiences with IOS.
IGP: The point here is that all of OSPF is under that section, all of ISAS section is under that section. Again not in any way similar to any other router manufacturer who may have released their code previous to this.
Therefore little differences. Things that trip you up occasionally or you wonder why you are getting a different command or otherwise. Like I said it's a world where you could assume you could be working in v4 or v6. There is couple of differences to some common commands. Show changes to the now preferred show BGP protocol, Unicast or multicast, I mean show I BGP some show BGP some should definitely return but we try and get people to use the right thing if at all possible.
How long table now updates after config changes, even without clearing sessions. Which is very nice, it means you are just doing one command rather than having to remember to step through a whole process and no policy is no routes exchange and it will throw up a warning on that.
Code evolution: Let's use evolution as a generous term here. It's a lot further along than it is in 2008 or December 2007 when we first played with the box. And in my defence we first played with the box before we had had any training. I had made my horrible mistake before I had got any official information from CISCO. 4.1.2 is launching in December. We haven't done a full version upgrade yet. They have only done point releases. The full version upgrade fills us with fear. But we don't have any choice any more. We have to do this. And it will be complicated. There is no two ways about it. Messages on upgrades are horrible. When we first did an upgrade we went what the hell is going on here? And the response was they are just spurious messages you can ignore. Area they in there then? Eh... so which ones can't we ignore? Oh you'll know them. That apparently hasn't changed very much, even though they have improved some of them.
Software maintenance upgrades, SMUs, apparently reducing upgrade needs, they will be wonderful and great. When we first installed the box the CLI was incredibly slow. It seemed to be doing a lot of things in the CPU and that was really slowing the CLI down. It's certainly better in 3.9.it in time and we are hoping that will continue to get better in version 4 and onwards.
we are hitting other NetFlow issues which are causing us to up great. NAT 64 is appearing in 4.1.2 which we want to play it, so that's one of the reasons we are thinking about waiting until that software is release and looking at it.
The bugs we have hit, my personal favourite of all of the bugs we have hit on it is adding a BGP peer could cause the entire BGP process to reload. All T this came in a shock in the middle of an afternoon. That peering session, go on...
There is still lots of might bes missing for v6, we haven't got the monitoring we would like and we are not sure at this stage why a modern operating system does not have this information. It kind of boggles the mind.
S M U reality in shocking news and this will come as stunning to all of you who have ever dealt with router manufacturer did not live up to reality. There is /SURPBL 13 SMUs out for 3.9.2. A reload will be required for this. Three of them say their hitless. And to be fair when they say they are hitless, they are almost certainly are. Those of who playing along will notice that 4 plus 3 is not equal to 13. Six of them: Who knows? You paid your money, you take your chance. Even Cisco, when I raised this with them recently went oh yeah, six of them don't say anything, yeah, what do they do? What happened when you installed them?
The H F 4 basis on H F 4, the 12,000s were the B FRs, these are the H FRs, one of the words is huge, one of the words is our. They will almost always reload the route processor. That's fine you can live with that in a resilient route processor set up. The situation is not clear so I always I /SAOUPL we are in/STAULGS an S M U where there is an interruption. This is supposed to be solving my problem. It is still far preferable to an upgrade. An upgrade takes approximately, and I think I have got it a bit further on, about four hours, including three 40 minute reloads.
I SS U is clearly better than SMUs because they have more letters in them are apparently the future. I think none of this is confidential any more. At least nobody in the meeting when they told me about it told me the /SKWREUS could he thing in as would breakthrough the window if I said something about it. There are no Windows in this room so we are doubly safe. If anyone from Cisco wants to sop me now, now is your chance, but remember mime a very big person.
May of next year that code is dropping. Upgrading the router with less than a six second outage is the marketing copy for this. So this means actual full code upgrades with less than a six second outage. There is all sorts of slides which involve one piece of hardware /SPOFG that it's doing another piece of hardware's work and all the rest it have I wasn't going to include them all here in no small part because I am not as it may become obvious, I don't work for Cisco.
There are obviously exceptions of course there are exceptions, they do look quite nice and considering the pain of an upgrade at the moment, even halving that would be a wonderful thing, so that would be lovely. One still has to do the ROM on upgrades for line cards, which will still cause the line card to have to reload, so there is no expectation of an inservice upgrade for those. But we have managed to stave off ROM on upgrades until that full point release which I was talking about.
When it all goes wrong, and it does, because it's an OS and if you remember, they all suck, trouble shooting commands seem to vary perversion, because there are so many there and because it's so distributed across its various processes, show tech just ain't enough any more, and if it was enough, show tech would probably take about half an hour to actually run, so every time we have a tack query in, we end up being sent back a list of commands we need to run and send back to them. It is still, and even though I seem to run into them randomly in pubs these days, it is still difficult to really shake the impression that there are very few people in the tack and in Cisco who have a full working familiarity with IOS XR, even after four years of us running it. It's better than it was certainly, I mean there is more than two people doing it these days, but considering how much time they put into it and how new still the code base is and how many problems there are, I was never really got that full impression that people really, that there is really enough people there to deal with it.
So, in kind of conclusion on all of that, our engineers are much more used to IOS XR. That doesn't mean they like it. On a random non scientific pole in the office, it was kind of 5050 what the racks would be if I /TPURPBD around and said right lads, we are installing dollar vendor new /PWORBGSings in here. Some of them are used to it and some of them are going no, if we got a chance to get rid of this we would. That's the truth of the matter. When the router /PWORBGS it works, when packets go, they just go. And at that client I was talking to mentioned if we didn't, they'd let us know in /TPHOPL ambiguous fashion. Our clients appear to be happy enough with the core we have in place, so, either I am exaggerating or we are doing a really good job of masking all the problems or probably both. Upgrades are not fun things, they really aren't and there is a fear surrounding upgrades, very definitely. That's not a pleasant situation to be in. The hardware upgrade path isn't straightforward. We think we have cut our way through it at this point in time. I can't tell you what it is because we haven't actually signed checks or figured out whether we have to tender. Cisco have done a lot of the work on the /TKPWRAUP grade path and we know where we have to go. Again, it's never, it's never easy with large routers like this.
When I first gave this talk in 2008 to US /TPHOF, one of my ex colleagues who now works for an IXP in the UK asked the question of do you wish you had /PWAUT a Juniper, which was a very fair question. I, at the time, (bought) was kind of going, well maybe a little, and then a little while later, a /TPREFPBD mine who runs a large insulation of Junipers sat looking at his laptop in panic as they crashed one after another due to a bug which cascaded across the entire system and I was suddenly very happy with nigh two CRS 1s, maybe we might go back and look at that tender process again, but really we have them, we are working with them and I could be standing up here an giving awe talk about how a different operating system vendor's systems are dodgy as all hell. They are far from perfect. They have many problems. They will still got a very long way to go to fix those issues and /O to get the CRS to be a very solid routing and MPLS platform, but I can't guarantee, like I said, that the thing is because every OS sucks.
So does anyone have any questions? Everyone believes me and there is no one from Cisco in the room. Excellent...
AUDIENCE SPEAKER: Sander Stefan: I was wondering, have you ever tried configuring the same IP address on two different interfaces at the same time? Because, I have actually seen a configuration be accepted on ASR 9 K. It doesn't work obviously, but Nesbitt Nesbitt I don't think we have, it's not the kind of thing we normally do on our network.
AUDIENCE SPEAKER: Just curious. Nesbitt Nesbitt some day, when evidence a couple of beers, kickback, log in...
Anything else? Thank you very much.
(Applause)
ROB BLOKZIJL: Thank you Brian. This was the last presentation of this slot. Before I let you go for a well deserved coffee break, there is one small announcement. Today the quarter to six in the room next door, the RIPE stats team will give a live demo of their work in programme, so if you are interested in measurements of data on the Internet and presenting them in a way that appears to be useful, you can go there and participate in the live demonstration. It is a live demonstration, not just you sit in the audience and there will be a live demo which, we all know is very dangerous, but the idea is that there is a lot of interaction between you as potential users of this new tool and developers, so if you are interested in that, a quarter to six in the room next door.
With that, I close this session and we continue at four o'clock after the coffee break with the postponed presentation by Daniel Karrenberg who right now sent me an SMS that he is in the taxi in Vienna.
(Coffee break)
LIVE CAPTIONING BY MARY McKEON RPR
DOYLE COURT REPORTERS LTD, DUBLIN, IRELAND.
WWW.DCR.IE
1