
Join for a lengthy discussion on Europe West‘s recent server overload and the consequent 20-win IP Boost reward! Also, check out Damiya‘s explanation on the PvP Client instability as well as the latest Champion/Skin sale!
AMA – EUW Server Issues
> Server Instability
> Player-focused
> Non-related questions
PvP Client
Champion/Skin Sale
AMA – EUW Server Issues
Rioters Demorphic, Riot tmx, sonicdeathriot, ScottThruDaHeart, Riot Pwyff and Riot Banksy discuss the recent server instability on EUW and the 20 win IP-boost compensation that’s now being sent to accounts affected by the issue.
Demorphic: Hi everyone!
I’m Matt “Demorphic” Elliot, and as promised we will be kicking off an AMA today at 6pm UK time with three members of Riot’s senior technology leadership:
Scott “Scott Thru The Heart” Gelb, Vice President of Technology Ron “sonicdeathriot” Williams, Vice President of Operations David “RiotBanksy” Banks, Technical Director.
Joining us will also be Tomasz “Riot tmx” Ankudowicz, our Live Services Producer for Europe.
We’re here to talk with you about the recent service disruptions you’ve experienced on EUW, and to answer any questions you have about how the League of Legends service operates, and why it sometimes breaks badly.
I’d humbly ask that you focus your questions on the service/technology side – the Rioters with me today are super busy working on improving the network infrastructure behind League of Legends to avoid issues like this in the future, and they don’t often get a chance to hop in and take your questions.
They will also be fielding questions from multiple European languages, so please bear with us. Realistically we will not be able to answer all questions live. However, we have commited to following up this session with a more comprehensive set of questions and answers before this coming weekend.
We’re going to be doing more deliberate community communication like this moving forward – we understand that not getting this kind of face time with the team at Riot HQ is also a big point of pain for EU players, and we hope this is a valuable step in the right direction.
Please kick things off and we will be jumping in at 6pm UK time. You then have our undivided attention for 2 hours. So fire away!
EUW Server Instability
How will you communicate with EUW players during technical issues?
Riot tmx: Beside keeping the platform in good health, communication is our main goal. It always was, but we know we didn’t really do great job here. So, about plans:
1) We’re working on standardizing Live Incident communication in all languages. These will be high quality, automated and localized announcements posted regularly during the incident and we know it’s going to work better, than our current system. We’re still finalizing the scripts.
2) We introduced new program called Post-Incident Messaging. Every time our platforms come through the issues, we’re going to tell you about it within next 24-48 hours. We will summarize our efforts, tell you what we’ve done, what kind of problems we noticed, what was the impact and how did we mitigate the incident. We will be actively answering all your questions and we really want to put our best effort into this.
3) We’re also working on improving the communication within the client, as we know that some players don’t even check forums for service status. More info coming soon.
Any plans on avoiding future EUW server overloads?

sonicdeathriot: Service quality across the many regions League of Legends runs is actually about the same when looked across a full year of data, however sometimes a certain region will have a bad run of issues that can last several days. Also, when issues happen our biggest regions do tend to have extra player impact when issues occur. This is due mainly to:
1. When a big region has an issue that kicks a lot of players out of the game or off the platform it puts a lot of stress on the systems that let players back into the system. This creates complications that lead to the need for us to limit the number of players per second we can let back into the service which means in large outages it can take several hours to let everyone back into the service. Also when this kind of load is placed on the system it can sometimes create secondary issues that will extend the unplanned downtime. There is a lot of work going on in our Operations and Development teams to greatly increase how many players per second we can let into the service and also improve the stability of the service when it is under stress when huge numbers of players are trying to log back in after an outage at peak times.
2. Our biggest regions like EUW are also our most complicated infrastructures to maintain. This means that new release can take longer to deploy, troubleshooting problems can take longer, and there are just more things that can break, be misconfigured, or have other issues. EUW and EU Nordic are both currently being rebuilt in Amsterdam on brand new servers, network, and secuirty gear (note that game servers will continue to be availble in Frankfurt with even more being added in Amsterdam). This will give players the latest architecture we have been testing for the last several months to ensure EUW and EU Nordic can continue to grow larger and have even better service stability. It is a huge undertaking but coming soon.
sonicdeathriot: I wanted to share a big wall of text for anybody who’d like to go deeper into the tech today.
This write-up describes some of the big problems that were impacting all the regions in Europe (and other platforms outside of Europe) until about mid-April (and more recently for Korea):
EU Core Switch Summary
Since late October there have been several major outages in Europe that were related to scaling problem with our core switches in Frankfurt. Our Korea players also suffered from this issue recently along with some other painful problems. Last week we finally figured out the root cause of this painful issue. The root cause around the malfunction of the core network switches was that our game servers were redundantly connected with two network interface cards (NIC) that were setup in an active/active load balancing mechanism we use in all of our environments around the world.* The network load balancing was accomplished through the use of a “smart” algorithm that dynamically moved traffic from one NIC to the other.* Since each server was connected to the network twice, every time the server flipped its active NIC, the entire Europe network would have to change its network packet forwarding tables to account for the change.* Specifically the MAC address table inside the switches needed to be updated constantly.* Typically updating a MAC address table isn’t as issue with just a few servers, but it becomes an issue as the server count increases when those severs are using “smart” load balancing. * When the game servers change their active NIC this has to be propagated across the entire network.* It starts from the top of rack switch that the server is directly connected to, sends the change to the core network switches and then is distributed out to all devices connected to the network. As network traffic increases, the servers will swap the active NIC more often to distribute the load, which causes the MAC address tables to be updated at a rate that cannot be handled by the core network switches.* In the case of our Eruope (and Korea) game servers this means the MAC address table was updating millions of times per second during peak playtimes. North America was spared this pain because we have fewer game servers on the core switches than we do in Europe (or Korea).
*
This is where the network flooding starts that would eventually crash the core switches that support the entire Eruope service. *Inside the core network switch, the MAC address table stores a MAC address mapped to a physical network port that is uses for forwarding and filtering network traffic. **As the rate of updates exceeds the capability of the switch, the switch falls back into a fail-safe mode where it sends the packets to all ports on the switch.* Traffic that is normally only sent to the destination that it addressed for, is now being broadcasted across the entire environment.* As a result all of our servers started getting lots of unneeded network traffic.** This had the net effect of essentially creating a denial of service attack across the entire network.* Welcome to meltdown city. *And when the core switches crashed they did not keep any diagnostic information we could access to help troubleshoot the problem.
**
To mitigate the above issue, we have done a few things:
1.*******Short Term Fix: *Disable secondary NICs on the game servers. The idea behind this is that if there is 1 NIC and 1 related MAC address per game server (instead of 2), this completely removes the possibility of traffic jumping around from one NIC to the other.* The network will no longer have to update itself as a result of any server side load balancing.
2.*******Long Term Fix: Change the environment to use active/passive failover on the NICs.* The server will have a backup connection but it will only use it in the case of the primary path failing.* This will stabilize the environment greatly but still allow us to provide high availability in the case of a localized failure.
*
3.*******Upgrade the core network infrastructure with hardware that can handle a huge amount of additional load.* Why not, I mean, more power is better, right?!
a. Upgrade the core switches so if we get surprised again with extra traffic for whatever reason, we are ready for it. *Different designs can keep up better with MAC address table updates which would allow us to better handle the load should something similar happen in the future.*
b. An improved design will also reduce the chance of one switch failure crashing everything in the data center.
*
Current Actions:
We have been working on a daily basis with our network switch vendor (the same one used by the leading Internet companies and other leading websites). During the outages we’ve been passing the switch vendor logs we gather as we make changes to the Europe (and Korea) network infrastructure, and have been working alongside of them to diagnose the root cause of why the core switch isn’t able to handle the rate of change to the MAC address table. * We have also been working with the vendor to expedite the new network gear to upgrade Europe’s network, gear we placed orders for several weeks ago, but due to the very big capacity of this new network gear, it is built to order by the manufacture. Some of this gear is already in installed and has helped stablize EUW and EU Nordic’s network since mid-April 2013.
We are sorry of this inconvenience, it has been a very difficult problem to track down and fix. We have learned from this issue and are building a better network to protect the quality of the service in Eruope and in our other service areas.
Will this time communication between Rioters and EUW Community be better?
Riot tmx: Since we’re in Europe and we work on LoL in 10 different languages, it’s quite demanding to support forums in all languages the way we all wish. We have our Community department, but we feel it’s not enough. Obviously we’re going to improve and we’ll eventually reach our point, but it’s quite different comparing to our US HQ, where most of Rioters speak and comment in English.
Our plan for this is to encourage the entire Dublin office to be more active on forums: to be part of discussions, to talk to you about champions, new maps and other features. Obviously we’re all swamped with our daily work, but being active on forums is another part of our work. And we want to be closer to our community, for sure. Also, we’re actively hiring more folks to our Community department in all languages, so this should be noticeable pretty soon.
Any plans on splitting the EUW servers?
ScottThruDaHeart: We currently do not have plans to split the EU-West platform. We are constantly upgrading the infrastructure and adding additional capacity to support the growth of EU-West, one of the largest League of Legends regions in the world. We recently addressed one of the technical challenges we’ve faced with the scale of our core network. Ron Williams, our VP of Operations, is going into more detail on the specific issue and how we’ve solved it in a separate post (http://euw.leagueoflegends.com/board…23700#11923700). We have engineers working around the clock to fix the stability issues and dedicated teams that are focused on scaling our systems proactively to support the live service.
Riot tmx: We don’t want to split EUW, I think this has been messaged out already. What we do instead is to improve our infrastructure even more. We already have top quality devices in place, but life brings us new solutions every month and we’re constantly upgrading our systems.
We recently performed large maintenance in the Frankfurt data center where we completely re-designed our network infrastructure or replaced all the database drives with the high-end solutions. It all gives us some additional power and if you take a look a year or two behind, we’re now able to provide service for over 100% more players comparing to the past. Also, not sure if you noticed, getting back on EUW platform is now quicker than ever, as we’re able to throttle players in much faster in the peak times or during the incidents
Will you be able to react to such problems faster?
Riot tmx: Yes, we know this can be improved on our side. First decision the live producer takes after getting alerts about serious incident is to turn on the Compensation Mode. We always do it, and, if executed properly, your loss should be prevented. If it’s not, that means we pushed the button a bit too late and that’s completely on us. We understand the importance of this compensation though and always think about your disrupted games.
Communication, troubleshooting & compensations

Riot Pwyff: While this is a very tech-focused AMA (given our subject matter experts), there was a specific wall of text that hit some good points, so we’ll try to address those. I’ll note here that this is outside of the scope of the AMA but silence can sometimes be interpreted as being ignored and that’s absolutely not the case.
Side note: Demorphic will be addressing compensation in this AMA, so that won’t be covered here. Just trying to get some other information out there.
Quote:
Originally Posted by abuzeus 
Let me start of by giving my opinion on EU forums. I don’t really care that much for red activity. Most development or policy insights are only available from the NA devs, and those don’t post here (more on that below). I also don’t particularly care whether community coordinator X likes champion Y, or whether she will play ARAMs with the community now. Those posts are nice to have though. What I care about in EU is reaction to incidents. When there is something many people care about, there should be a red response. If the servers go down, we want that to be acknowledged within an hour. Sure, there is no hope to beat the hundreds of reddit threads that will pop up, and the “better give NA RP” joke is getting super old really fast, but sometimes it (feels like it) takes two or three hours until there is something going on in the service status forums. During the incidents over the Pentecost weekend, the responses felt super delayed at times. If the server just disconnects for a minute and then there is a small queue for 20 minutes, that is maybe not something that needs to be discussed, but if this happens very frequently, or the downtime exceeds a few minutes, there should be a red post.
|
Communication on live incidents has been a pain point for you, and that’s something we’re looking to fix in the long term. Going to echo something Riot tmx mentioned in an earlier post: “We’re working on standardizing Live Incident communication in all languages. These will be high quality, automated and localized announcements posted regularly during the incident and we know it’s going to work better, than our current system. We’re still finalizing the scripts.”
So things are on the horizon to address this!
—
Quote:
Originally Posted by abuzeus 
Or take the recent discussion about Riot Korea giving out Silent Night Sona. A lot of people wanted to know what happened here. It was very obvious that the problems in EU had reached the point that some compensation would be handed out, and, in retrospect, it was also very obvious that Riot Korea caught you out cold there. However, it should not take eight or nine hours for a response to come in. When angry threads flare up before noon, then people expect an answer during normal office times, and not late in the evening. This answer does not have to be comprehensive, but a quick “I can confirm that KR gets X, we can’t say anything for EU yet, hang on” would be nice. Also keep in mind that you are already an hour behind the majority of the EU crowd. If a response is not out at 7 pm UK time, many people on the continent will be back from work and: no red response yet… Maybe another problem here is that you (Riot) refrain from posting, because you feel that a red response carries so much weight what it has to wait until everything is ironed out. I feel that this problem could be ameliorated a bit by having more “fluff” posting going on. Because then it’s “Oh, Demorphic said something about X issue” instead of “This is one of five Riot posts in the last three days. It *must* be super important and it *must* have gone through all kinds of review”. Haven’t thought this really through, though.
|
Waiting too long to respond to issues in an official capacity is a tricky subject, but on your point (that sometimes we wait too long and are too careful) is a valid one. Speeding this up and getting you the right information as it develops is something we really want to bolster going forward, although I will note we did basically say that “I can confirm that KR gets X, we can’t say anything for EU yet, hang on” on that day!
Quote:
Originally Posted by abuzeus 
By the way, I agree with the decision not to give out a limited skin (even though I don’t have it and I really like it). The IP boost is okay, although I feel that a 10 win boost and some monetary or other gesture would have been nice (maybe 200 RP, or Aatrox going on sale the first week or something). Personally, I don’t care about a small amount of RP (I can buy those, after all), I care more about the gesture of saying that the recent incident were more serious than a normal cluster of downtime with the associated 4-win IP boost. I do feel though that Riot needs to teach their departments more to think about the impact of their decisions on the other regions. Obviously, Riot KR did not think about the angry crowd popping up in Europe. Or two weeks ago, when some NA dev made a poll on what the community wanted the Madred’s proc to be, this happened only on NA. No big deal, and I think people blow this out of proportion, but still: There is this nicking feeling: Why not here too? Such polls never happen in EU (okay, it happened once, and then angry NA players forced a sister conversation on NA).
|
Also valid. I’ll add one caveat here in that our designers do tend to go “heads down” on their work, so asking them to stay cognizant of a global communication strategy in addition to their work is a tough one. That’s certainly not an excuse, however, and it’s something we’re actively working on. To give some concrete examples to that, we have people working directly with our live designers to bolster their communications, but it’s an ongoing process.
Quote:
Originally Posted by abuzeus 
A thing that is being brought up all the time is that most dev interaction is centralized on NA. Some devs, e.g., Morello, have explicitly said that they will only read NA General Discussion. While it certainly is annoying, I can understand that most devs can not handle the logistics of keeping multiple conversations over multiple boards. Since everyone is free to post on the NA boards (and Riot is not being overly zealous with the inactivity purges), I think it’s okay the way it is. I think that it’s a laudable effort of Xelnath to talk about the Xerath changes on multiple boards, but I can see that this won’t work for the high-level devs that have to talk about a lot of stuff. For a while, we had FeralPony and Meddler as the dedicated EU devs, but that seems to have died down to intermittent “batch mode” posting. Still, I can live with the current solution. It’s surprisingly easy to get something out of Morello when you have the right timing.
|
As you’ve noted, bandwidth is a huge issue for key designers that you want to talk to and I understand there are frustrations with the situation being something like “go to Twitter to talk to Morello!” or “Time your windows with Xypherous being on the NA forums to chat with him!” and so forth. That said, this is such a broad issue that it can be difficult to mandate a single solution without completely burning everyone out (IE: Posting schedules, etc). A very small part of me wants to say that communication is a two way street, so making an effort to go to someone is part of it, but sometimes that’s not a great experience for you and I get that. What I can say on this front is that it’s something we’re aware of, but the issue is fairly deep. Not a great answer here but we’d like to work with you to get there.
Quote:
Originally Posted by abuzeus 
Do you agree with my assessment that EU summoners can sometimes feel a bit behind when seeing all those things play out against them (even though none of the decisions and issues behind these things are *meant* to screw EU)? If yes, can you think of ways to give *us* something cool, given that changing most of the things I outlined above seems either pointless or useless, seen from a cost/benefit point of view?
|
This is a super leading question! Being very careful here, I would just say that making you feel unique is far more important than making you feel like you’re ahead or behind any other region. It’s on that note that I won’t try to debate you on points that you feel EUW was left behind because that will inevitably lead me to draw comparisons to other regions and so forth. As I said, being careful here because this is drawing us to some odd conclusions in the first place, but wanted to at least throw out something vague so it doesn’t look like it’s been deliberately skipped.
Quote:
Originally Posted by abuzeus 
Are we going to be in 2011 again? The recent server pains and the increasing frequency of server hiccups makes me fear that there will be capacity issues again. Are there? (Of course it’s not nearly as bad as in 2011. Yet?)
|
There are some huge plans (including a move to a brand new data center that can handle some fairly staggering capacity loads) in the works, but there will always be hiccups in transition. Just know that we’re consistently pushing to improve our service.
How to resolve lag
sonicdeathriot: There are a few things players can do to minimize the chance of lag:
1. Playing through wireless connections increase the chance of running into issues that can create lag. If you have to play through wireless, try setting your wifi router to a different channel. Most wifi routers sold ship on the same wireless channel, so if you around other people using wifi it can create interference that will affect the speed of your wifi connection from time to time.
2. Ensure your PC’s network is dedicated as much as possible to the game. If you are running other programs that use the network like music streaming, video streaming, file downloads, etc. these could be overwhelming your Internet connection for short periods of time.
3. Make sure you are seeing lag, video card framerate dropping can appear like lag. Check the FPS counter in the upper right corner of the game screen. If it drops below 20 you could see symptoms that look like lag but are actually related to something overloading your PC’s CPU, memory, or disk resources. Shut down other things running in the background and if you are constantly seeing low FPS numbers then you might want to consider upgrading your video card or other PC resources.
Can the players help in any way to resolve technical issues?
Riot tmx: Most of the issues are on our side and we’re able to track them down relatively quickly. But sometimes we do ask for Bug Splat logs, tracert/ping details, screenshots, champion bug videos or r3d logs. These might actually be very helpful in the investigation or at least speed things up.
Riot Banksy: When issues arise, players can always contribute to helping Riot debug.
- Providing detailed information about what happened from the player’s perspective
- Including screenshots when appropriate
- Posting logs from the session
__________________________________________________
Instructions on how to obtain a DxDiag file
Where to find your game logs
Player-focused
20-win IP boost to compensate
Demorphic: I want to address the fat elephant in the room: the 20 win IP boost we’re delivering as compensation. We will be going into more details about this during the AMA.
First up though:
We’ve started running the script to add the 20 Win IP Boost to your accounts, but it might take up to 3 days to complete, just because of the large number of players on EUW. Thanks for your patience!
How do you know which players to reward?
Riot tmx: We’re giving this compensation to players who were affected by our poor service. In other words, if you tried to log-in during this period at least once, you’re on our list and you will be compensated. Other story is if you tried to log-in and were unsuccessful for full two weeks. It should not happen, but if you are one of these unlucky players, please contact our Player Support and we will get back to you.
Why IP and not RP for compensation?
Riot Pwyff: I suspect we’ll be answering this multiple times down the road but to reiterate:
We’ve never given RP as a widespread compensation for server instability. There was one case where less than 1% of NA players were unable to access their purchased content for more than a week and we compensated them with RP. To reiterate: we have never given out RP for any server instability in any region.
Followup
Quote:
Originally Posted by Spooned 
Yeh, that’s not true at all. As far as I remember, there’s been given out RP two times since ‘League of Legends’ launched (one for 350RP and another for 550RP). I’m sure of the 550RP one, and I even found a link for you: Post by Tryndamere. I’m sure you’re in a better position to inquire for the other.
This was in direct relation to server instability as well, so your statement is completely wrong.
|
Riot Pwyff: AM CAUGHT. In this regard I’d rather not argue technicalities (it’s DDoS attacks!), so I’ll just go ahead and eat this humble pie.
Apologies, this is actually a demonstrable case where we’ve given out RP for this specific type of server instability. On that note, giving out RP to our players has problems outside of cost (of which IP boosts actually have a monetary cost as well), so this isn’t a simple case of greed.
Final note, and not meaning to be too jovial here, but I find it rather ironic that the one demonstrable case of giving out RP for server instability was… Europe. Sorry. HAD TO POINT IT OUT.
Demorphic: Moving forward, compensation is a concept we’re going to have to reevaluate. It’s clear that IP boosts arent good for everyone anymore. Also legacy skins aren’t realistic for established platforms. This causes as much pain as it solves.
Should we have service disruption in the future, we’d like to be able to offer an apology that’s satisfactory to as many players as possible. We’ve heard your feedback loud and clear.
All I can do is promise that we will be smarter about communication both internally and here with you guys when it comes to compensation.
We get you if you don’t accept this apology, but we still hope to earn back your trust in the future.
I still can’t connect, what should I do?

Riot tmx: Yes, we’re still working on this issue and I’m sad seeing some players still having connection error. Our recent fix was based on the network configuration changes and we know it finally worked for majority of players. Connection Error does not happen to players facing this issue week ago. If you still can’t connect and it lasts for days, please contact EU Player Support and ask them to forward your e-mail address over to me. I’ll work with you directly to get it resolved. We did it in the past with some other players.
What about LP lost during server instability?
Riot Pwyff: On that front, because it’s just so difficult to clearly prove a loss due to server issues (and at what point we apply a cut-off point) that retroactively compensating LP is nearly impossible for the entire player base. We do turn on loss forgiven mode as soon as we know issues arise, but there can be delays and we apologize for that.
Is the PBE involved in the technical issues lately?

Riot tmx: We do use PBE for testing our deploys, patches, hotfixes, etc. It’s in constant use for client-based issues and new releases are always first deployed to the test server and many bugs are caught this way. It does not cover all the re-configurations (this word is unfortunate, but I’m talking about all platform changes, firewall policies, drives, front and backend applications, database modifications, Coherence tweaks, etc) we perform on our network and system infrastructure. On the other hand, PBE lives in our datacenter and is hosted from some of our standard game servers, so in fact it is a part of our bug investigation.
Non-related questions
Why was the decision to split European servers made?

sonicdeathriot: EUW and EU Nordic do not share the same servers. They do share certain network switch infrastrcuture today. We are currently completely seperating EUW and EU Nordic from any shared network gear to prevent a network gear issue from impacting both regions at the same time. It will always be possible that Internet or Data Center facilities that suport EUW or EU Nordic could still create issues that affect both services.
What happened to the Custom Sets tab?

Riot Banksy: Soon after the 3.7 release, we started to see potentially similar service disruption issues globally. We quickly ascertained that the root of the issue was network related and wanted to quickly nullify any variables that may be causing the issue. As a result, we decided to revert the 3.7 client to remove the possibility that we were in fact creating a self-inflicted problem. In the roll back to an older client, we removed Custom Item Sets. This feature will be returning in an upcoming release.
How was the server issue in Korea different from the one in Europe?
sonicdeathriot: I just posted a long read on the core switch issue that has occasionaly affected EUW and other regions over the last 6 months. We upgraded the Europe based networks in April to eliminate this problem (we did not know root cause at that time but we thought we would try a different architecture with even larger capacity equipment to see if we could make it go away, and it went away).
Once we saw Europe improve we ordered more gear for Europe and gear for Korea. As mentioned in my long read, this gear takes a long time to get from the manufacture as it is made to order (it is too expensive for them to keep it sitting on the shelf waiting for someone to order and it is customized for each customer’s needs). So around the first week of May we ordered for Korea but knew we still had the problem to deal with until we could get the new gear shipped into Korea and installed. So we continued to work on root cause of the problem in Korea with the manufacture of the core switches. Wednesday the 22nd of May we pushed a change to the Korea core switches in an attempt to address what the manufacture thought was the root cause. Things worked well for a couple of days until Korea hit their Friday night weekend peak player load. Korea crashed twice at their peak Friday night the 24th and then every peak time for the next 4 days. Creating huge login queues and multiple additional off peak outages as we tried to stablize the service. On Weds the 29th of May we finally stablized the Korea service during peak times but continued to deal with daily outages as we had to clean up lots of small problems still affecting them. We don’t have Korea fully out of the woods yet, we are dealing with another issue that has affected 10′s of thousands of players about every other day at peak times.
PvP Client
Rioter Damiya shares a few words on the dreaded Adobe Air Client, as well as opinion on the way lore is implemented in the menus.
Will you be switching the Adobe Air Client?

Damiya: There are no current plans to move away from Air.
Damiya: I’ll be the first to admit there are some problems with the client as it exists right now, but Air (and more specifically the Flash ecosphere) is not the culprit. PVP.Net is a complex piece of software that has to be shipped out to over 30 million people twice a month (more or less); improving legacy code designed for a more humble 10 thousand players is somewhat akin to replacing parts of an airplane engine… While midflight, over the Atlantic, at about 35 thousand feet. It’s a delicate process, and the number one priority is ensuring the continued stability of the player experience.
That said, the process of improving parts is ongoing; the Login screen was reworked a few months back, and with the 3.9 PBE patch, we’ll be rolling out a rewrite of the view code for the Runes screen; it should be more responsive and less buggy.
Keep an eye on the PBE forums for more info on that in the near future.
Will you improve the way lore is presented?

Damiya: We’re aggressively exploring options to improve the presentation of lore in PVP.Net. I feel like Freljord was just the tip, as they say. Of the iceberg, that is 
I’m a huge fan of the stuff Kitae’s team is doing 
I’ll bring up the Ping thing but I don’t know that there’s a really good way to do that.
Champion/Skin Sale

Pick up the following champions and skins at a discount until June 10th!
Champions:
Karthus – 395 RP
Sejuani – 487 RP
Swain – 440 RP
Almost-Prom King Amumu – 260 RP

Foxfire Ahri 487 RP

General Wukong – 375 RP

Good luck on the login queue!