So the word outage was on everyone’s lips this week, not least at Mobile Europe where a malware exploit of our webhost’s platform disabled our site for most of the week.
That experience, where we were essentially in the hands of a third party provider gave me new insight into the key concerns people have around outsourcing and managed services. A clear line of communication to our provider was vital in finding out what was going on. Even so, it was a frustrating three or four days.
But this was one small B2B media website going down. How much worse for O2 and Orange France, both of which saw potential brand and revenue losses due to major service outages. National media piled on. In France, Sarkozy’s buddy and Orange CEO Stephane Richard was called to a newly Socialist-packed National Assembly to give evidence as to what caused the outage. In the UK, O2’s initial slow reaction attracted criticism, although by the end of the second day its Twitter team were being praised in some quarters for their breezy handling of some unpleasant suggestions as to what O2 could do to itself. (Look them up, they’re too rude for a family newsletter like this) Once the good ship O2 was out of stormy waters CEO Ronan Dunne even took time out from his usual Twitter MO of updating us on his sporting engagements to retweet a satisfied customer, and apologise for the loss of service. It seems unlikely Dunne will be following Richard in popping up before a Parliamentary committee, but I wouldn’t yet rule it out.
So how much will the outages cost the operators? At the time of writing, O2 is still to react with a customer offer of any kind and it will take more than a few deftly handled tweets to satisfy those customers looking for recompense for a loss of service. France Telecom, by the way, said it will give all mobile customers a free day of calling in September to compensate them for the service disruption. Richard said that the incident is likely to cost the company "several dozen million euros”.
Naturally, in the industry, speculation centred on what had gone wrong. In France, the operator said that its Next Generation HLR (subscriber database) from Alcatel-Lucent had malfunctioned. There were data inconsistencies between the different elements of the HLR, Orange explained, leading to a storm of messages flying back and forth as the system tried to resolve the issue, eventually overcoming the HLR. Users were OK as long as they stayed in a cell, but as soon as their details were updated on the HLR, they too would be deregistered in the network. The software had been updated just before the outage.
In the UK, industry sources told Mobile Europe that O2 too looked to have been the victim of an HLR outage, as the issue didn’t seem to be anything to do with the access network, but rather the inability of some handsets (O2 said about a third of its user base) to register at all with the network. That looked to indicate a similar suspect for the outage. The Register’s Bill Ray went further, and said sources indicated that an Ericsson-supplied database had gone down during a data transition, taking with it affected subscribers’ ability to connect.
Whatever the cause of the outage, what we can learn immediately? First, the core is as important as access. An HLR may sound like some esoteric telco TLA, but a bungled update has cost France’s flagship operator millions. At a time when there is concern about operators de-skilling (Telefonica is outsourcing its core network operations to Huawei, don’t forget) this is a useful reminder that core competencies extend also to the, er, core.
Two, communicate. Yes, O2’s social media team did a decent job, but its CEO’s absence was noted (yes, what can he do, not much, but if you have a Twitter presence, then use it – even if it is just to point people towards a customer service feed), and intitial information was scarce. No doubt that was because the operator didn't have any and didn't wish to mislead. But when the operator did start communicating, it managed to get a grip on the situation quickly, showing the benefits of communicating.
Three, it’s not the end of the world. It’s not the event, it’s how you deal with it that counts with your users. That’s why I hope O2 is planning some comprehensive customer communications over the next day or so to apologise, yes, but also to state what it will do to make it up.
Right, onto other matters. We’ve seen the LTE licenses awarded to Russia’s big four this week. For free. The proviso is that the operators have to pay for refarming the spectrum that is currently used by the armed services. The extent of that spectrum is a state secret, though, so it’s hard for even the informed observer to judge what the capex required will be. One Russian financial analyst put had a huge spread on potential costs to the operator. Despite this somewhat opaque situation, the question UK readers will now be asking themselves is: will Russia have 4G before us? You wouldn’t bet against it.
Something else you wouldn’t bet against is more network sharing deals, as we saw Vodafone and Three Ireland confirm that they would be sharing a network infrastructure. I though this was quite interesting for the fact that it was Three and not Telefonica that Vodaone has gone teamed up with. Following the ambitious UK network sharing deal announced last month, you might have expected the two group operators to further consolidate in Ireland. But the deal is an indication that where possible, operators will look for deals that limit their infrastructure commitments. Just keep investing in the core, please.
Have a good weekend everyone, and to our French readers, have a happy Bastille Day.
Keith Dyer
Editor
Mobile Europe