Research In Motion was able to restore service Thursday to millions of BlackBerry users around the world who experienced outages in recent days, but the damage to the company's reputation could not come at a worse time.
The outage is considered RIM's biggest service failure in years. It comes as the company is already facing declining sales and increasing competition from companies like Google, Apple and Microsoft, which are all launching new products.
Mike Lazaridis, the co-CEO at RIM told reporters in a Thursday morning conference call that "we've now restored full services."
Lazaridis said it had been "frustrating" for RIM to not be able to fix the problems more quickly.
"You should know that we're taking immediate and aggressive steps to minimize risk of this happening again," Lazaridis said Thursday.
About 70 million people around the world use a BlackBerry and while many users were not affected by the outage, discontent is easily sown in the age of Twitter and Facebook.
Some users complained not only about the outage but about the company's slow response to its failure.
"It's important for brands to be perceived as being responsible and perceived as if they care very much about the brand community," Sidneyeve Matrix, a professor of media and mass communications at Queen's University, told The Canadian Press.
"The damage that can be done to the brand is something worth worrying about."
Lazaridis posted a video to YouTube on Thursday morning, apologizing directly to BlackBerry users for the disruption.
"Since launching BlackBerry in 1999, it's been my goal to provide reliable, real-time communications around the world. We did not deliver on this goal this week; not even close," he said in the video.
"I apologize for the service outages this week. We've let many of you down, but let me assure you that we're working around the clock to fix this. You expect better from us and I expect better from us."
The Waterloo, Ont.,-based RIM has explained that a crucial link in its European infrastructure failed Monday and then, its backup failed as well.
Lazaridis provided details on those events during the conference call on Thursday morning.
"A dual-redundant, high-capacity core switch designed to protect the infrastructure failed and caused outages and delays for some customers in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina. This caused a cascade failure in our system," he said.
"There was a backup switch, but the backup didn't function as intended. This led to a backlog of data in the system. The failure in Europe, in turn, overloaded systems elsewhere."
RIM is now conducting "root cause analysis" to pinpoint just how the outage occurred.
Lazaridis told reporters that he cannot put a timeline on how long this investigation will take to complete.
"Systems like this don't fail this way. They are designed not to fail this way, so it's going to take some time, I'm sure of that," he said.
RIM shared dropped 15 cents to $24.12 Thursday on the Toronto Stock Exchange.
Is RIM system vulnerable to 'cascading failure'?
During the outage, many BlackBerry users have also been letting RIM know that they know there are other options on the market.
When services were restored on Thursday, Holly Lauren Bryant of Edmonton tweeted that BlackBerry and RIM are "letting me down."
"Be careful or I'll go #android," Bryant wrote on Twitter, making reference to Google's mobile phone operating system.
Others made less plausible suggestions about the alternatives available to them.
Political strategist Greg MacEachern joked on Twitter that he was "about to push for the return of the fax," while he waited for BlackBerry to get its services up and running.
Technology analyst Carmi Levy explained to CTV's Canada AM that the way that RIM's networking architecture is built makes it vulnerable to this kind of "cascading failure."
He notes the BlackBerry service is unique in that it works through network operations centres, or NOCs. These are essentially large data centres that process email and text messages and then push them out to the BlackBerry device automatically and instantaneously.
Levy said no matter what the cause of the widespread outage, it is a major problem that RIM's backup system failed to function as expected.
"It isn't amazing that they had a piece of equipment failure; that happens all the time in data centres around the world. What is amazing is that the backup that was supposed to kick in when this thing went down didn't kick in and took the entire facility down," he said.
