Updated Some of Rackspace’s hosted Microsoft Exchange services have been taken down by what the company has described as a “security incident”.
The company’s most recent incident report at the time of writing, time-stamped 01: 57 Eastern Time on December 3rd, offers the following information.
“On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.”
The incident is further described as “isolated to a portion of our Hosted Exchange platform”
Rackspace has no idea when it will be able to restore its service to those impacted by the security incident.
“We are actively working with our support teams and anticipate our work may take several days,” its status page advises.
The incident manifested as what Rackspace described as “connectivity and login issues.
An update time-stamped 08: 19PM Eastern Time on December 2nd went a little further, describing it as “a significant failure in our Hosted Exchange environment.”
But no information about the cause of the incident is available at this time, however the combination of an outage and a lengthy restoration process suggest ransomware could well be a factor.
- Will cloud giants really drive colos off a financial cliff?
- Rackspace considers selling part of business: ‘Everything’ on the table
- Rackspace decimates workforce: One in ten staffers let go this week
Rackspace has offered impacted customers free access to Microsoft Exchange Plan 1 licenses on Microsoft 365 for the duration of the incident and shared instructions on how to get that up and running. The instructions suggest the work to get it running will take 30 minutes to an hour.
Which won’t ruin weekends for the IT pros asked to get the job done, although Rackspace has warned that preserving data will be tricky for those with hybrid environments. What happens for pure-play hosted Exchange is not addressed in the status document.
So even if companies can make the move, Monday could be interesting at as the move to Microsoft 365 means mail will flow but likely means archives won’t be available. Good luck making Monday morning meetings or accessing email archives, Exchange-on-Rackspace customers.
Rackspace’s business is predicated on provision of secure and resilient services, so this incident is likely to go down very badly with customers as Exchange typically stores lots of sensitive data, and messaging and calendaring are core business functions.
If the incident spreads beyond hosted Exchange, their anger will be even stronger, as Rackspace host many mission-critical applications and vital data.
If you know more about this incident or you are one of the organisations impacted by it, let us know! ®
Updated at 11: 15PM UTC, December 3rd, to add:
Rackspace’s outage continues. The company’s most recent status update states: “Our security and operations teams continue to work both internally and closely with outside experts to determine the full scope and impact of the issue involving our Hosted Exchange environment.”
But there’s no sign of the services coming back online.
The Register has received accounts of the incident from customers trying to access Rackspace support.
They’re not pretty.
“The way they have handled this has been HORRIBLE. NO SUPPORT. NO EMAIL. NO NOTHING. Who knows when we will have answers,” one customer wrote.
“I called the support line, held and listened to lousy music for three hours and 14 minutes and 19 seconds and finally had to terminate the call,” wrote another.
We’ve also seen reports that the process of migrating to Office365 is not straightforward, as it may require information sent to email addresses that are inaccessible because of Rackspace’s outage.
Rackspace has promised to update users every twelve hours. The sun never sets on The Register’s editorial team, so we’ll keep reading them and reporting significant developments.
Updated at 07: 00 UTC, December 4th, to add:
Rackspace’s latest status update states “We continue to make progress in addressing the incident”. But that progress is not swift: the update also states “this will continue to be an extended outage of Hosted Exchange.”
The company has also acknowledged that its offer of free migration to Microsoft 365 is not going well for some customers.
“We have heard and understand that self-migration may not be simple and can be challenging to implement,” the update states, adding that “thousands” of Rackspace customers have been able to make the move.
“Current wait times for customer support are much longer than usual,” the update adds. “For those who are finding the process challenging and are awaiting support, we ask for your patience as we increase staff to help every customer. Since our last update, we have mobilized roughly 1000 support Rackers to reduce wait times and address ticket queues.”
Updated Some of Rackspace’s hosted Microsoft Exchange services have been taken down by what the company has described as a “security incident”.
The company’s most recent incident report at the time of writing, time-stamped 01: 57 Eastern Time on December 3rd, offers the following information.
“On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.”
The incident is further described as “isolated to a portion of our Hosted Exchange platform”
Rackspace has no idea when it will be able to restore its service to those impacted by the security incident.
“We are actively working with our support teams and anticipate our work may take several days,” its status page advises.
The incident manifested as what Rackspace described as “connectivity and login issues.
An update time-stamped 08: 19PM Eastern Time on December 2nd went a little further, describing it as “a significant failure in our Hosted Exchange environment.”
But no information about the cause of the incident is available at this time, however the combination of an outage and a lengthy restoration process suggest ransomware could well be a factor.
- Will cloud giants really drive colos off a financial cliff?
- Rackspace considers selling part of business: ‘Everything’ on the table
- Rackspace decimates workforce: One in ten staffers let go this week
Rackspace has offered impacted customers free access to Microsoft Exchange Plan 1 licenses on Microsoft 365 for the duration of the incident and shared instructions on how to get that up and running. The instructions suggest the work to get it running will take 30 minutes to an hour.
Which won’t ruin weekends for the IT pros asked to get the job done, although Rackspace has warned that preserving data will be tricky for those with hybrid environments. What happens for pure-play hosted Exchange is not addressed in the status document.
So even if companies can make the move, Monday could be interesting at as the move to Microsoft 365 means mail will flow but likely means archives won’t be available. Good luck making Monday morning meetings or accessing email archives, Exchange-on-Rackspace customers.
Rackspace’s business is predicated on provision of secure and resilient services, so this incident is likely to go down very badly with customers as Exchange typically stores lots of sensitive data, and messaging and calendaring are core business functions.
If the incident spreads beyond hosted Exchange, their anger will be even stronger, as Rackspace host many mission-critical applications and vital data.
If you know more about this incident or you are one of the organisations impacted by it, let us know! ®
Updated at 11: 15PM UTC, December 3rd, to add:
Rackspace’s outage continues. The company’s most recent status update states: “Our security and operations teams continue to work both internally and closely with outside experts to determine the full scope and impact of the issue involving our Hosted Exchange environment.”
But there’s no sign of the services coming back online.
The Register has received accounts of the incident from customers trying to access Rackspace support.
They’re not pretty.
“The way they have handled this has been HORRIBLE. NO SUPPORT. NO EMAIL. NO NOTHING. Who knows when we will have answers,” one customer wrote.
“I called the support line, held and listened to lousy music for three hours and 14 minutes and 19 seconds and finally had to terminate the call,” wrote another.
We’ve also seen reports that the process of migrating to Office365 is not straightforward, as it may require information sent to email addresses that are inaccessible because of Rackspace’s outage.
Rackspace has promised to update users every twelve hours. The sun never sets on The Register’s editorial team, so we’ll keep reading them and reporting significant developments.
Updated at 07: 00 UTC, December 4th, to add:
Rackspace’s latest status update states “We continue to make progress in addressing the incident”. But that progress is not swift: the update also states “this will continue to be an extended outage of Hosted Exchange.”
The company has also acknowledged that its offer of free migration to Microsoft 365 is not going well for some customers.
“We have heard and understand that self-migration may not be simple and can be challenging to implement,” the update states, adding that “thousands” of Rackspace customers have been able to make the move.
“Current wait times for customer support are much longer than usual,” the update adds. “For those who are finding the process challenging and are awaiting support, we ask for your patience as we increase staff to help every customer. Since our last update, we have mobilized roughly 1000 support Rackers to reduce wait times and address ticket queues.”
