Customers hosted on our UK Grid on the Service63 and Service64 cluster pair should now have normal service. Email is flowing normally but there may be some residual queues while the backlog clears.

We are very sorry for this further inconvenience. Any local issues should be reported to our Incident Hotline on 020 7843 2302.

Please continue to check this blog and our twitter feed https://twitter.com/mimecast for more details. We will provide further updates throughout the day.

Thank you again for your patience.

 

_____________
Previous updates and email routing information can be found here.
Mimecast Outage – Update for Customers on Service63 and Service64
Mimecast Outage – Status Update – 1045hrs
Mimecast Service Outage – Status Update – 0930hrs

Add your comment (0)

Customers hosted on our Service63 and Service64 cluster pair should expect a loss of service for the next hour. Regrettably we have suffered a hardware failure on that system which is affecting the backend storage arrays.

In order to resolve this problem we must take the entire cluster offline for a period of time. Unfortunately there is no alternative for customers hosted on this cluster.

We are very sorry for this further inconvenience. This short outage will only affect customer hosted on Service63 and Service64. Our Incident Hotline is available on 020 7843 2302

Please continue to check this blog and our twitter feed https://twitter.com/mimecast for more details. We will provide more updates throughout the day.

Thank you again for your patience.

 

_____________
Previous updates and email routing information can be found here.
Mimecast Outage – Status Update – 1045hrs
Mimecast Service Outage – Status Update – 0930hrs

 

Add your comment (0)

Our infrastructure and technical teams are bringing more services back online in an ordered manner.

Mail delivery in and out of the Mimecast Platform is flowing as normal. Please direct any isolated local problems to our service delivery team on 020 7843 2302

We are slowly removing the work-arounds in place, although customers will not notice or be affected by these changes.

Some residual problems may still exist with Mimecast Services for Outlook, but we are aware of these issues and are working to resolve them as soon as possible.

Please continue to check this blog and our twitter feed https://twitter.com/mimecast for more details. We will provide more details throughout the day.

Thank you again for your patience.

 

_____________
Previous update and email routing information can be found here.

Add your comment (2)

Our infrastructure and technical teams have been working through the night to restore as many services to normal as possible.

Mail delivery to and from the Mimecast platform is flowing as normal, and in most cases there should be no queues or delays; there may be some residual mail queues where a large backlog existed overnight.

We still have some issues to resolve. The failed network infrastructure that caused the problems yesterday has been fully restored and protected from further outage, and we have made DNS updates that will route email away from hosts that remain offline. There are a small number of temporary work-arounds in place that will provide service to customers and allow us to restore normal service in the back ground.

Our infrastructure teams will bring service back online in a managed process so normal service will be restored throughout the day. Residual problems with Mimecast Services for Exchange and Mimecast Services for Outlook may occur, and customers may see queued messages with an error of “Unable to send message body” or “could not open file”. We are aware of these problems and will resolve them as quickly as possible.

Please continue to check this blog and our twitter feed https://twitter.com/mimecast for more details. We will provide more details throughout the day.

Thank you again for your patience.

_____________

Previous update and email routing information can be found here.

Add your comment (0)

A Tough Email Day

Today Mimecast suffered an HA network hardware failure that shut down services from one of our data centres in the UK. The outage lasted a little over three hours from about 11am UK time to just after 2pm. Afterwards some customers experienced slower service responses due to back log recoveries underway.

I wanted to take this opportunity to say sorry personally and on behalf of Mimecast to our customers and partners affected by this issue today. I also wanted to give some background to the problem and our response.

First of all, I know how critically important email is to you and to your businesses. The importance and value of email and the challenges of running robust email infrastructures were among the main reasons Neil Murray, my co-founder and our CTO, and I started Mimecast just over 10 years ago. We sincerely appreciate the faith and trust that you place in us as your email gateway and your email continuity provider. We make promises to you that we will always be there to deliver your messages. We work day and night to meet that promise and invest extensively in our software development, data centres and infrastructure resilience to meet it. We have teams of people who work around the clock to support the service and support you as our customers and partners.

For three hours today we did not live up to our availability promise. We are very sorry.

Over the last ten years we have not had any significant outages because of our infrastructure and because of the constant scenario planning we conduct to ensure we’re mitigating against any points of failure.

As a cloud vendor, our platform infrastructure works in an active-active model, where communications are handled by all sides of our grid. If there is any unavailability in a component another part of the grid can take over. Failing over an entire data centre happens extremely rarely and we deliberately do it manually as an automatic failover of this scale brings significant risks.  The plans we had in place underestimated the time it would take to complete the task.  We aim for under 30 minutes, however this one took us over 2 hours.

We will be reviewing this procedure and making sure that we can do it faster – much faster – should we be called upon to do it again.

In terms of next steps, we will of course honour our SLA obligations, and we’ll be in touch proactively with all affected customers on this issue in the coming days.  We appreciate the patience that many of our customers have shown during this tough day and we will be working extra hard to ensure it doesn’t happen ever again.

__________
Previous update and email routing information can be found here.

Add your comment (14)

Co-founder and CEO
Mimecast

First, an apology.  Today Mimecast UK customers have experienced problems with our email services, caused by a network hardware failure at our Woking data centre.  Our infrastructure teams have identified and isolated the problem, and are bringing all affected customer systems back online now.

Customers’ systems will be available soon, but there will be a backlog of email to process, so email may take some time to return to normal.

There are several things you can also do as a customer to make sure your email service is restored as quickly as possible.

  • Firstly check your AdCon addresses for your system availability. Once you can log back into AdCon your email will be flowing too. If you’re not sure of the AdCon address you can click the “Log In” link on our website.
  • Only the even numbered hostname for AdCon will be available for now. They will look something like this. http://serviceNN.mimecast.com/mimecast/admin , where ‘serviceNN’ is the cluster number for your account.
  • Importantly, please check your outbound SMTP smarthost connectors on your Exchange server. You should have two configured, one for each host name. Mail will only be delivered to the live host because the other half of the pair will be offline. If you only have one hostname, please make sure you add the second hostname to an outbound SMTP smart host connector.  Please refer to this these KB articles: Exchange 2003, 2007, 2010.
  • MTA2 Customers ONLY: If you’re using MTA2, please check your smart hosts are directed to eu-smtp-1.mimecast.com and eu-smtp-2.mimecast.com
  • Your email will be queued at the sending servers, so you will see email start to flow in and out of your mail server without needing to resend it.

Please note, this is not a full incident report, but an interim update to help you get back online. Our CEO, Peter Bauer, will be posting about the issue later today. But in the meantime, we hope this list of tips helps to resolve the issue as soon as possible.

We would like to take the opportunity to thank you for your patience and we will continue to post updates on our Twitter feed @mimecast

Add your comment (3)

passport stampLast month Israeli security forces imposed their right to examine your email at their border crossings; the initial panic was calmed by a clarification from the Israeli Attorney-General stating the specific circumstances for the search. Previously, in 2009, the United States imposed a right to search your electronic devices, and keep them for further examination, at border crossings too—without any suspicions of wrongdoing. Are these signs that our local data is no longer private when we travel?

State sponsored search of your devices, and data, now becomes the latest privacy worry for any international travellers; we’ve always been worried about malicious attempts to gain access to our data, or having our laptops stolen from airport security screening points, but now the case for travelling completely clean is made.

Many technology travellers I talk to have always maintained a set of clean equipment, which is only used on trips outside their native country. Before and after every trip their laptop, smartphone, and tablet get factory wiped and restored from a known good image. This is especially important when returning from a trip to ensure the platforms remain clean – those devices are also replaced more frequently than home devices, and are occasionally stripped to check for “extra hardware”.

Paranoid?

Maybe; but more sensible than paranoid, as we’re in the days of state sponsored hacking such as Flame, Stuxnet and Duqu.

If you’re wondering how you manage to work in such a sterile environment – have a think about how the cloud supports your remote working now. Keeping your data on your local hard drive isn’t the necessity it once was; it seems quite antiquated to me.

Cloud services that allow you to store your data online mean you’re only ever a click away from that data, and given the ubiquity of Internet access these days, that’s never a problem. Of course data stored in the Cloud isn’t beyond the reach of search warrant of subpoena, but at least it’s not local on your device being carried through a border crossing.

Email inboxes should remain empty until you’re safely through a border crossing, and on a known and trusted network. Once you’ve downloaded your recent email remember to remove the account and wipe the device before you leave the country too; there’s no sense taking the precaution for inbound border crossing and forgetting about the outbound.

The same applies to file data, leave your files in the cloud and only access them when it’s safe. Don’t store anything locally unless you can securely wipe the hard drive after use.

From an enterprise IT perspective; CISOs and CIOs should educate their users on how to handle such incidents, and of course draw up a policy for international travellers. It does occur to me that your IT department can help, by disabling your access to ‘their’ services on your devices until you give them the go ahead once safely at your destination. Deleting your stored passwords on devices would also prevent the access of data not stored locally.

For travellers the Cloud should now be as essential as your flight socks and money belt. As someone before me once said – “Don’t leave home without it”.

Add your comment (0)

CISSP, CCSK
Mimecast

Article Tags

,

The news is out…as reported in the FT blog corporate Google users have been caught out by recent disruption in Gmail services. Twitter chatter suggests that its Apps Status Dashboard is pointing towards Postini Services causing the problem. The outage seems to now be resolved after 17 or so hours of Google “continuing to investigate the issue” updates, but no real clue to the cause of the problem.Postini

The impact is acute. In tough economic conditions when every sale matters Google customers are missing orders from customers or are unable to work effectively with their colleagues. Email is such a critical tool for most businesses. Suddenly the $50 / year / user costs are looking like a false saving for businesses and the 1% downtime in the Google Apps Gmail SLA is a costly compromise, which was probably glossed over at time of signing.

So what recourse is there for these businesses?  Probably not much and Twitter complaints suggest that Google Apps customer service didn’t live up to expectations either, a reputation Google are struggling to shed.

When email is critical to your business – uptime is critical too.  Any downtime can be expensive and disruptive. That is why Mimecast offers a 100% uptime SLA to our customers, because the difference between the ‘nines’ becomes critical when an email outage hits. Something that Google customers are finding out the hard way.

Beyond email, as customers turn to the Cloud for greater reliability, they will also demand transparency and continued improvement in SLAs as a core justification for their cloud purchase. Just as there are good and bad Clouds – there are good and bad cloud services. Google aside, any Cloud vendor who does not meet the expectations of their customers and their market will hamper the growth of cloud for us all.

Google customers received a stark reminder that there’s no such thing as acceptable downtime, particularly for any business that considers email a strategic application. Reliability is arguably the biggest barrier to adoption of Cloud services so the need for continuity offerings like ours, which keeps email flowing in the event of an outage, will become more and more important as businesses move their critical IT functions to the Cloud.

Perhaps Google is more focussed on its latest eye-wear than its email and app customers?

Add your comment (6)

London was again the venue for the 18th Infosecurity Europe conference last week. Along with over 100 other exhibitors, it was a busy three days for Mimecast – security workshops (summarized in our blog post last week), talking to crowds attracted to our eye-catching stand and some great conversations with media, customers and prospects.

As expected at the premier security event, security was hotly discussed with topics such as mobile security, cyber warfare, threat detection and prevention reoccurring themes.

Given security is a vital part of our offering, we’re most interested in the evolution of the security landscape and how it impacts communication technology in business. From this viewpoint, we noticed a clear point emerging from the conversations this year – we’re entering a new chapter in the maturation of how businesses consider cloud services.

Gone are the days of businesses questioning whether its data is safer in the cloud, now the focus is on issues such as whether a vendor truly believes in industry standards – for instance, there is an increasing expectation of vendors to be accredited against third party standards e.g. ISO 27001 and participate in transparency initiatives such as the CSA STAR registry.

In addition, IT teams are becoming increasingly sophisticated in testing whether vendors can stand by their SLAs. On this subject, one of our customers Paul Dryden invoked a vivid example in one of our workshops about how he evaluates cloud vendors – during a tour of the data centre he spontaneously asks the vendor to cut the power to see how the system reacts. Apparently, only one vendor has managed to perform the immediate simulated power cut for Paul and while this is one of the most extreme examples, we’ve encountered other customers and prospects that have indicated that they’re testing the SLAs of cloud vendors more rigorously.

With increasing pressure to comply with industry standards and more demanding tests around the strength and depth of their service, cloud vendors seem to be at a cross-road. Those services which have the scale and rigour to meet these growing expectations can look forward to growing recurring revenue, while the others will find themselves outside of the commercial conversation.

It’s possible that we’ll look back at 2013 as the year that there was a shake-out of the cloud service vendors, with security one of the key drivers for this change.

Add your comment (0)

Social Media Director
Mimecast

Spring finally arrived and ironically, the sun was shining in London at Infosecurity Europe with no clouds to be seen. The good news for us was this didn’t deter people from joining our Chief Scientist Nathaniel Borenstein and Technical Evangelist, Orlando Scott-Cowley to talk about the cloud. The session was so well received, we thought it’d be useful to summarize the content of the presentation (below):

Left to right: Orlando Scott-Cowley, Nathaniel Borenstein, Paul Dryden

Left to right: Orlando Scott-Cowley, Nathaniel Borenstein, Paul Dryden

They started by agreeing what the cloud is and what it means for security. There’s the public cloud (fully open and accessible), which many vendors use for customer’s data; private cloud (closed), which offer private, business-sensitive uses, and hybrid cloud, which combines features of both. Each allows you different levels of control and security.

“There’s plenty of cloud washing going on with many vendors claiming things to be in the cloud that aren’t.”  — Orlando Scott-Cowley

The Cloud is now accepted as being more secure than your own network.

Putting your data in the cloud does give you an opportunity for better security, as cloud vendors’ security is usually a core part of their business. They’ll have more security and cloud expertise available to them, and are strongly motivated to do a great job – developing a reputation for poor security would likely destroy them. Generally, reputable cloud vendors have the resources to keep up to date with advances in technology and are highly motivated to do a good job and continue innovating.

But it’s also fair to say that cloud providers are bigger targets for attack. So a good place to start your assessment is taking a look at the vendor’s security reputation. If they’ve been around for a while (Mimecast has been here for over 10 years BTW) and you haven’t found any horrifying stories then as Nathaniel said they’re “…likely to be good at cloud security. Cloud vendors live or die by their security. The trick is really knowing whether or not a particular vendor is good at it”. Good cloud vendors are deeply committed to security and very open to talking about it.

So once you know you want a cloud how do you assess a vendor – what questions do you need to ask about them?

Talk to them about security standards. ISO 27001 accreditation is important. But assessing the scope of their compliance is vital – ensure the scope of the accreditation includes the production systems that process customer data, rather than unrelated systems like internal HR or billing platforms.

Also, the workshop discussed the CSA STAR registry from Cloud Security Alliance which allows customers to see detail on participating vendors’ activities and procedures, helping you to compare and evaluate how they protect your data.

Willingness to be open about security standards is an important test for vendors.  If they’re happy to share this they have nothing to hide.  (Of course, there are certain kinds of data that they don’t disclose because it would be a security leak to do so; passwords are just the most obvious example of this class of information.)

Where is my data?

Some customers also need to know where their data is housed and under what jurisdiction it sits.  Assess what this means for your business. If this matters to you, then the cloud vendor should be willing to discuss this with you. This is not just a matter of legal concerns. Think also about connectivity – businesses in areas with poor Internet connectivity will often be much better off accessing servers that are nearby.

Will you get the service you want if the data is located somewhere you can’t guarantee the network performance you need? What continuity plans does the vendor have in place to keep their performance guarantees? It’s always acceptable to ask questions about the service – a good vendor will say ‘yes’ to allowing you to test the reliability of their service too.   (However, if they’ve already been tested by several independent auditors that you’re inclined to trust, it’s not necessary that you burden them by repeating the tests.)

What do you take to the cloud?

When you’ve a service or application that is commoditized, it’s well suited to benefiting from the cloud. There’s also a whole set of apps, such as data mining, that largely can’t exist outside the cloud – they’re made possible by the characteristics of processing data aggregated in the cloud, or analytics for example. With older apps and services hybrid systems are often a good option – ask, ‘can you get the benefits of the cloud without going fully to the cloud?’

Nathaniel then laid out a list of questions customers should get answers to from all vendors – the questions that vendors “dread being asked.”  The questions were:

  • How do you manage your cryptographic keys?
  • How do you handle change control in your software?
  • How do you handle patches to your OS and other key software?
  • How do you encrypt all client data at rest? Do you guarantee its integrity? What is my role in keeping it safe?
  • Are your development and operational platforms well separated?
  • What access do your administrators have to customer data?
  • What are BCPs on matters like testing, documentation etc?
  • How redundant is your data and how do you prevent/recover from outages?
  • Do your employees have constrained, granular roles that are easily configured?
  • How do you manage security incidents? What is logged? How long is it retained?
  • Who are your third party security auditors?
  • Do you do regular penetration testing and vulnerability scanning?
  • Is your platform and business IOS 27001 accredited? If not, why not?

By the end of the session, it was clear both that there’s a strong appetite for this kind of help in assessing cloud vendors, and that there are even more questions that belong on the list.

Watch this space for more on this as we will explore the questions in a future post.

If we’ve missed out a great question that worked for you we’d love to hear it – post the question here or email Orlando at osc@mimecast.com.

Add your comment (0)