Why you need to double (or triple!) check cloud performance expectations and SLAs

If you buy public cloud services without double-checking exactly what you’re getting with a service-level agreement (SLA), you might as well  be buying a pig in a poke.

How much trouble? Peter Wayner, contributing editor of the InfoWorld Test Center, who benchmarked “identical” Amazon EC2 Micro cloud instances with the Da Capo Java benchmarking suite found the “results were all over the map.”

How erratic were they? “There was one instance that could index files with Lucene (The Apache search software) in three different runs of 3.4, 4.0, and 4.1 seconds; it was predictable like a watch. But another instance started at 3.4 seconds, then took 39 seconds for the second run and 34 seconds for the third. Another instance took 14, 47, and 18 seconds to build the same Lucene index with the same file.”

Wayner also benchmarked medium machines with 3.75GBs of RAM and a promise of one virtual core with two EC2 Compute Units. These gave much more consistent results but “even these numbers weren’t that close to a Swiss watch.”

I’ve seen the same thing myself. When I benchmarked infrastructre-as-a-service (IaaS) clouds recently for PC Magazine, I had to run the benchmarks over and over again to get rational numbers.

You see, this isn’t just an AWS problem. It’s a problem with any public cloud service. That’s because on a public cloud, you’re trading control for flexibility. As Wayner wrote, “On a bad day, you could end up sharing a CPU with 10 crazy guys trying to get rich by minting Bitcoins by melting the processor; on a good day … you could end up sharing a machine with 10 grandmothers who only update the Web pages with a new copy of the church bulletin each Sunday.”

So what can you do about this? Well, for starters, you could run your own private cloud.

True, you won’t have as much flexibility as you would with a public cloud. But you can have as much control over your own cloud as you do now with your traditional data-center servers. The rub is, of course, you’d also many of the same expenses of running your own data-center.

The other alternative is to get a SLA that makes sure you get the full resources you need when you need them. Alas! It’s not that wasy.

You see, as Lydia Leong, a research vice president in the Technology and Service Providers group at Gartner, recently observed, “cloud IaaS SLAs can readily be structured to make it unlikely that you’ll ever see a penny of money back — greatly reducing the provider’s financial risks in the event of an outage.”

In the case of Amazon, who she singles out, “AWS both define their SLA not in terms of instance availability, or even AZ (Availability Zone) availability, but in terms of region availability. In the AWS case, a region is considered unavailable if you’re running instances in at least two AZs within that region, and in both of those AZs, your instances have no external network connectivity and you can’t launch instances in that AZ that do; this is metered in five-minute intervals.”

You’ll note that, as poor as this SLA is, they don’t cover even quality of service (QoS) issues. These concerns are endemic to public cloud providers. The Cloud Standards Customer Council a user advocacy group has found SLAs sorely lacking in protection.

Tiny differences in SLA language can also make an enormous difference. For example,as Charles Babcock of InformationWeek pointed out, many “cloud service providers average their SLA uptime percentage [99.95%] over a year.” Others, such as Amazon’s EC2, Microsoft Azure, and Google Compute Engine offer 99.95% up time over a month.

Don’t think for one second that this a is difference that doesn’t make a difference. Over a year, your cloud system could be down for four-hours and 23-minutes before you’d get any cash back. Over a month, you can only be down for no more than 22-minutes? Still don’t get it? Same you’re a larger retailer. Would you rather be guaranteed you’d be down for 22 minutes or less in December, or, in the worst case,over four hours during the holiday rush?

The bottom line? Before investing in any public cloud, you’re going to need to get your in-house general counsel as well as your CIO and CTO involved. When it comes to getting the most from the public cloud, you’re going to need both technology and legal expertise.


Security as a Service: Do you get the service you expected?

The cloud land-grab continues: Netflix, Spotify, Bitly, GE Healthcare and DropBox

RightScale sees the cloud going hybrid



  1. […] SoftLayer. It enables you to compare and contrast basic cloud options such as virtual machine size, service level agreements (SLA), region support, and operating system […]


  2. […] Cloud Service Level Agreements (SLAs) are not created equal. Even minute differences in SLA language can make an enormous difference to your bottom line. For example, “many cloud service providers average their SLA uptime percentage [99.95%] over a year.” Others, offer 99.95% uptime over a month. […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: