Pulp Free

It’s been in the works for a while - but the last few weeks really just bothered me in that the performance of my Web hosting has been so dismal that I’ve finally upgraded to a dedicated virtual machine from MediaTemple.

I’ve sang the praises of MT before specifically around their GridServices offering - but more specifically around their transparency in discussing their own problems. However - even I have not been blind to their performance issues - now going on for the past 5 months or so…

While I’m sure that in “a few weeks” there will be an announcement of a new GS system (or some kind of new branding for it) - I just couldn’t wait any longer. I actually had been using GoDaddy’s virtual private server for these sites before moving to MediaTemple. But like all things technical in nature - their performance was terrible at the end. CPU utilization wasn’t contained in their shared environment and they didn’t seem to want to fix it. Will MT be any better in their offerings? Time will tell. So far I’ve been pretty happy in the difference. Load times are pretty fast and I have the access to install anything I want again (not that I really needed to… But I’ll probably start running my Cacti scripts with RRD up there again…)

One of the sad parts about this whole experience was the let down in support that I received from MT surrounding my efforts to stay on the GS system. While the performance was pretty lousy (so much that I started keeping my own monitoring stats on it) - I finally opened a ticket on January 16th, 2008 expressing my concerns and the data that supported it.

Hello… I’ve been tracking the response time of my Web site for several months now and it seems to have been getting slower and slower. I’ve only used 11 GPUs (on average) out of 1000 available per month. So I know I’m not a burden on the system by any means.

I’ve tracked response times both from my home Internet connection as well as other Internet monitoring sites and they go from half a second to over 10 seconds - some upwards of 20 seconds to respond for the same page - just seconds apart (refresh - no cache, etc.)

I’m curious to know how this wide deviation in response times from multiple locations for the same resources reflects on how the gird system is performing.

For the most part - it’s “ok” - but nothing near what I would have expected from a low-volume Web site such as mine on such a highly performing GRID cluster system.

And for this incident - it’s not just today or yesterday… I’ve been tracking this issues for about 40 days now. The average was around 4 seconds (still very slow) and now it’s closer to 8 seconds.

What did I get back? Yeah - the standard

We have actually noticed a possible network routing issue in our past support history to your area around St. Louis.

I think I coughed and said something under my breath…

This kind of bantering went on and on. A nice gem:

We sincerely appreciate you taking the time and effort to collect this data. We are aware that the (gs) is currently not in a perfect state of stability yet, but we are getting there. We are continuing to implement the re-architecture our storage segment in an effort to stabilize and increase the performance of this segment.

However, what really set me off was blaming my own Web site… A pretty lean WordPress install that may have upwards of two people a day looking at it (including myself…)

After speaking with our CTO, who has reviewed your data, we have come up with some possibilities of explaining what is going on.

While the hills and valleys in the data are expected over the past week of performance, which we are currently taking measures to resolve, we can’t attribute the increase in response time from the beginning of your monitoring to recent date to these issues.

We believe that you may have a growing database that may be not indexed properly or possibly a sessions directory that is growing and not be flushed properly. Either one of these cases could lead to the symptoms we are seeing.

Another possible issue could be plugins. You may want to try disabling some of our plugins to see if that improves the performance of your site as well.

We urge you to investigate the above and let us know if any of the above as helped.

Ummm… My response?

I’d like to take exception to the conclusions of your research. I have not changed any behavior of my database or the plugins and the performance of the Web site has not added a second (averaging 5 seconds now) than what it was last week.

I’ve actually done a performance of just an image and it’s performance stays pretty constant. - But even then - it’s sometimes 200 MS, sometimes 1000 MS - such a non-database driven request should be the same all the time (it’s only 60 KB in size). I’m going to do the same for a PHP info page and measure the results. And again, for the 1000 GPUs I have - only about 15 are used a month. My tables are very small and the system as a whole should be quite a bit faster…

And a last note - your conclusions about performance and databases would not explain why the request responses are anywhere between 800 MS to 8 seconds for a standard front page - they would be the same time all the time (based on the very low levels of GPU and traffic).

That’s right - with MT’s slogan for their GS services as:

I should have the power of hundreds! My 15 GPUs a month out of 1,000 seemed like I should be able to use a little more of my paid “horsepower” to server up some pages a bit faster. Even if it wasn’t faster - perhaps it could at least be consistent?

I turned to caching and limiting plugins just to do my due diligence:

I’ve now set up database tuning jobs, installed a WordPress cache plugin… And it’s still anywhere between 500 ms (with cached paged) to 15 seconds…

<!– Dynamic Page Served (once) in 15.196 seconds –>

I took a copy of my Web site, installed it (database and pages) on a VMWare Centos install and was able to get an average of 700 ms (not even using the cache plugin!!!)

So 15 seconds for your Grid system - 700 ms for my VMWare Linux install that averages a 3% CPU utilization.

Then silence from the support staff…

Your response: 2008/02/14 19:57
It’s been another 2 weeks and no response to my additional information. I’m still seeing between 150 ms to over 5 seconds for the same page from the same monitoring service. I’ve even gone with caching pages to try and improve performance… Is it time to move? How does this compare to a VPS?
Your response: 2008/02/20 21:06
Hello? Another week has passed…
Your response: 2008/02/24 07:16
4 more days…
Your response: 2008/03/02 11:30
7 more days
Your response: 2008/03/06 19:21
4 more days

It wasn’t for another 11 days after that one that I received word back from MT - March 17th. They said that my ticket wasn’t coming up in their system and did apologize. In the mean time, I did open up two more tickets referencing my first ticket which were in fact closed without answers…

So yes - MediaTemple’s Grid-Service isn’t ready yet. I read the reviews before hand - same issues, same problems. It was great when I opened my account for about a month and then was poor ever since… Never any consistency and rather unacceptable customer service. I actually had the chance to talk to my support technician on the phone and I will say that he was very intelligent, well spoken, technical and very aware of their internal storage subsystem problems. I asked him “Should I move to your DV system to rid myself of these issues? Or are you close to fixing them?”

I moved to a DV system.

As Fox Mulder once said:

How’s it working so far?

Before		After

I’ll check back in a bit and let you know…

BTW - 100th post! A bit far down the path from my first post back on July 23rd, 2006 - 631 days ago. Statistically speaking, I think I need to write more frequently! Almost once a week isn’t too bad…

A few months back I decided to move my Web hosting over to Media Temple for several reasons… More space, ability to host multiple Web sites on a single account (and not just with domain forwarding) along with a lot more disk space. However - the real cool part of MT is the usage of their Grid-Service (GS) platform. Lately my own Web sites have been affected by some performance issues just like when the Grid-Service was put online… Latent Web site performance, seemingly numerous maintenance windows, etc. Knowing that I really only pulled in just over 3100 visits from around 2100 users in November (13 GB transfer) really meant that such outages weren’t really that much of an impact on my low volume Web sites.

However - this observational story of mine comes in two parts…

Part One - What is Grid Computing?

You would think that with a grid computing model that MT has in place would allow individual Web sites for flip around to different servers and I should “never” seen an outage… So why am I impacted? Is it really a grid system? Well - MT doesn’t really release any information about the technical nature of their hosting plans… There’s some nice “feel-good” information online that really sells the product… But again - light on details.

Now, for my daytime job - I architect & engineer fairly large WebSphere XD clusters for a large financial company. Probably have $10M spent on software and hardware alone… We’re clustered all over the place, N+1, no single points of failure… blah blah blah - and it really seems to work… Most of the time… However - just like MT - we have single points of failure when it comes to infrastructure outside of the application hosting model… For MT - this now seems to be related to their storage systems. So with great designs - decisions as to what gets engineered for different components really make the difference. I’m guessing that a lot of effort went into storage planning at MT - but when you build systems like this for a living (or for a company’s hosting plans) - engineering efforts really just shift the bottlenecks from one system to another. Fix the application layer and you could start having issues with database. Fix the database and you highlight storage performance issues (see a trend here?). It’s a moving effort - and in terms of how technology works - your customers continuously see “issues” all of the time… You have to set the expectation that those items happen… That is unless you get yourself into a circular path of destruction (whaa ha ha ha!!) or if you’re Google…. Okay - next point…

Part Two - Taking Responsibility

Even before I moved my Web sites to MT I rambled a bit about how I liked MT’s company philosophy on taking responsibly. Today there was a note simply called “We Apologize” from Demian Sellfors - CEO of MT.

Media Temple would like to apologize to our (gs) Grid-Service customers for the series of issues relating to the (gs) system in the past few months.

and

The situation with the storage upgrade is particularly frustrating because the vendor supplied update was intended to fix issues - not create new ones.

This one hits me at work since we fight such items with our own vendors… However, the company I work for (and MT) have a responsibility to provide hosting solution to our business partners. Failures, no matter if by a vendor or by our own actions, are still our responsibility. In this case, MT informs their customers about what has happened, what they’re doing to fix it, and an apology for the effect their service problems have had on their customers. Granted, MT is offering 2 months of credited hosting to help lesson the damages… But I wonder - do most corporate management teams offer their customers 2 free months of service for such ongoing problems? Do corporate customers actually “pay” for their IT services? Could you even offer a refund? Providing ownership and accountability for what services are provide really drive home the benefits and drawbacks of IT resources when there are real dollars involved…

In either case - it’s once again refreshing to see in writing efforts by a company to just do the right thing. (I do wonder what’s happening behind the walls at MT… perhaps not all of their team feel disclosure is best for business… I, for one, champion transparency when put into the right context…)