The Industry Standard - Praise and anger over The Planet data center fire - Comments

We've been with them in

Matt — Sun, 08 Jun 2008 01:10:32 -0700

We've been with them in various incarnations (Rackshack, EV1, The Planet) for over 10 years. This explosion was the first outage we have experienced of any kind. That's 100% up time for more than 10 years.

We got email from them very quickly after the incident and have been satisfied with the information provided in the updates. One of our servers suffered disk damage from one of the generator glitches and their staff worked two days round the clock running manual filesystem checks to get it back.

We will not be moving anywhere else. Most other hosting providers have far more problems just in their normal day-to-day ops. It took an explosion to cause trouble here, and they worked hard to recover from it.

The planet told everyone to

Former customer — Sat, 07 Jun 2008 08:14:41 -0700

The planet told everyone to go to the forums, but when you ask questions you get no answers and if you ask what some feel are the wrong questions one of the planets managers calling customers trolls. When I posted telling this manager it was sad to see a company manager calling customers trolls as anyone that has spent any time on the Internet knows calling someone a troll is getting close to using the "N" word with someone.

After I posted that it was sad, he banned me from posting anymore comments. I have yet to receive any contact from his bosses (I sent copies of the comment to all of them) so I can only assume that they are aware or feel the same way about the customer.

Folks systems fail, it is just how it is. But to then have the company call it's customers trolls for asking about info related to the service that company is being paid to provide is just too much for me. I have moved my server and I know many who once they get the rest of their data from their servers will be gone as well. I feel the planet should know that for some of these folks the last straw was your manager calling the customers trolls.

In years to come when things are calmed down what will be remembered about this whole incident is how at a time of crisis the customer where called TROLLS by management

Update: One of our

Ian Lamont929744 — Fri, 06 Jun 2008 07:58:01 -0700

Update: One of our contributors, developer Larry Borsato, is working on a piece that discusses the redundancy issue. He was directly impacted by The Planet outage. It should be published within a week or two. You can see a list of his earlier posts here.

To Affected customer: Yes,

Richard from GoStats — Thu, 05 Jun 2008 11:22:33 -0700

To Affected customer: Yes, nginx is crutial in managing which servers or centers to send requests to. If one server or center goes dark, nginx can rely on the static cache or send the remaining data requests to other up servers. Using a simple apache will not be able to handle down servers in the datagroup.

To Richard from GoStats:

Affected customer — Thu, 05 Jun 2008 09:19:59 -0700

To Richard from GoStats: it's still unclear to me how nginx (or any other reverse proxy) can possibly help in case of datacenter-wide disaster (as nginx in failed datacenter will fail itself, and nginx in working datacenter won't be able to help); can you elaborate on it a bit?

To Affected customer: Using

Richard from GoStats — Wed, 04 Jun 2008 14:14:58 -0700

To Affected customer: Using nginx or any suitable http proxy is an important part of the failover solution. DNS alone will only get you do far. Nginx helps keep the servers "sane" and manages the requests sent to the remaining servers. Using an httpd proxy is the first step in managing the inner data of anyone's site.

As for a relevant forum with no registration, I don't know of any off hand. You could send me an email and we can talk more until I find one. (or you could try setting up "autocomplete" in your browser)

To Richard from

Affected customer — Wed, 04 Jun 2008 13:20:26 -0700

To Richard from GoStats:
"The purpose of my article was to explain how to mitigate a datacenter disaster in general with special regard to DNS support." - it looks to me that it can be written in a simple phrase "use 'DNS failover' from DNSMadeEasy", that's about it (as nginx and so on aren't really related to DNS resilience); am I right? (it's a good suggestion BTW, thanks).

"or meet in a relevant forum" - could be interesting, but do you know any such forum which doesn't need registration (I'm way too lazy to go through all that registration pages :-) )?

To Affected customer: The

Richard from GoStats — Wed, 04 Jun 2008 12:57:16 -0700

To Affected customer: The purpose of my article was to explain how to mitigate a datacenter disaster in general with special regard to DNS support. Everyone has different inner data technology needs and that's not the focus of the article. I wasn't clear if you were asking for help, just curious, or merely skeptical.

Ah, a good database discussion - exceeds the scope of this forum, but email me through gostats if you want to talk more or meet in a relevant forum.
-Yes, you could call it "magic technology" ;)

To Richard from GoStats: so

Affected customer — Wed, 04 Jun 2008 12:13:13 -0700

To Richard from GoStats: so basically the answer is "we have some magic technology", and you're proposing to consult about it, which means that the whole article was written not to help others but to promote your own consulting business. A pity.

Actually, if some restrictions are acceptable (for example, requirement to have nodes absolutely coherent is dropped, which most likely you've did as absolute coherency isn't important for stats) it's not that difficult to build such a thing (for example, using simple database log rollforwarding). I was just curious if you came with something really interesting (like all-coherent multi-node database in active-active mode without incurring intra-node latencies ;-) ).

To Affected customer: Yes, a

Richard from GoStats — Wed, 04 Jun 2008 10:00:28 -0700

To Affected customer: Yes, a database is used, but no data was lost because the GoStats technology used to collect the data is inherently fault tolerant. Since we rely on collecting all data interrupted, it's important to not simply use the default database configuration and setup. It's a strategic advantage that we planned for from the beginning. The details of this setup are a trade secret.
There is a less technical press release here: http://gostats.com/press/data-center-explosion/

Sorry I can't give you the "how"; but if you are interested in some consulting, perhaps we could work something out.

To Richard from GoStats:

Affected customer — Wed, 04 Jun 2008 07:20:20 -0700

To Richard from GoStats: thanks for suggestions, but one key question was not covered: do you have any database and if so, how did you manage to synchronize databases between the servers? Or you just don't care about loss of data (which vast majority of sites does care about)?

About Hosting.com : I've

Affected customer — Wed, 04 Jun 2008 07:09:36 -0700

About Hosting.com : I've took a quick look and didn't like "Request a Quote" there, as it's usually a sign of exorbitant pricing. Any idea what is their pricing for cheaper hardware (NOT virtual) server segment and what you can get for it?

"Is this an insult?" - No,

Affected customer — Wed, 04 Jun 2008 06:59:19 -0700

"Is this an insult?" - No, it's not an insult, it's SLA.

On advice where to move - we've had THAT bad experience with several companies, that I'd not risk moving to anybody except for top guys like RackSpace or DataPipe; it will cost you though.

We moved 16 servers out of

GoneFromThePlanet — Tue, 03 Jun 2008 22:25:38 -0700

We moved 16 servers out of the facility on Sunday to another ISP and had our lawyers send The Planet's lawyers a letter indicating that they won't be receiving any more money from us and if they don't like it we are more than happy to go to court.

Hosting.com is quickly

Answer — Tue, 03 Jun 2008 14:32:55 -0700

Hosting.com is quickly becoming one of the most reliant and successful hosting sources in the country. I would get a quote from them first. They have a great product as well as an amazing team of innovative people on staff to assist in your every need. Has anyone else been pleased with Hosting.com?

Praise and anger over The Planet data center fire

Ian Lamont — Mon, 02 Jun 2008 07:26:54 -0700

An explosion at The Planet datacenter in Texas took out thousands of Web servers over the weekend, including those belonging to StatCounter, RepuMetrix, and LinkToMeet. No one was injured or killed, but some servers are still offline as the company struggles to make repairs. I read the accounts over the weekend, and received this message this morning when I logged onto StatCounter to check the traffic to my personal blogs:

(Note that StatCounter has had several recent vendor issues, such as the PayPal fiasco described here)

Despite the fire and related outages, The Planet's quick response is being hailed by some customers, such as Joseph Fiore of Repumetrix, who had this to say about The Planet:

"[I] believe that much of the reputation management drill was superbly held together by the online forum updates (rather than overloading their phone system), time oriented announcements, calm and courteous support people both online and over the phone, and phone messages in case you were out of the online loop."

However, other customers were not impressed. A writer who identified himself as Tom Tsillas, CEO of Tamion, posted a comment that claimed his company was losing "thousands of dollars" and the goodwill of more than one million members of LinkToMeet, a website Tamion operates. He also vowed to relocate his company's servers in the near future.

Were you affected by the outage? What's your take on The Planet's response or disaster recovery procedures? Leave your comments below.

More news, commentary, and predictions from The Industry Standard:

Special Feature: Where are they now? The Industry Standard tracks down 10 dot-coms from the Web bubble of the late 1990s
Special Feature: The Industry Standard's Top 25 B-to-Z List Blogs
Special Feature: 10 'Net services that will succeed (and 10 that will probably fail)