Monday, February 06, 2006

Mystery of the Vanishing Entry

I posted a new entry last Saturday. It just disappeared into thin air. It wasn't on my blog at all. How did that happen? Just this morning, it was there. Even the link that I had put in the Category entry wasn't there.

Any cyber-sleuths out there who can solve this mystery?


Lydia Teh said...

Immediately after publishing this post, I checked out Blogger's maintenance page. Aha, found the culprit.

This is their report.

Blogger Status
Sunday, February 05, 2006

Just a quick reminder that we will be going ahead with a planned network maintenance on Monday the 6th from 7–8PM PST. Blogger and Blog*Spot blogs will be unavailable during that time.

This maintenance won’t fix everything, but it will make things better. I promise.

Posted by Pete at 21:18 PST
Saturday, February 04, 2006

As many people will have noticed, we've had significant hardware trouble over the past couple days. We have tried to keep all of you informed about the status of the service and our efforts to correct the problems. Unfortunately, today's problems were widespread enough such that we were unable to provide ongoing updates. One of the things we'll be doing in the short term is to come up with ways to prevent this.

Eric details the chronology below. There is one residual issue related to the outage: some legacy users have domain associations between Blog*Spot and external domains. These associations are not functional at this time.

In addition to working on this problem, we'll also be making the necessary changes to guard against the outages we experienced over the past 48 hours.

We will provide updates on our progress as changes are made. I sincerely apologize for the downtime - in the past six months I feel we've made significant strides to limit this type of failure. We obviously have more work to do and carrying it out will be our highest priority.

Posted by Jason at 23:12 PST

Blogspot is again experiencing problems - we are investigating.

Update, 8PM: We have restored all of Blog*Spot, save one of our filers. This means that some blogs will still be unpublishable and inaccessible. Our engineers are continuing to work on this problem.

Update, 11PM: Blog*Spot servers are restarting now and connecting with the filer. All blogs should be publishable and accessible within the next 20–30 minutes.

[So how has everyone else’s weekend been? —Pete]

Posted by Eric at 16:04 PST
Friday, February 03, 2006

We are currently experiencing some problems with Blog*Spot. A percentage of blogs are completely inaccessible. For others, you may need to reload a few times before you connect to a web server that will serve them. We anticipate things will be calmed down within the hour.

Update, 5:20PM: Blogger and Blog*Spot should be fully operational with the exception of one of our file servers. This means a percentage of blogs will be unpublishable and inaccessible. We are currently waiting for a lengthy disk integrity check to complete on the file server. There may also be transient delays serving blog pages while our load balancing evens out.

Update, 8:00PM: The Blog*Spot web servers are coming back online with connections to the file server that was out for most of the evening. (Its disk is totally okay, btw.) So, within the next 10–15 minutes everything should be completely back to normal.

Again, super major apologies for this.

Posted by Pete at 16:02 PST is back working again, again. Unscheduled outage hat trick tonight. As I get more information from our worldwide operations team I’ll let you know a bit more of why went down, and what we’ve done to keep this particular problem at bay.

Posted by Pete at 03:49 PST is now back from a 45 minute outage. Two outages in one night. I feel like Dante from Clerks: “I’m not even supposed to be carrying the pager today!”

We’re really sorry about this spate of random downtime. As Jason mentioned, I mentioned that Monday’s fixup will lessen these types of outages in the future. The Blogger and Google engineers and ops folks are not just sitting around waiting for the next failure; we’re actively improving our infrastructure to lessen both the planned and unplanned Blogger outages.

Posted by Pete at 01:37 PST
Thursday, February 02, 2006

We're recovering now from a unscheduled outage of about 20 minutes. As Pete mentions below, we are almost finished migrating from some unreliable hardware. This should help to prevent downtimes like this.

Posted by Jason at 18:39 PST

Syafique said...

I had the same problem too...thought I was the only one!

Lydia Teh said...

Syaf, we're all on the same boat. Up till now the vanishing posts haven't been restored. Will they ever?