What a Nightmare!
What a rough stretch it has been for the past several days.
Starting mid-last week we were seeing major performance issues on BUMPzee. The database slowed to a crawl and queries that were previously taking fractions of a second to execute were taking tens of seconds. With the constant data updates that happen on BZ, frequent deadlocks were occurring, sending the server spiraling down into sometimes complete unresponsiveness.
We tried everything we could to optimize queries, tables, data, indexes, stripped out some heavy queries, and more. Nothing would work.
Several times, we had to take BUMPzee into maintenance mode so we could do some major database operations. And a few times, I disabled the widgets so they wouldn’t cause your websites to slow down while they loaded.
To make matters even more stressful, I was a groomsman in a wedding this weekend. I was checking on the server and cleaning up the deadlock from my iPhone when necessary. (Don’t tell the bride!)
Once it started to look like the problems were deeper than the data. We upgraded the database, restarted servers and services. Something, or some combinations of things did the trick. Now finally, things are looking back to normal. Unfortunately, I changed so much that I won’t ever know what the root cause of the problems were. On the bright side, as I write this, things are looking faster than ever thanks to all of the database optimization and tuning we did.
I sincerely apologize for the slowdowns you were experiencing here and on your blogs due to the widgets. I hope that we’re out of the woods on this one.
September 17th, 2007 at 10:21 am
Thank you for the update. This kind of information is helpful.
I wonder if it would not have been better to take BZ offline on Friday and just leave a message that it was down for maintenance. As it was, we could almost sort of get the site, and then not, which led to frustrations.
September 17th, 2007 at 10:48 am
In hindsight, it probably would have been better. But it was the sort of problem that was hard to know existed when the site was down for maintenance.
I disabled widgets on Saturday, which I felt was the most critical problem caused by this as it affected others’ sites, and we had the whole site down for a good part of Sunday.
Hopefully we won’t have a next time, but if we do, we’ve learned a lot from this one and will do better.
October 15th, 2007 at 12:36 am
Is it happening again? Some parts of the site are easy to access, and other parts (like adding a blog) take forever.
October 15th, 2007 at 2:00 am
yeah, some slowdowns this weekend. I’m working on some more speedups. thanks for your patience.
February 14th, 2008 at 7:06 am
Teen stabs, kills therapist in…
Teen stabs, kills therapist in Walker Street apartmentEagle Tribune, MA -33 minutes agoShe is listed as the program director for the Lawrence…
February 18th, 2008 at 8:54 pm
Environmental Power Announces Achievement of…
… including plant outages and repair requirements; the difficulty of estimating construction, development, repair and maintenance costs and timeframes;…
February 20th, 2008 at 6:45 pm
Jack…
and readers, Lets DIGG this Blog post so we can get more blog posts FAST!! Good stuff….
April 6th, 2008 at 12:47 pm
What was the interesting thing? Oh! That. Yeah. I should have guessed. This post is good, but not good enough.
April 9th, 2008 at 6:23 am
Sir, yes sir! I’m agreeing with you, but I don’t think everybody do. You should not be so rude, it frightens of.