What a rough stretch it has been for the past several days.

Starting mid-last week we were seeing major performance issues on BUMPzee. The database slowed to a crawl and queries that were previously taking fractions of a second to execute were taking tens of seconds. With the constant data updates that happen on BZ, frequent deadlocks were occurring, sending the server spiraling down into sometimes complete unresponsiveness.

We tried everything we could to optimize queries, tables, data, indexes, stripped out some heavy queries, and more. Nothing would work.

Several times, we had to take BUMPzee into maintenance mode so we could do some major database operations. And a few times, I disabled the widgets so they wouldn’t cause your websites to slow down while they loaded.

To make matters even more stressful, I was a groomsman in a wedding this weekend. I was checking on the server and cleaning up the deadlock from my iPhone when necessary. (Don’t tell the bride!)

Once it started to look like the problems were deeper than the data. We upgraded the database, restarted servers and services. Something, or some combinations of things did the trick. Now finally, things are looking back to normal. Unfortunately, I changed so much that I won’t ever know what the root cause of the problems were. On the bright side, as I write this, things are looking faster than ever thanks to all of the database optimization and tuning we did.

I sincerely apologize for the slowdowns you were experiencing here and on your blogs due to the widgets. I hope that we’re out of the woods on this one.