ipb Report post Posted June 12, 2010 Around 9 AM (PST) on 9 June, our database crashed due to widespread corruption in the tables. After discussing the options with our host, we decided we would rather be down longer and fix the tables than revert. This is for several reasons, but the biggest reason was a technical one: we believed that it was quite likely (due to how corrupt the database was) that the backups were also corrupted. As the repairs were on going, it was determined that we had insufficient memory to fix the tables. This was not entirely unexpected, as we've been having memory issues during the nightly backup. So we authorized an increase in RAM, and more tables were repaired. Last night, it was determined that the partition the directory that mysql uses to store temporary data was too small to complete the repair, which is why it had taken more than 2 days to repair the posts table and it still wasn't done. So we had them stop the repair, reconfigure mysql, and start the repair again, and when I woke up, it was done. We optimized the tables before opening up, to reduce overhead and speed things up a bit. Share this post Link to post Share on other sites
ipb Report post Posted June 13, 2010 Oh, and the nightly slow down should be gone now. The problem with that was we were running out of RAM and mysql was running out of temporary storage -- both issues we had to fix to repair the database We did just change our caching engine, so it might come back -- we're running a little higher on RAM right now than I'd like. If it does, we'll probably add more RAM or revert the caching engine. Share this post Link to post Share on other sites
