Notice - 14 December 2007
Online on new server
Thanks to Razer we finally got new server and are back online at full speed! Now we run on 2 servers, one for httpd/mysqld and another one for Squid. Old server acts as squid - that's why transaction went so fast.
Why did it went so slow in the first place?
In past 6 month or so we were using budget dedicated server with only 1.6Ghz Celeron and 1 GB RAM. And the only reason we were online was thanks to Squid - proxy/cache. But it depends on having as many pages in cache as possible so there is no need to re-generate them and run advanced PHP script therefore safe CPU time and RAM.
Anyway, somewhere during update we installed mediawiki extension (RSS for the news) that purged all cache. There was only 1 line but it did it all. After that load on server was so high that cache never had a chance to fill up before page was changed again - so it entered loop of death with several mysql crashed and other bad stuff.
First of all we have a fast server - but it also has mirrored RAID 1 - so there is basically 2 copies of database and all files - in case one crashed it will be all safe stored on the other one. There is also external backup.
If there is any problems - please report.