The Hippie Hosting Co-op server has been humming along for several months now. We've had our share of growing pains, and recently we'd been seeing memory usage on the server spiking pretty severely.
For the non-Hippies: the Co-op is run on a Mediatemple (dv) dedicated virtual server, and we have 197 domains for 84 members running on the box. Most are pretty low demand, simple blogs with low traffic. Some are higher demand. But, on average, we should be well within the limits of what the server can do.
To try tracking down the resource issues, I fired up
top to see what was sucking up memory. It looked like MySQL was using waaaaay more than it needed to, followed by apache and PHP (all expected on a server that's basically just a LAMP webserver...). Thankfully, Mediatemple provides some really great recipes to help track down and solve these issues. I followed the MySQL Tuning howto, and it seems to have stuck. Memory spikes are gone, and we're back well into the normal range. Awesome.
We're still seeing higher-than-expected memory usage, and the CPU is running a little higher than I'd like.
Next, we need to try to optimize the apache/PHP side of things a bit more, and maybe come back to MySQL to tweak performance a little. But, we're back on our feet (touch wood).