Like Russell Coker I had to deal with an apache OOM crashing my main Xen instance yesterday.
In my case, it is due to the Czech IP 88.102.6.114, which decided it
would be a good idea to download the entire output of viewvc, recursively,
and apparently also in parallel. After 2 hours and some 40 thousand hits,
this eventually started enough instances of viewvc and svn that it used up
512 mb of ram and OOMed the system.
I don't have a satesfactory fix yet. Viewvc shouldn't let itself be run dozens of times simulantaneuosly, surely. Apache doesn't seem to provide a way to limit the total number of CGIs it can launch at once, except by limiting the total number of connections. Apache's rlimitnproc option sadly only applies the limits after starting a CGI, rather that applying them to apache itself. Something that could control the number of CGIs started on a per-directory and per-vhost basis would be ideal.
For now, I used ulimit -u when starting apache, which prevents many
viewvc processes from running at once, but allows anyone who DOSes
viewvc to also DOS all other CGIs on the system.