John Herren’s Blog

Typically if you need to monitor Apache, you have to rely on log file analysis or the built in server-status tool via mod_status. I wanted something better to monitor all the hosts on my web cluster.

Now that I’ve got realtime Apache log consolidation working, I found a cool utility in Theo Schlossnagle’s book, Scalable Internet Architectures that let’s you see realtime response code stats for your web cluster. I call the thing wwwtop.

The output looks like this, using the first four hosts with a one-second interval:

Looks like I’ve got some 404’s to hunt down.

Some To-dos:

• pretty aggregate output, similar to mytop.
• a version to monitor APC stats the same way

Combined with top, vmstat, bmon, mytop, and now this, I’ve got a complete toolkit for monitoring the web stack.

At $dayjob I’ve got several front end web servers in load balancing, and I needed a good solution to  consolidate all of those log files. Traditionally this is done by moving the files periodically to a central host and then crunching them down. This did not seem ideal to me because if you are generating gigabytes of log a day, it takes a while to move them around. Instead, I wanted a way to consolidate the logs in realtime. Taking some hints from Theo Schlossnagle, I have a solution working that consists of:

1. Spread toolkit - handles talking to the network
2. mod_log_spread - let’s Apache hosts log to the Spread ring
3. spreadlogd - reads from the Spread ring and writes to a consolidated log file

So far it’s working well. There is a slight but acceptable CPU hit from spread. Getting everything installed was a pain in the ass–there was quite a bit of Googling involved to get all the pieces working, so it’s not a task for the faint of heart.

What are some other solutions?