Consolidating Apache Log Files with Spread

At $dayjob I’ve got several front end web servers in load balancing, and I needed a good solution to  consolidate all of those log files. Traditionally this is done by moving the files periodically to a central host and then crunching them down. This did not seem ideal to me because if you are generating gigabytes of log a day, it takes a while to move them around. Instead, I wanted a way to consolidate the logs in realtime. Taking some hints from Theo Schlossnagle, I have a solution working that consists of:

  1. Spread toolkit – handles talking to the network
  2. mod_log_spread – let’s Apache hosts log to the Spread ring
  3. spreadlogd – reads from the Spread ring and writes to a consolidated log file

So far it’s working well. There is a slight but acceptable CPU hit from spread. Getting everything installed was a pain in the ass–there was quite a bit of Googling involved to get all the pieces working, so it’s not a task for the faint of heart.

What are some other solutions?

About John Herren

John Herren is a developer and technical consultant with focus on web applications. He currently serves as Director of Development for Primetime US, the company behind the hit movie and book The Secet. John was formerly staff writer and developer community evangelist for Zend Technologies. Along with founding neat experiments like TagCloud.com, John is an active member in the mashup community, working with API providers and speaking at conferences. He is a published author of Linux certification study material. John enjoys using open source software like PHP and Ruby on Rails to bend the web into exciting new chimeras of hyperlinked goodness. View all posts by John Herren

4 responses to “Consolidating Apache Log Files with Spread

  • Michael

    I have been attempting to play around with spread over the past day. “quite a bit of Googling” was pretty accurate. Did you get mod_log_spread to compile and install for Apache 2? I seem to run into a problem when compiling both the original mod_log_spread and the alpha version that was ported for Apache 2. Any recommendations or references? Thanks!

  • John Herren

    @Michael: It’s been so long ago that I don’t remember what I had to tweak. I’m thinking I got the beta Apache2 version working, but I had to tweak the make file. I also remember modifying spreadlogd’s source file to support log files over 2GB. George and Theo at Omni TI are nice guys, so you can probably track them down or find the appropriate mailing list if you’re still struggling. I’ve built plenty of software in my day, but I haven’t done a lot of C development; these packages were particularly tough for me to get up and running.

  • John

    I have found that mod_log_spread2 does not function ootb with apache 2.2.4 ubuntu. After building and installing, I was getting parse errors on the log format strings. I solved the problem by changing the CustomLog config keyword to SpreadCustomLog. I am getting correctly formatted log data on the logging system.

  • Rob

    @John: Could you post a link to your working config? I’m trying to get this working, too.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: