Scribe (& thrift) for Debian Squeeze

Some time ago, I asked what are the common solutions for services remote logging. It was suggested scribe, an application log server developed at Facebook and later released as an open source project, and we finally managed to get this ready for Debian squeeze.

Scribe requires a couple of other tools to work: thrift, a framework for services development (born at Facebook too), and thrift-fb303, a piece of thrift used as a bassline layer for Facebook services.

Searching the web for some work-in-progress of a Debian packaging, I found the wonderful work done by SimpleGeo; since I have some changes to do (and also take the occasion to play a bit with github), I forked their repos in these ones: scribe, thrift and thrift-fb303.

I've prepared a repository with those 3 packages, if you would like to try scribe, give it a look. They are for squeeze because, well, after all I need it for work and since it requires Python 2.6, using squeeze was the easiest choice (backporting 2.6 to lenny could be a mess, even if there are packages out there) and so I could also see how the upgrade from lenny would be.

Let's now dive into the doc, and some examples too, to find out the several configuration options and decide what's best for us.

Remote Apache logging with syslog: is there anything better?

We're doing a pilot to do remote logging for some Apache logs (possibly other services in the future). We've heard of remote syslog capability (and since syslog is on all Linux system) we're giving it a try.

The configuration is quite simple:

1. [srv] prepare a machine to do the log server
2. [srv] open it's rsyslogd to receive messages on UDP (or TCP) port
3. [srv] log the selected facility.level to a log file
4. [cli] forward the above facility.level to @srv

Ok, easy and it works. Unfortunately we face pretty fast the limitation of syslog:

1. only 8 facilities for users custom log (local0-local7)
2. only 8 levels for logs severity

this is a big block (only 64 combinations, if you agree to do some "dirty" stuff) if you want to log remotely several services on several different platform on the syslog server.

rsyslog is quite flexible and it allows you to filter messages based on the tag in them, and log in different files, but it's still something very "home-made" and not that professional.

I don't we're the only one needing a remote logging tool, and while syslog is the classic solution, is very bind to the system logs and not to the applications logs: any suggestions for this task? I'd like to hear how you solved this task, possibly without a custom tool, but using something standard.