2010-10-22

Scribe (& thrift) for Debian Squeeze

Some time ago, I asked what are the common solutions for services remote logging. It was suggested scribe, an application log server developed at Facebook and later released as an open source project, and we finally managed to get this ready for Debian squeeze.

Scribe requires a couple of other tools to work: thrift, a framework for services development (born at Facebook too), and thrift-fb303, a piece of thrift used as a bassline layer for Facebook services.

Searching the web for some work-in-progress of a Debian packaging, I found the wonderful work done by SimpleGeo; since I have some changes to do (and also take the occasion to play a bit with github), I forked their repos in these ones: scribe, thrift and thrift-fb303.

I've prepared a repository with those 3 packages, if you would like to try scribe, give it a look. They are for squeeze because, well, after all I need it for work and since it requires Python 2.6, using squeeze was the easiest choice (backporting 2.6 to lenny could be a mess, even if there are packages out there) and so I could also see how the upgrade from lenny would be.

Let's now dive into the doc, and some examples too, to find out the several configuration options and decide what's best for us.

9 comments:

Justin A said...

Hi :-) I think I might have been the one that recommended Scribe (almost a year ago!)

You may also want to look at flume: http://github.com/cloudera/flume

It is "new" but seems to address a number of limitations in scribe, and has a lot more features.

Sandro Tosi said...

@Justin: yep, indeed you are. Anyhow, you should definitely stop giving me more work to do :) I'll give flume a look, but the fact it runs on a JVM might be a showstopper for us.

Timur Izhbulatov said...

Hi Sandro,

Thank you very much for you work! It was really helpful for me when I was building Scribe for Ubuntu 10.10. I had to tweak your packages to work with Ubuntu's Java deps. Also the scribed init script need to be reworked. Currently I hacked it quickly to meet my requirements. Plus, I needed the i386 version and the source packages.

Would it be feasible to give you my changes for review? I think it would be nice to have everything available from your repo.

Thanks,
Timur

Sandro Tosi said...

@Timur: sure I can host a ubuntu/ dir with your packages: just give me an address where I can rsync them (or a url to wget, bonus points if it's just one big tar :) ).

popofr13 said...

Hello,

We are interesting for using Scribe in production, and we search solution to deploy on our production machines.
We would like to re-build the package for thrift, fb303 and Scribe for non 64arch, and maybe update thrift to version 0.7. We clone your thrift repo and try to use the command dpkg-buildpackage, but without success… (dpkg-source: error: can't build with source format '3.0 (quilt)': no orig.tar file found).

Could you help us ? Could you explain us how to build the package from the GitHub thrift repository ?

We notice a dependance on python<<2.7, but this is the default version on our ubuntu version. What about this dependance?

Thanks in advance.

Sebastien
Riverline (French company)
sebastien |at| riverline |dot| com

popofr13 said...

Sorry, my email address is sebastien |at| riverline |dot| fr

Sandro Tosi said...

Hi Sebastian,
when I tried to build for the first time scribe & all-its-deps IU fought a lot to find the right combinations, that works, all packages build fine and do the job. And the combination of versions I found was the one in my repos, so I can't help you much on upgrading thrift.

The python (<< 2.7) dependencies is because I'm only interested in squeeze packages, the stable debian version, that has 2.6 as default. I'm sure the 2.6 is the lowest release usable, but I don't remember if 2.7 wasn't supported at all or I just ignored it because I didn't care.

the dpkg-buildpackage errore is weird but you can take the orig.tar.gz from my debian repo, maybe that way the package will build.

Please let me know of any success story of building scribe & friends on higher versions of python/thrift, it would be cool to know :)

Anirudh said...

Great to have a repo, I've been trying to compile it myself and am getting stuck. A totally n00b question, how do I add the repo to the sources.lst? Can you tell me what the line looks like?

Sandro Tosi said...

@Anirudh: google is always your friend ;) anyhow the repo line is like this:

deb http://people.debian.org/~morph/thrift-scribe/ ./