Introduction

Scribe is a server designed to aggregate logging from multiple clients in real time. It was designed by Facebook to be scalable and reliable, which is especially important when you are dealing with the amount of traffic they have.

According to the "Scribe Wiki" - http://wiki.github.com/facebook/scribe/ on GitHub:

The installation at facebook runs on thousands of machines and reliably delivers tens of billions of messages a day.

Obviously this is overkill for the majority of projects, but if you are working on a large-scale project with millions of users then Scribe could be a perfect fit. It does not come with great documentation however, and the installation process is tricky due to having to manually compile a lot of dependencies.

In this tutorial I will show you how to install a test instance of Scribe on Fedora 8. The steps may also be useful for other Linux users, especially if you are on another Redhat derivative.

First steps

We will begin by creating a location to store all of the software we are about to install. Here I am using /opt/scribe but you can use whatever you like:

[jcollins@redhat ~]$ cd /opt
[jcollins@redhat opt]$ mkdir scribe
[jcollins@redhat opt]$ cd scribe/

Now download the latest for Scribe from here (I used scribe-version-2.01.tar.gz): http://sourceforge.net/projects/scribeserver/

Now extract it into the /opt/scribe directory:

[jcollins@redhat scribe]$ tar -zxvf scribe-version-2.01.tar.gz

Sending up the dependencies

From the README.BUILD distributed with Scribe:

 Requirements
 ============

 [libevent] Event Notification library
 [boost] Boost C++ library (version 1.36 or later)
 [thrift] Thrift framework
 [fb303] Facebook Bassline (included in thrift/contrib/fb303/)
   fb303 r697294 or later is required.

We will now install each one of these dependencies in turn.

libevent

I found on my system, I already had libevent installed, but still needed to install libevent-devel which was trivial with yum while running as root:

[root@redhat jcollins]# yum install libevent

Package libevent-1.3b-1.fc7.i386 already installed and latest version

[root@redhat jcollins]# yum install libevent-devel

Boost

Firstly you should check to see if you have Boost already installed:

[root@redhat jcollins]# yum install boost

Package boost-1.34.1-7.fc8.i386 already installed and latest version

Here I have boost-1.34.x, but this is too old for Scribe (which needs version 1.36 or later), so go ahead and remove this package:

[root@redhat jcollins]# yum erase boost

Now we will need to download the required version of Boost and build it from source. Begin by creating a directory to store the download in (here I'm using /opt/scribe/boost):

[root@redhat scribe]# mkdir boost
[root@redhat scribe]# cd boost/

Now download tarball from here: http://sourceforge.net/projects/boost/files/boost/1.36.0/

And configure/make/make install (as the root user):

[root@redhat scribe]# cd boost/boost_1_36_0
[root@redhat boost_1_36_0]# ./configure
[root@redhat boost_1_36_0]# make
[root@redhat boost_1_36_0]# make install

Thrift

Now that we have the correct version of Boost installed, we can install Thrift. Begin by creating a directory to store the download in (here I'm using /opt/scribe/thrift):

[root@redhat scribe]# mkdir thrift
[root@redhat scribe]# cd thrift/

And download the tarball from here (I used thrift-0.4.0.tar.gz): http://incubator.apache.org/thrift/download/

I did experience some (hopefully temporary) broken linkage while attempting to download Thrift from the above URL, but did manage to find it on this mirror in Ireland that you can use as a direct download link:

http://apache.mirrors.esat.net/incubator/thrift/0.4.0-incubating/

Now from within /opt/scribe/thrift, extract the tarball:

[root@redhat thrift]$ tar -zxvf thrift-0.4.0.tar.gz
[root@redhat thrift]$ cd thrift-0.4.0

And configure/make/make install (as the root user):

[root@redhat thrift-0.4.0]$ ./configure
[root@redhat thrift-0.4.0]$ make
[root@redhat thrift-0.4.0]$ make install

Note that at this stage I experienced a Ruby related error, which I was able to resolve by installing the ruby-devel package (yum install ruby-devel)

can't find header files for ruby.
setup.rb:655:in `command': system("/usr/bin/ruby" "/opt/scribe/thrift-0.4.0/lib/rb/ext/
extconf.rb") failed (RuntimeError)
   from setup.rb:660:in `ruby'
   from setup.rb:1238:in `extconf'
   from setup.rb:1230:in `config_dir_ext'
   from setup.rb:1532:in `__send__'
   from setup.rb:1532:in `traverse'
   from setup.rb:1549:in `dive_into'
   from setup.rb:1530:in `traverse'
   from setup.rb:1524:in `exec_task_traverse'
   from setup.rb:1519:in `each'
   from setup.rb:1519:in `exec_task_traverse'
   from setup.rb:1223:in `exec_config'
   from setup.rb:991:in `exec_config'
   from setup.rb:826:in `__send__'
   from setup.rb:826:in `invoke'
   from setup.rb:773:in `invoke'
   from setup.rb:1578
make[2]: *** [all-local] Error 1

fb303

Next up is to install the fb303 library from Facebook. It comes with Thrift in the contrib directory, so you just need to change directory there and run the provided bootstrap.sh script, followed by configure/make/make install:

[root@redhat thrift-0.4.0]# cd /opt/scribe/thrift/thrift-0.4.0/contrib/fb303
[root@redhat fb303]# ./bootstrap
[root@redhat fb303]# ./configure
[root@redhat fb303]# make
[root@redhat fb303]# make install

Finally we are ready to install Scribe

You will need to run the following commands to install Scribe:

[root@redhat scribe]# export LD_LIBRARY_PATH="/usr/local/lib"
[root@redhat scribe]# /sbin/ldconfig
[root@redhat scribe]# ./bootstrap.sh
[root@redhat scribe]# ./configure
[root@redhat scribe]# make
[root@redhat scribe]# make install

Is it working?

Now from the same terminal, run the following command to start-up Scribe with a sample configuration file provided with the release:

  [root@redhat scribe]# scribed -c /opt/scribe/scribe/examples/example1.conf
  [Fri Sep 17 10:44:35 2010] "STATUS: STARTING" 
  [Fri Sep 17 10:44:35 2010] "STATUS: configuring" 
  [Fri Sep 17 10:44:35 2010] "got configuration data from file </opt/scribe/scribe/examples/example1.conf>" 
  [Fri Sep 17 10:44:35 2010] "CATEGORY : default" 
  [Fri Sep 17 10:44:35 2010] "[default] choosing new retry interval <322seconds" 
  [Fri Sep 17 10:44:35 2010] "Creating default store" 
  [Fri Sep 17 10:44:35 2010] "configured <1stores" 
  [Fri Sep 17 10:44:35 2010] "STATUS: " 
  [Fri Sep 17 10:44:35 2010] "STATUS: ALIVE" 
  [Fri Sep 17 10:44:35 2010] "Starting scribe server on port 1463" 
  Thrift: Fri Sep 17 10:44:35 2010 libevent 1.3b method epoll

It's alive!

Further reading

Scribe is a very complex piece of software. This tutorial simply showed you how to install a test Scribe server, but there is a lot more to it than that. Here is some follow-up reading for you to digest:

Scribe Home Page on GitHub: http://github.com/facebook/scribe

Scribe Wiki: http://wiki.github.com/facebook/scribe/

Scribe Facebook Page: http://www.facebook.com/pages/Scribe/84758500701

---

Updated 2021 : note that the above post was originally published in 2010, but is left here for archival purposes. Sadly just a few years after I published this, work stopped on Scribe. It is a shame, as it was a very nice solution for log file aggregation, which is still a vital DevOps requirement today. I have unlinked all of the links, most of which are now broken.