AddThis have open sourced their large-scale data processing solution "Hydra":

Hydra on Github

FLOSS Weekly covered this in a podcast recently, that included Chris Burroughs the engineer at AddThis who lead the effort to open source Hydra:

Hydra on FLOSS Weekly

It looks like an interesting tool for processing large volumes of data, I'm really looking forward to playing with this. Kudos to AddThis for releasing it.