Now that the stream is blasting in, I plan to start regularly (hourly?) importing it into Hadoop via Kafka. Once it is there, it can be poked and prodded in any way you like :)
On Nov 29, 2012, at 7:05 PM, Diederik van Liere dvanliere@wikimedia.org wrote:
My main questions right now:
- Would vanadium or kraken be better suited for building this sooner than later (within a few weeks)
I definitely think that this is something for Kraken but would like David and Ottomata to chime in as well.
- Would anyone like to help? (David, your guidance around coding the frontend visualization would be highly valued even if you don't have a day or two to personally throw at it)
I would say, let's use Etsy's statsd & graphite visualization solution, see http://github.com/etsy/statsd statsd is network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services like Graphite and posssibly also Ganglia. Very little coding involved :)
just my 2 cents.
D
Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops