On Friday, March 1, 2013, Tyler Romeo wrote:
On Fri, Mar 1, 2013 at 11:46 AM, Asher Feldman <afeldman@wikimedia.orgjavascript:;
wrote:
don't think a custom daemon would actually be needed.
While I was at flickr, we implemented a pubsub based system to push notifications of all photo uploads and metadata changes to google using redis as the backend. The rate of uploads and edits at flickr in 2010 was orders of magnitude greater than the rate of edits across all wmf
projects.
Publishing to a redis pubsub channel does grow in cost as the number of subscribers increases but I don't see a problem at our scale. If so,
there
are ways around it.
We are planning on migrating the wiki job queues from mysql to redis in
the
next few weeks, so it's already a growing piece of our infrastructure. I think the bulk of the work here would actually just be in building a frontend webservice that supports websockets / long polling, provides a clean api, and preferably uses oauth or some form of registration to ward off abuse and allow us to limit the growth of subscribers as we scale.
Interesting. Didn't know Redis had something like this. I'm not too knowledgeable about Redis, but would clients be able to subscribe directly to Redis queues? Or would that be a security issue (like allowing people to access Memcached would be) and we would have to implement our own notification service anyway?
I think a very light weight proxy that only passes subscribe commands to redis would work. A read only redis slave could be provided but I don't think it includes a way to limit what commands clients can run, including administrative ones. I think we'd want a thin proxy layer in front anyways, to track and if necessary, selectively limit access. It could be very simple though.
0mq? RabbitMQ? Seem to fit the use case pretty well / closely.
Hmm, I've always only thought of RabbitMQ as a messaging service between linked applications, but I guess it could be used as a type of push notification service as well.
*--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com javascript:; _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l