Since there is some confusion on the thread, I would like to clarify that I am using EventStreams in the labs version. There is no way too use Kafka outside production and no way to replay historical events (which was what made this so much better in production).
(code for this is mostly in *https://github.com/jdlrobson/wikitrender/blob/master/index.js https://github.com/jdlrobson/wikitrender/blob/master/index.js *for those interested)
On Tue, 12 Dec 2017 at 13:33 Andrew Otto otto@wikimedia.org wrote:
This is a little inferior to the production version as it is unable to
use production kafka and if it has any outages it will lose data.
EventStreams isn’t as good as using Kafka, but an outage shouldn’t be a reason to lose data. Store the Last-Event-ID https://wikitech.wikimedia.org/wiki/EventStreams#Format that EventStreams gives you, and you can use it when the service starts back up to start from where you left off.
> and maybe, as others have mentioned, a good reason to get production Kafka events flowing into Cloud VPS backed projects.
Def not opposed to a Kafka cluster in Cloud mirroring from Prod. :)
BTW, this is a wee relevant: https://wikitech.wikimedia.org/wiki/User:Ottomata/Stream_Data_Platform
This is a draft! I’m shopping this around as a program for next FY. We will see!
On Tue, Dec 12, 2017 at 4:16 PM, Gergo Tisza gtisza@wikimedia.org wrote:
On Tue, Dec 12, 2017 at 12:12 PM, Jon Robson jdlrobson@gmail.com
wrote:
This is a little inferior to the production version as it is unable to
use
production kafka and if it has any outages it will lose data.
Hopefully that gets fixed soon, Cloud VPS / Toolforge is the foundation
for
out volunteer tool developer community who really shouldn't be treated as second-class citizens.
Other than that, moving to the Cloud is not a bad thing for an
experimental
project IMO. It makes it easier to experiment with minimal risk, and it makes it easy to add co-collaborators to your project without having to
get
prod access for them.
I'm hoping to get this onto IFTTT https://ifttt.com/wikipedia with
help
from Stephen Laporte in my volunteer time, as I think this feature is a pretty powerful one which has failed to find its use case in the wiki world. As Kaldari points out it's incredibly good at detecting edit
wars
and I personally have learned a lot about what our editors see as
important
and notable in the world (our editors really seem to like wrestling). I think there are ample and exciting things people could build on top of
this
api.
Yeah a "give me push notifications about ongoing edit wars" tool for
admins
sounds really cool. Although you'd probably want to look at revert trends (or both edit and revert trends) for that. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l