Hi everyone!
Wikimedia is releasing a new service: EventStreams https://wikitech.wikimedia.org/wiki/EventStreams[1]. This service allows us to publish arbitrary streams of JSON event data to the public. (Psssst: we’re looking for cool new uses https://www.mediawiki.org/wiki/EventStreams/Blog_-_Call_For_Entries[2] to put on an upcoming blog post.)
Initially, the only stream available will be good ol’ RecentChanges https://www.mediawiki.org/wiki/Manual:RCFeed. This event stream overlaps functionality already provided by irc.wikimedia.org and RCStream https://wikitech.wikimedia.org/wiki/RCStream. However, this new service has advantages over these (now deprecated) services.
1.
We can expose more than just RecentChanges. 2.
Events are delivered over streaming HTTP (chunked transfer) instead of IRC or socket.io. This requires less client side code and fewer special routing cases on the server side. 3.
Streams can be resumed from the past. By using EventSource, a disconnected client will automatically resume the stream from where it left off, as long as it resumes within one week. In the future, we would like to allow users to specify historical timestamps from which they would like to begin consuming, if this proves safe and tractable.
I did say deprecated! Okay okay, we may never be able to fully deprecate irc.wikimedia.org. It’s used by too many (probably sentient by now) bots out there. We do plan to obsolete RCStream, and to turn it off in a reasonable amount of time. The deadline iiiiiis July 7th, 2017. All services that rely on RCStream should migrate to the HTTP based EventStreams service by this date. We are committed to assisting you in this transition, so let us know how we can help.
Unfortunately, unlike RCStream, EventStreams does not have server side event filtering (e.g. by wiki) quite yet. How and if this should be done is still under discussion https://phabricator.wikimedia.org/T152731.
The RecentChanges data you are used to remains the same, and is available at https://stream.wikimedia.org/v2/stream/recentchange. However, we may have something different for you, if you find it useful. We have been internally producing new Mediawiki specific events https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki [3] for a while now, and could expose these via EventStreams as well.
Take a look at these events, and tell us what you think. Would you find them useful? How would you like to subscribe to them? Individually as separate streams, or would you like to be able to compose multiple event types into a single stream via an API? These things are all possible.
I asked for a lot of feedback in the above paragraphs. Let’s try and centralize this discussion over on the mediawiki.org EventStreams talk page https://www.mediawiki.org/wiki/Talk:EventStreams[4]. In summary, the questions are:
-
What RCStream clients do you maintain, and how can we help you migrate to EventStreams? https://www.mediawiki.org/wiki/Topic:Tkjkee2j684hkwc9 -
Is server side filtering, by wiki or arbitrary event field, useful to you? https://www.mediawiki.org/wiki/Topic:Tkjkabtyakpm967t -
Would you like to consume streams other than RecentChanges? https://www.mediawiki.org/wiki/Topic:Tkjk4ezxb4u01a61 (Currently available events are described the event-schemas repository https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki .)
Thanks! - Andrew Otto
[1] https://wikitech.wikimedia.org/wiki/EventStreams [2] https://www.mediawiki.org/wiki/EventStreams/Blog_-_Call_For_Entries [3] https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/... [4] https://www.mediawiki.org/wiki/Talk:EventStreams
Congratulations on the launch of EventStreams.
Andrew Otto wrote:
I did say deprecated! Okay okay, we may never be able to fully deprecate irc.wikimedia.org. It’s used by too many (probably sentient by now) bots out there.
I monitor irc.wikimedia.org and it has proven itself to be highly reliable for many years in production, which is a lot more than can be said for any of these proposed alternatives. I'm glad to hear that irc.wikimedia.org will be left alone. If people want to be nasty and call irc.wikimedia.org deprecated, I suppose that's fine, as long as it remains the functional and dependable (and completely quirky) API it continues to serve as.
MZMcBride
I monitor irc.wikimedia.org and it has proven itself to be highly reliable
for many years in production, which is a lot more than can be said for any of these proposed alternatives .
Great! But, as is, it is impossible for us to restart or migrate the irc.wikimedia.org to new servers without also disrupting client connections. RCStream also has this problem. EventStreams does not.
We have talked about reworking the irc.wikimedia.org backend so that it is easier to maintain, but as of now there is no official internal project to do this. Either way, I think it is safe to say that irc.wikimedia.org will remain online for the foreseeable future.
> is there any working example code, preferably written in Python +1 to what Alex said: https://wikitech.wikimedia.org/wiki/EventStreams#Python
Also, if you want to fix or add more client examples up there, please feel free to do so! :)
> Do you really have a date for decommission of a working service already Yes, as you note, MW devs (and developers in general) have a history of creating new services to deprecate old ones. Sometimes this is done for trivial reasons, other times not (but I’m sure no developer ever thinks their own reasons are trivial ;) ). We also have a history of creating new services and not ever actually shutting old ones off, which creates maintenance headaches for the ops team. To help mitigate those headaches, we agreed to commit to a shut off date for RCStream before launching EventStreams. I think the date is far enough in the future that folks should have time to find any bugs, and for us to work out kinks on our side. Of course, If EventStreams isn't working when the July shut off date rolls around, we won’t just blindly turn off RCStream.
-Ao
On Sat, Feb 25, 2017 at 2:31 PM, MZMcBride z@mzmcbride.com wrote:
Congratulations on the launch of EventStreams.
Andrew Otto wrote:
I did say deprecated! Okay okay, we may never be able to fully deprecate irc.wikimedia.org. It’s used by too many (probably sentient by now) bots out there.
I monitor irc.wikimedia.org and it has proven itself to be highly reliable for many years in production, which is a lot more than can be said for any of these proposed alternatives. I'm glad to hear that irc.wikimedia.org will be left alone. If people want to be nasty and call irc.wikimedia.org deprecated, I suppose that's fine, as long as it remains the functional and dependable (and completely quirky) API it continues to serve as.
MZMcBride
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hello,
Thanks for the info, this means that we need to switch from RCStream, is there any working example code, preferably written in Python or any other sane terminal friendly language (eg. not html or JS) that is capable of connecting to this stream and obtaining at least identicat information from it as we did in case of RCStream?
P.S. I can't help myself but I really have to rant now because I hate how much MW devs love to deprecate stuff and break compatibilty so often just for trivial reasons. I am really glad that instead of using RCStream directly I created that XmlRcs proxy in past, it's gonna save me massive amount of work, because no, I am not gonna deprecate XmlRcs [1].
P.S. 2.0: Do you really have a date for decommission of a working service already even though the replacement is still in early-cradle pre-alpha phase? Is this really the best approach?
1: https://wikitech.wikimedia.org/wiki/XmlRcs
On Thu, 23 Feb 2017 at 22:11, Andrew Otto otto@wikimedia.org wrote:
Hi everyone!
Wikimedia is releasing a new service: EventStreams https://wikitech.wikimedia.org/wiki/EventStreams[1]. This service allows us to publish arbitrary streams of JSON event data to the public. (Psssst: we’re looking for cool new uses https://www.mediawiki.org/wiki/EventStreams/Blog_-_Call_For_Entries[2] to put on an upcoming blog post.)
Initially, the only stream available will be good ol’ RecentChanges https://www.mediawiki.org/wiki/Manual:RCFeed. This event stream overlaps functionality already provided by irc.wikimedia.org and RCStream https://wikitech.wikimedia.org/wiki/RCStream. However, this new service has advantages over these (now deprecated) services.
We can expose more than just RecentChanges. 2.
Events are delivered over streaming HTTP (chunked transfer) instead of IRC or socket.io. This requires less client side code and fewer special routing cases on the server side. 3.
Streams can be resumed from the past. By using EventSource, a disconnected client will automatically resume the stream from where it left off, as long as it resumes within one week. In the future, we would like to allow users to specify historical timestamps from which they would like to begin consuming, if this proves safe and tractable.
I did say deprecated! Okay okay, we may never be able to fully deprecate irc.wikimedia.org. It’s used by too many (probably sentient by now) bots out there. We do plan to obsolete RCStream, and to turn it off in a reasonable amount of time. The deadline iiiiiis July 7th, 2017. All services that rely on RCStream should migrate to the HTTP based EventStreams service by this date. We are committed to assisting you in this transition, so let us know how we can help.
Unfortunately, unlike RCStream, EventStreams does not have server side event filtering (e.g. by wiki) quite yet. How and if this should be done is still under discussion https://phabricator.wikimedia.org/T152731.
The RecentChanges data you are used to remains the same, and is available at https://stream.wikimedia.org/v2/stream/recentchange. However, we may have something different for you, if you find it useful. We have been internally producing new Mediawiki specific events https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki [3] for a while now, and could expose these via EventStreams as well.
Take a look at these events, and tell us what you think. Would you find them useful? How would you like to subscribe to them? Individually as separate streams, or would you like to be able to compose multiple event types into a single stream via an API? These things are all possible.
I asked for a lot of feedback in the above paragraphs. Let’s try and centralize this discussion over on the mediawiki.org EventStreams talk page https://www.mediawiki.org/wiki/Talk:EventStreams[4]. In summary, the questions are:
What RCStream clients do you maintain, and how can we help you migrate to EventStreams? https://www.mediawiki.org/wiki/Topic:Tkjkee2j684hkwc9
Is server side filtering, by wiki or arbitrary event field, useful to you? https://www.mediawiki.org/wiki/Topic:Tkjkabtyakpm967t
Would you like to consume streams other than RecentChanges? https://www.mediawiki.org/wiki/Topic:Tkjk4ezxb4u01a61 (Currently available events are described the event-schemas repository https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki .)
Thanks!
- Andrew Otto
[1] https://wikitech.wikimedia.org/wiki/EventStreams [2] https://www.mediawiki.org/wiki/EventStreams/Blog_-_Call_For_Entries [3] https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/... [4] https://www.mediawiki.org/wiki/Talk:EventStreams
Huggle mailing list Huggle@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/huggle
https://wikitech.wikimedia.org/wiki/EventStreams#Python seems to work if you fix the obvious missing apostrophe and do `pip install sseclient`. The CLI version below also appears to work, and this seems much nicer to me than RCStream.
On 25 February 2017 at 21:25, Petr Bena benapetr@gmail.com wrote:
Hello,
Thanks for the info, this means that we need to switch from RCStream, is there any working example code, preferably written in Python or any other sane terminal friendly language (eg. not html or JS) that is capable of connecting to this stream and obtaining at least identicat information from it as we did in case of RCStream?
P.S. I can't help myself but I really have to rant now because I hate how much MW devs love to deprecate stuff and break compatibilty so often just for trivial reasons. I am really glad that instead of using RCStream directly I created that XmlRcs proxy in past, it's gonna save me massive amount of work, because no, I am not gonna deprecate XmlRcs [1].
P.S. 2.0: Do you really have a date for decommission of a working service already even though the replacement is still in early-cradle pre-alpha phase? Is this really the best approach?
1: https://wikitech.wikimedia.org/wiki/XmlRcs
On Thu, 23 Feb 2017 at 22:11, Andrew Otto otto@wikimedia.org wrote:
Hi everyone!
Wikimedia is releasing a new service: EventStreams https://wikitech.wikimedia.org/wiki/EventStreams[1]. This service allows us to publish arbitrary streams of JSON event data to the public. (Psssst: we’re looking for cool new uses https://www.mediawiki.org/wiki/EventStreams/Blog_-_Call_For_Entries[2] to put on an upcoming blog post.)
Initially, the only stream available will be good ol’ RecentChanges https://www.mediawiki.org/wiki/Manual:RCFeed. This event stream overlaps functionality already provided by irc.wikimedia.org and RCStream https://wikitech.wikimedia.org/wiki/RCStream. However, this new service has advantages over these (now deprecated) services.
We can expose more than just RecentChanges. 2.
Events are delivered over streaming HTTP (chunked transfer) instead of IRC or socket.io. This requires less client side code and fewer special routing cases on the server side. 3.
Streams can be resumed from the past. By using EventSource, a disconnected client will automatically resume the stream from where it left off, as long as it resumes within one week. In the future, we would like to allow users to specify historical timestamps from which they would like to begin consuming, if this proves safe and tractable.
I did say deprecated! Okay okay, we may never be able to fully deprecate irc.wikimedia.org. It’s used by too many (probably sentient by now) bots out there. We do plan to obsolete RCStream, and to turn it off in a reasonable amount of time. The deadline iiiiiis July 7th, 2017. All services that rely on RCStream should migrate to the HTTP based EventStreams service by this date. We are committed to assisting you in this transition, so let us know how we can help.
Unfortunately, unlike RCStream, EventStreams does not have server side event filtering (e.g. by wiki) quite yet. How and if this should be done is still under discussion https://phabricator.wikimedia.org/T152731.
The RecentChanges data you are used to remains the same, and is available at https://stream.wikimedia.org/v2/stream/recentchange. However, we may have something different for you, if you find it useful. We have been internally producing new Mediawiki specific events https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki [3] for a while now, and could expose these via EventStreams as well.
Take a look at these events, and tell us what you think. Would you find them useful? How would you like to subscribe to them? Individually as separate streams, or would you like to be able to compose multiple event types into a single stream via an API? These things are all possible.
I asked for a lot of feedback in the above paragraphs. Let’s try and centralize this discussion over on the mediawiki.org EventStreams talk page https://www.mediawiki.org/wiki/Talk:EventStreams[4]. In summary, the questions are:
What RCStream clients do you maintain, and how can we help you migrate to EventStreams? https://www.mediawiki.org/wiki/Topic:Tkjkee2j684hkwc9
Is server side filtering, by wiki or arbitrary event field, useful to you? https://www.mediawiki.org/wiki/Topic:Tkjkabtyakpm967t
Would you like to consume streams other than RecentChanges? https://www.mediawiki.org/wiki/Topic:Tkjk4ezxb4u01a61 (Currently available events are described the event-schemas repository https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki .)
Thanks!
- Andrew Otto
[1] https://wikitech.wikimedia.org/wiki/EventStreams [2] https://www.mediawiki.org/wiki/EventStreams/Blog_-_Call_For_Entries [3] https://github.com/wikimedia/mediawiki-event-schemas/tree/master/ jsonschema/mediawiki [4] https://www.mediawiki.org/wiki/Talk:EventStreams
Huggle mailing list Huggle@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/huggle
Huggle mailing list Huggle@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/huggle
wikitech-l@lists.wikimedia.org