Hi everyone!
Wikimedia is releasing a new service today: EventStreams
<https://wikitech.wikimedia.org/wiki/EventStreams>. This service allows us
to publish arbitrary streams of JSON event data to the public. Initially,
the only stream available will be good ol’ RecentChanges
<https://www.mediawiki.org/wiki/Manual:RCFeed>. This event stream overlaps
functionality already provided by irc.wikimedia.org and RCStream
<https://wikitech.wikimedia.org/wiki/RCStream>. However, this new service
has advantages over these (now deprecated) services.
1.
We can expose more than just RecentChanges.
2.
Events are delivered over streaming HTTP (chunked transfer) instead of
IRC or socket.io. This requires less client side code and fewer special
routing cases on the server side.
3.
Streams can be resumed from the past. By using EventSource, a
disconnected client will automatically resume the stream from where it left
off, as long as it resumes within one week. In the future, we would like
to allow users to specify historical timestamps from which they would like
to begin consuming, if this proves safe and tractable.
I did say deprecated! Okay okay, we may never be able to fully deprecate
irc.wikimedia.org. It’s used by too many (probably sentient by now) bots
out there. We do plan to obsolete RCStream, and to turn it off in a
reasonable amount of time. The deadline iiiiiis July 7th, 2017. All
services that rely on RCStream should migrate to the HTTP based
EventStreams service by this date. We are committed to assisting you in
this transition, so let us know how we can help.
Unfortunately, unlike RCStream, EventStreams does not have server side
event filtering (e.g. by wiki) quite yet. How and if this should be done
is still under discussion <https://phabricator.wikimedia.org/T152731>.
The RecentChanges data you are used to remains the same, and is available
at https://stream.wikimedia.org/v2/stream/recentchange. However, we may
have something different for you, if you find it useful. We have been
internally producing new Mediawiki specific events
<https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema…>
for a while now, and could expose these via EventStreams as well.
Take a look at these events, and tell us what you think. Would you find
them useful? How would you like to subscribe to them? Individually as
separate streams, or would you like to be able to compose multiple event
types into a single stream via an API? These things are all possible.
I asked for a lot of feedback in the above paragraphs. Let’s try and
centralize this discussion over on the mediawiki.org EventStreams talk page
<https://www.mediawiki.org/wiki/Talk:EventStreams>. In summary, the
questions are:
-
What RCStream clients do you maintain, and how can we help you migrate
to EventStreams? <https://www.mediawiki.org/wiki/Topic:Tkjkee2j684hkwc9>
-
Is server side filtering, by wiki or arbitrary event field, useful to
you? <https://www.mediawiki.org/wiki/Topic:Tkjkabtyakpm967t>
-
Would you like to consume streams other than RecentChanges?
<https://www.mediawiki.org/wiki/Topic:Tkjk4ezxb4u01a61> (Currently
available events are described here
<https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema…>
.)
Thanks!
- Andrew Otto
Hi Marko,
I looked at the document and I would have multiple patches to it; alas, I
don't have time to go through it with the needed attention right now
because (as anticipated before the quarter) I have zero time to spend on
this.
Ops have some experience in running a cluster orchestration system in
production (for toollabs) and I have thought about it for quite some time
now; I have some ideas on how things should be done to have a decent,
manageable "elastic" environment with advantages for developers; I would
love to integrate your document with ideas/a more general vision about
production; this is probably not going to happen for at least one month
though.
Can we hold on before we declare this document to be "definitive"?
Also, can we stop calling it a "container-based" infrastructure? :) I
seriously think containers are little more than an implementation detail of
the general vision.
Cheers,
Giuseppe
On Wed, Feb 15, 2017 at 11:28 PM, Marko Obrovac <mobrovac(a)wikimedia.org>
wrote:
> Hello,
>
> In light of the upcoming annual planning for the joint technology goal of
> having a shared container-based infrastructure, the Services team has
> started collecting requirements for the platform in terms of development,
> testing and operation of services (together with some other considerations
> like automation and configuration management)~[1]. Please take a look at
> the document and add/remove/improve/suggest as you see fit. Note that the
> document is to be considered only a draft at this point.
>
> Cheers,
> Marko
>
> [1] https://docs.google.com/a/wikimedia.org/document/d/
> 1QsCVooqxkeE6tKYTxgoRvRdK2M3tDk4UyvmnHJrdag4/edit?usp=sharing
>
> --
> Marko Obrovac, PhD
> Senior Services Engineer
> Wikimedia Foundation
>
> _______________________________________________
> Ops mailing list
> Ops(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/ops
>
>
--
Giuseppe Lavagetto, Ph.d.
Senior Technical Operations Engineer, Wikimedia Foundation
Hello,
In the second half of January 2017, the version of Node running in WMF
production has been updated from v4 to v6~[1]. Parsoid, RESTBase, AQS and
the services running on the SCB cluster (Graphoid, Mathoid, Mobile Content
Service, to mention just a few) are now running on Node v6. The only
outstanding Node service which is still running Node v4 is Maps, but we are
making good progress on moving it to Node v6 and expect it to happen
soon~[2].
In the case of services running in WMF production, we have seen a slight
increase in performance as well as a substantial decrease in memory
consumption across the board, which improves the stability of our services.
You can read more about the experience of the update and its results in the
post published on Wikimedia's blog~[3].
Big thanks to all of the people that helped to make it happen!
Happy Friday,
Marko && the Services Team
[1] https://phabricator.wikimedia.org/T149331
[2] https://phabricator.wikimedia.org/T150354
[3] https://blog.wikimedia.org/2017/02/17/node-6-wikimedia/
--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation
Hi,
recently I saw some phabricator tickets that people complain about
having difficulties to setup the math extension and configure it to
get it working with restbase [1,2,3 ...].
I wonder if that also effects other service dependent extensions such
as cite or VE?
I am aware of the medium time goal to find a good and convinient
solution for this problem.
However, I think we should find a intermediate solution and provide
some documentation for average skilled admins of private wikis.
At least for Extension:Math that is relatively straigt forward, (even
though settig up a restbase instance seems like a slight overkill for
a wiki like pokewiki http://pokewiki.de/ that use only some formulae.)
The most difficult point during the setup is the creation of the
swagger config [4]. One of my students suggested that we should try
http://editor.swagger.io to simplyfiy the creation of the swagger
config.
Do you have experience with editing the services config files with
that editor, or do you use a different editor?
Best
physikerwelt
[1] https://phabricator.wikimedia.org/T155201
[2] https://phabricator.wikimedia.org/T151311
[3] https://phabricator.wikimedia.org/T119817
[4] https://raw.githubusercontent.com/physikerwelt/restbase-config-fse/master/f…