Hi all,
For all Hive users using stat1002/1004, you might have seen a deprecation
warning when you launch the hive client - that claims it's being replaced
with Beeline. The Beeline shell has always been available to use, but it
required supplying a database connection string every time, which was
pretty annoying. We now have a wrapper
<https://github.com/wikimedia/operations-puppet/blob/production/modules/role…>
script
setup to make this easier. The old Hive CLI will continue to exist, but we
encourage moving over to Beeline. You can use it by logging into the
stat1002/1004 boxes as usual, and launching `beeline`.
There is some documentation on this here:
https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Beeline.
If you run into any issues using this interface, please ping us on the
Analytics list or #wikimedia-analytics or file a bug on Phabricator
<http://phabricator.wikimedia.org/tag/analytics>.
(If you are wondering stat1004 whaaat - there should be an announcement
coming up about it soon!)
Best,
--Madhu :)
Hello!
The Analytics team would like to announce that we have migrated the
reportcard to a new domain:
https://analytics.wikimedia.org/dashboards/reportcard/#pageviews-july-2015-…
The migrated reportcard includes both legacy and current pageview data,
daily unique devices and new editors data. Pageview and devices data is
updated daily but editor data is still updated ad-hoc.
The team is working at this time on revamping the way we compute edit data
and we hope to be able to provide monthly updates for the main edit metrics
this quarter. Some of those will be visible in the reportcard but the new
wikistats will have more detailed reports.
You can follow the new wikistats project here:
https://phabricator.wikimedia.org/T130256
Thanks,
Nuria
We’re glad to announce the release of an aggregate clickstream dataset extracted from English Wikipedia
http://dx.doi.org/10.6084/m9.figshare.1305770 <http://dx.doi.org/10.6084/m9.figshare.1305770>
This dataset contains counts of (referer, article) pairs aggregated from the HTTP request logs of English Wikipedia. This snapshot captures 22 million (referer, article) pairs from a total of 4 billion requests collected during the month of January 2015.
This data can be used for various purposes:
• determining the most frequent links people click on for a given article
• determining the most common links people followed to an article
• determining how much of the total traffic to an article clicked on a link in that article
• generating a Markov chain over English Wikipedia
We created a page on Meta for feedback and discussion about this release: https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_clickstream <https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_clickstream>
Ellery and Dario
Hi everyone!
Wikimedia is releasing a new service today: EventStreams
<https://wikitech.wikimedia.org/wiki/EventStreams>. This service allows us
to publish arbitrary streams of JSON event data to the public. Initially,
the only stream available will be good ol’ RecentChanges
<https://www.mediawiki.org/wiki/Manual:RCFeed>. This event stream overlaps
functionality already provided by irc.wikimedia.org and RCStream
<https://wikitech.wikimedia.org/wiki/RCStream>. However, this new service
has advantages over these (now deprecated) services.
1.
We can expose more than just RecentChanges.
2.
Events are delivered over streaming HTTP (chunked transfer) instead of
IRC or socket.io. This requires less client side code and fewer special
routing cases on the server side.
3.
Streams can be resumed from the past. By using EventSource, a
disconnected client will automatically resume the stream from where it left
off, as long as it resumes within one week. In the future, we would like
to allow users to specify historical timestamps from which they would like
to begin consuming, if this proves safe and tractable.
I did say deprecated! Okay okay, we may never be able to fully deprecate
irc.wikimedia.org. It’s used by too many (probably sentient by now) bots
out there. We do plan to obsolete RCStream, and to turn it off in a
reasonable amount of time. The deadline iiiiiis July 7th, 2017. All
services that rely on RCStream should migrate to the HTTP based
EventStreams service by this date. We are committed to assisting you in
this transition, so let us know how we can help.
Unfortunately, unlike RCStream, EventStreams does not have server side
event filtering (e.g. by wiki) quite yet. How and if this should be done
is still under discussion <https://phabricator.wikimedia.org/T152731>.
The RecentChanges data you are used to remains the same, and is available
at https://stream.wikimedia.org/v2/stream/recentchange. However, we may
have something different for you, if you find it useful. We have been
internally producing new Mediawiki specific events
<https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema…>
for a while now, and could expose these via EventStreams as well.
Take a look at these events, and tell us what you think. Would you find
them useful? How would you like to subscribe to them? Individually as
separate streams, or would you like to be able to compose multiple event
types into a single stream via an API? These things are all possible.
I asked for a lot of feedback in the above paragraphs. Let’s try and
centralize this discussion over on the mediawiki.org EventStreams talk page
<https://www.mediawiki.org/wiki/Talk:EventStreams>. In summary, the
questions are:
-
What RCStream clients do you maintain, and how can we help you migrate
to EventStreams? <https://www.mediawiki.org/wiki/Topic:Tkjkee2j684hkwc9>
-
Is server side filtering, by wiki or arbitrary event field, useful to
you? <https://www.mediawiki.org/wiki/Topic:Tkjkabtyakpm967t>
-
Would you like to consume streams other than RecentChanges?
<https://www.mediawiki.org/wiki/Topic:Tkjk4ezxb4u01a61> (Currently
available events are described here
<https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema…>
.)
Thanks!
- Andrew Otto
Hi,
Does anyone know of a way to look up the top editors for a certain
namespace (like "Module") across all Wikimedia sites?
I'm asking as I'm wondering how to get more aware of developer activity
outside of Wikimedia Git/Gerrit.
Thanks for any ideas (or pointing out a better place to ask)!
Cheers,
andre
--
Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/
Team:
This is just a friendly remainder regarding the fact that if you want to
preserve data (of non private nature) in eventlogging beyond the 90 period
you have to let us know via ticket or similar. The info per schema should
be available on each schema's talk page.
Purging info can be found here:
https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/Data_retention_a…
Thanks,
Nuria
Hi all!
The server that hosts s1, s2 analytics-slaves (db1047) needs a little
love. It’ll be offline for a little while on Wed May 3, starting at 10am
EST time.
https://phabricator.wikimedia.org/T159266
-Ao
Tracking task: T148506
Should last approximately 10/15 mins from now and it will affect all the
Analytics websites (Yarn, Hue, Pivot, etc..).
Please reach out to me (elukey) on IRC if you have further questions.
Luca