Hi All,
A reminder that TechCom is hosting an IRC meeting tomorrow (Wednesday
14 November) on RfC: Session storage service interface
<https://phabricator.wikimedia.org/T206010>
This RFC outlines an API for a new multi-master session storage
service for use in multi-DC session management.
The meeting is scheduled for Wednesday 14 November 2pm PST(22:00 UTC,
23:00 CET) in #wikimedia-office.
If you haven't joined a #wikimedia-office meeting before more
information can be found here
<https://meta.wikimedia.org/wiki/IRC_office_hours#How_to_participate>
More information regarding the TechCom RFC process is available here
<https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Processes#RFC_…>
Thanks,
-Kate
--
Kate Chapman
Senior Program Manager, Core Platform
Wikimedia Foundation
kchapman(a)wikimedia.org
This is a more solemn email than is usual. I recognize that this email
reflects my personal view, and if this email is not something that you
appreciate then I invite you to disregard it and write your own email
regarding something that makes you happy or grateful this week.
The 11th of November is commemorated in some parts of the world as
Armistice Day, Remembrance Sunday, or Veterans Day. The year 2018 marks the
100th anniversary of Armistice Day. I would like to take a moment to
reflect on the subject of Armistice Day, and on the roles of Wikimedia --
especially Wikipedia -- in sharing knowledge of history and being a
repository of our collective memory.
"Armistice Day is commemorated... to mark the armistice signed between the
Allies of World War I and Germany at Compiègne, France, for the cessation
of hostilities on the Western Front of World War I, which took effect at
eleven o'clock in the morning—the "eleventh hour of the eleventh day of the
eleventh month" of 1918." [1 <https://en.wikipedia.org/wiki/Armistice_Day>]
World War I was one of the deadliest conflicts in human history, with a
total of approximately 17 million civilian and military deaths. [2
<https://en.wikipedia.org/wiki/World_War_I_casualties>]
I would like to share a story.
John McCrae (photo here
<https://en.wikipedia.org/wiki/File:John_McCrae_in_uniform_circa_1914.jpg>)
was a medical doctor and Canadian soldier during World War I. He wrote a
famous poem, “In Flanders Fields
<https://en.wikipedia.org/wiki/File:In_Flanders_fields_and_other_poems,_hand…>”.
The poem refers to the red poppies that grew over the graves of soldiers
who died in the Second Battle of Ypres in Belgium. There are variants of
the wording of the poem. I quote one of them below.
In Flanders fields the poppies blow
Between the crosses, row on row,
That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.
We are the Dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved and were loved, and now we lie
In Flanders fields.
Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high!
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields."
Here are a few images:
* Poppies in the sunset on Lake Geneva, Montreux, Switzerland
<https://en.wikipedia.org/wiki/File:Poppies_in_the_Sunset_on_Lake_Geneva.jpg>
* Canadian Tomb of the Unknown Soldier
<https://en.wikipedia.org/wiki/File:Canadian_Tomb_of_the_Unknown_Soldier_wit…>
* Remembrance Day 2010 in Ottawa, Canada
<https://en.wikipedia.org/wiki/File:Remembrance_Day_National_War_Memorial_Ot…>
* Memorial of "In Flanders Fields"
<https://en.wikipedia.org/wiki/File:Inflandersfieldslestweforget01.JPG>
In our contemporary world where there are many disputes about history,
resources are limited, and sometimes it is difficult to be optimistic about
human nature, I am especially grateful for Wikipedia's aspiration to be a
place to share neutral, reliable, and verifiable information with an open
license.
Wikimedia has remarkable success at being a collaborative endeavor for the
education and information of humanity. Wikimedia content is collaboratively
developed by thousands of diverse individuals, many of whom are volunteers
and never meet in person. Content that is shared on Wikimedia sites is
viewed by millions of people around the world. Although we sometimes
caution the public that Wikipedia is not a primary source, for many people
Wikipedia seems to be a good starting point, and the references that we
provide allow people to perform their own research regarding history and
many other topics.
Thank you to everyone who documents history on Wikimedia, and to the people
who support this effort behind the scenes. We all benefit from your
generosity to our common memory. By documenting and learning about our
history, I hope that we improve our understanding of ourselves and our
potential, and can make wise decisions about our future.
I close with a poem by Catherine Munro
<https://en.wikipedia.org/wiki/User:CatherineMunro>:
THIS IS AN ENCYCLOPEDIA
One gateway to the wide garden
of knowledge, where lies
The deep rock of our past,
in which we must delve
the well of our future,
The clear water we must leave untainted
for those who come after us,
The fertile earth, in which
truth may grow in bright places,
tended by many hands,
And the broad fall of sunshine,
warming our first steps toward knowing
how much we do not know.
Ever onward,
Pine
( https://meta.wikimedia.org/wiki/User:Pine )
(I sent this to xmldatadumps-l yesterday but just realised that this
might be a more suitable place.)
Hallo,
I'm looking at the data dumps for all Wikipedia languages and noticed
that for some larger wikis, the geo_tags.sql.gz dump file does not
include any geotags found in articles. Is it possible to determine why
this is, and for which languages this is the case?
For example, the geotags dump file for Indonesian (a wiki with
>400,000 articles) is only 7kb large, and all geotags in it are from
user pages, file uploads, or file templates, but not from articles:
https://dumps.wikimedia.org/idwiki/20181020/
Yet it doesn't take much effort to find pages that are geotagged, such
as this one (see the infobox): https://id.wikipedia.org/wiki/London
I realise that there are a number of alternative geotagging
conventions. Does idwiki possibly use a geotagging scheme that is not
supported by some part of this data ingestion/export process? Which
other wikis/languages may fall in this category?
I tried to find the script(s) that populate the geo_tags table from
page content but so far had no luck, as I'm not sufficiently familiar
with WP's software architecture; if someone can point me in the right
direction I'd be happy to investigate myself.
Many thanks!
m.
Sorry for cross posting.
The Technical Wishes team has released a new version of the beta feature
“two-column edit conflict“ on all wikis.[1] In order to make edit conflict
solving even easier than the first beta feature, the new interface allows
to solve edit conflicts line by line.
To try the new interface, activate the beta feature two-column edit
conflict in your preferences. If you were already using the old beta
feature, you’ll get the new interface automatically.
Let us know what you think on the central feedback page.[2]
Thanks,
Michi for the Technical Wishes team
[1] https://www.mediawiki.org/wiki/Help:Two_Column_Edit_Conflict_View
[2] https://www.mediawiki.org/wiki/Help_talk:Two_Column_Edit_Conflict_View
PS: This morning there was a problem after enabling the new version, this
has been fixed. For background on this issue see
https://lists.wikimedia.org/pipermail/wikitech-l/2018-November/091103.html
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
Hi,
Since over one year the Technical Wishes team of Wikimedia Deutschland has
been working on an extension for an easier and less scary handling of edit
conflicts. After a first beta-version (available since beginning of 2017)
this week a new version is being rolled out as part of the branch
wmf/1.33.0-wmf.3. This version introduced a bug which led to the edit
conflict resolution page being shown when using the preview function
anthrough there was no edit conflict. This led to edits being lost in some
cases.
As a reaction to various reports about this (T208840
<https://phabricator.wikimedia.org/T208840>, T209012
<https://phabricator.wikimedia.org/T209012>, T209036
<https://phabricator.wikimedia.org/T209036> and others), we temporarily
disabled the extension completely his morning (2018-11-08 10:49 UTC). The
bug was found and fixed and we plan to reenable the extension again at
19:00 UTC.
To our knowledge only users who had the beta feature enabled were affected.
Cheers,
Michael Schönitzer for the Technical Wishes team
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
Hi all,
I'm Alex Ezell and I'm the Engineering Manager for the Anti-Harassment
Tools team at the WMF. I have some details about user blocking that I'd
like to share.
*tl:dr;*
On a wiki with the new Partial Blocks enabled (currently only testwiki), if
the code is checking User::isBlocked() to determine edit rights, it should
instead check User::isBlockedFrom( Title ). The code could also check
isBlocked()
&& $block->isSitewide(). If it doesn’t, the code may block users that
shouldn’t be blocked.
*More details:*
Recently, the Anti-Harassment Tools team merged code to enable a new
feature called Partial Blocks. This feature lets admins block users from
editing particular pages instead of only being able to block users from the
entire site. It is currently enabled on testwiki.
This means that there are now multiple types of blocks (and more to come in
the future, ie namespace blocks). The specific new types are “partial” as
opposed to “sitewide” and some non-editing types of blocks (send email,
edit talk page, etc.) Previously, a developer could assume that if a user
was “blocked” that meant the user couldn’t do much of anything because that
was a “sitewide” block and the only kind of block. Now, there are more
cases to be concerned about.
Specifically, we’ve seen some extensions using User::isBlocked() and then
assuming that a user can’t edit the particular page that the extension
might be concerned with. User::isBlockedFrom( Title ) with a Title object
will be the more correct way to check because of the possibility that a
user might not be blocked from that particular page. If the code isn’t
concerned with editing, it would be appropriate to use User::isAllowed()
which will determine blocked status by way of User::getRights().
There is also a new method Block::isSitewide() which can help a developer
determine if the block is “sitewide” or some other type. This is useful if
the code doesn’t care about anything but the “sitewide” block type.
We believe that keeping User::isBlocked() in its current state is the safer
way to proceed because in cases where it’s being used incorrectly it would
result in over-enforcing blocks rather than under-enforcing them. A user
who is partially blocked might be treated like a sitewide block by an
extension. That seems safer to us than potentially allowing a user more
freedom than an admin intended with a partial block.
We found at least one extension using User::getRights() in a way that would
over-enforce on a partially blocked user. We created a patch to change how
User::getRights() works
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/471210>. In addition to
checking that a block exists, it will also ensure that the block is a
sitewide block. This will spare the partially blocked user from being
blocked in these cases.
In summary, all MediaWiki code (especially extension code) that is
concerned with checking user blocks should be aware of the distinction
between User::isBlocked() and User::isBlockedFrom( Title ) and use the
appropriate method for the kind of blocking the code is concerned with.
Additionally, using the helper method Block::isSitewide() is handy for
certain usages.
Alex Ezell
Engineering Manager, Anti-Harassment Tools Team (WMF)
https://www.mediawiki.org/wiki/Scrum_of_scrums/2018-11-07
*=2018-11-07=*
== Callouts ==
* Fundraising campaigns
https://meta.wikimedia.org/wiki/CentralNotice/Calendar
* UI Standardization: RelEng, we're in need PHP 7 capable server for public
OOUI demos https://phabricator.wikimedia.org/T206046
* Release Engineering: Wikimedia Release Engineering's 1st Annual Developer
Satisfaction Survey
https://phabricator.wikimedia.org/phame/post/view/126/wikimedia_release_eng…
* Readers Web: 14 year old bug T208909 (Thanks Tilman Bayer and Tim
Starling)
* Readers Web: Alex Hollender's very cool mobile navigation prototype
https://en.wikipedia.org/wiki/User:AHollender_(WMF)/sandbox#Advanced_mobile…
* Readers Web: SEO A/B test launch for page schemas T208755
* Readers Web: page issues A/B test complete T200792
* Follow progress for edit data lake PUBLIC dataset release on labs, will
be available via PRESTO and usable for tools such as quarry:
https://phabricator.wikimedia.org/T208752
== Audiences ==
=== Contributors ===
==== Community Tech ====
* Blocked by:
* Blocking:
* Updates:
**
==== Anti-Harassment Tools ====
* Blocked by:
* Blocking:
* Updates:
**
==== Editing ====
* Blocked by:
* Blocking:
** Updates:
**
==== Growth ====
* Blocked by: Secruity on https://phabricator.wikimedia.org/T207990 and
Services (?) https://phabricator.wikimedia.org/T207329
* Blocking:
* Updates:
**EditorJourney schema is going live on Thursday or Monday
==== Language ====
* Blocked by:
* Blocking:
* Updates:
**
=== Readers ===
==== iOS native app ====
* Blocked by:
* Blocking:
* Updates:
**In bug-fixing mode for 6.1.1 (
https://phabricator.wikimedia.org/tag/ios-app-v6.1.1-narwhal-on-a-magic-pum…),
release probably next week. 6.1.1 has fixes for login, event logging and a
bunch of UI tweaks.
**Next up, we'll work on 6.2 (
https://phabricator.wikimedia.org/tag/ios-app-v6.2/); enabling wikidata
description editing on English, user testing wikidata description editing,
logging article reading.
==== Android native app ====
* Blocked by:
* Blocking:
* Updates:
**Released bugfix release to production; continuing to refine interface
based on user feedback.
==== Readers Web ====
* Blocked by:
* Blocking:
* Updates:
** Mobile website (MinervaNeue / MobileFrontend):
*** Invest in the MobileFrontend & MinervaNeue frontend architecture
https://www.mediawiki.org/wiki/Reading/Web/Projects/Invest_in_the_MobileFro…
**** Collapse mobile.search.util into mobile.startup ResourceLoader module
T206027
**** Increase test coverage for non-View files with 0% coverage T206698
**** MobileFrontend pre-commit hooks don't work on Windows T208143
**** Add TopicTitleList component T173534
**** Remove unused MW configs loaded on desktop pageviews T186062
*** Page issues
https://www.mediawiki.org/wiki/Reading/Web/Projects/Mobile_Page_Issues
**** A/B test complete; wrapping up subtasks and performing data analysis
T200792
*** Allow users to change their mobile skin preference T173527
*** Collect data on users who are switching from mobile to desktop T208457
*** Maintenance and bug fixes T205359 T193061 T208340
** SEO:
*** Server side split test T206868
*** Enable page schemas on the beta cluster T208763
*** Old page_random values are nonuniformly distributed T208909 (Thanks
Tilman and Tim Starling)
*** Launch A/B test for sameAs property T208755
** PDF rendering (Proton)
https://www.mediawiki.org/wiki/Reading/Web/PDF_Functionality
*** Remaining work tracked in T186748
*** Rewrite Queue to Promises T204055
*** Advanced mobile contributions
https://www.mediawiki.org/wiki/Reading/Web/Advanced_mobile_contributions
**** Design working on navigation prototype:
https://en.wikipedia.org/wiki/User:AHollender_(WMF)/sandbox#Advanced_mobile…
** Design working on the community health dashboard project and
interviewing QA engineers.
** Management interviewing for apps engineering manager role.
** Post-WMTechConf/Offsite follow-ups.
==== Readers Infrastructure ====
* Waiting on Kartographer/JsonConfig security review (not immediately
blocked)
==== Multimedia ====
* Updates
** First release (multi-lingual captions) live on beta commons ... not
announcing it yet because we've bugs to work through
** Working on phase 2 (statements) and hiring
==== Parsing ====
* Blocked by: None
* Blocking: None
* Updates: None
==== UI Standardization ====
* Blocked by:
* Blocking:
* Updates:
**
== Technology ==
=== Analytics ===
* Updates:
**
** Met with security and SRE to plan best architecture to put edit data
lake data on labs via presto cluster. This is adataset that quarry can use
to answer questions that are just to hard for labs replicas, see
conversations: https://phabricator.wikimedia.org/T207321
** Will produce design document and cloud team can also take a look
** We are about to update cloudera distro in cluster to 5.15
** Research and analytics collaborating as time permits in classifier to
better identify bots in pageview data
** Reactive: When communication works: we had an issue that “looked” like
data loss but was actually request for WebP assets being double logged. We
were able to figure out the connection via updates on SRE meeting. See
ticket: https://phabricator.wikimedia.org/T208752 kudos to everyone for
fast responses.
=== Cloud Services ===
* Blocked by:
* Blocking:
* Updates:
**
=== Fundraising Tech ===
* Blocked by:
* Blocking:
* Updates:
**Still tightening up some stuff in CentralNotice
**CiviCRM UI fixes
**Dealing with PayPal inconsistencies
**Fixing authentication for internal dashboard
=== MediaWiki Core Platform ===
* Blocked by:
* Blocking:
* Updates:
** Session Management Service RFC is available
** Team is beginning the process of extracting actionables from TechConf
** PHP7 work ongoing
** Automating the process of creating the MediaWiki tarball
=== Performance ===
* Blocked by:
**
* Blocking:
**
* Updates:
**
=== Release Engineering ===
* Blocked by:
* Blocking:
** Train Health:
*** Last week: 1.33.0-wmf.2 deployment blockers
https://phabricator.wikimedia.org/T206656
**** wmf.2 was late last week due to an odd HHVM issue:
https://phabricator.wikimedia.org/T208549
*** This week: 1.33.0-wmf.3 deployment blockers
https://phabricator.wikimedia.org/T206657
**** All systems nominal.
*** Thanks James_F and Roan for wmf.2 work! (and for deploys this morning)
* Updates:
** Wikimedia Release Engineering's 1st Annual Developer Satisfaction Survey
https://phabricator.wikimedia.org/phame/post/view/126/wikimedia_release_eng…
*** Please take it! :)
=== Research ===
* Blocked by: None
* Blocking: None
* Updates:
** Team is at CSCW 2018.
** Offsite in the second part of the week.
=== Scoring Platform ===
* Blocked by:
* Blocking:
* Updates:
**Amir pushed out big changes to ORES -- CODFW is having poolcounter
slowness -- Implementing a better poolcounter restart strategy withtimeouts
(deployed on 11/5)
**Celery 4 shows a huge reduction latency for precached (single revision
scoring, 300% faster), but increase in latency for multi-revision scoring
*** Celery4 is now able to recover from redis outages
=== Search Platform ===
* Blocked by: None
* Blocking: None
* Updates:
** Added syntax for searching Wikidata for any qualifier value:
https://phabricator.wikimedia.org/T207016
** Made code to calculate autocomplete examination probabilities for
Wikidata: https://phabricator.wikimedia.org/T205348
** Made some updates to WDQS network and logging configurations to improve
stability under load: https://phabricator.wikimedia.org/T206105https://phabricator.wikimedia.org/T207834https://phabricator.wikimedia.org/T207843related to
https://wikitech.wikimedia.org/wiki/Incident_documentation/20181024-WDQS
** Changed throttling setting for WDQS internal cluster:
https://phabricator.wikimedia.org/T206648
** Added support for millisecond timestamps to WDQS Kafka poller:
https://phabricator.wikimedia.org/T207873
** Working on ES 6 upgrade: https://phabricator.wikimedia.org/T183282
** Working on running multiple Elastic instances on the same hardware:
https://phabricator.wikimedia.org/T193654
** Working on query parsing refactoring:
https://phabricator.wikimedia.org/T185108
=== Security ===
* Blocked by:None
* Blocking:https://phabricator.wikimedia.org/T144467#3932037
**T200279 Security review for WikibaseMediaInfo
**T207798 Security review for GrowthExperiments extension
* Updates:T207798 is complete
**T200279 and T144467 are in process
=== Services ===
* Blocked by:
* Blocking:
* Updates:
**
=== Site Reliability Engineering ===
* Blocked by:
* Blocking:
* Updates:
**
== Wikidata ==
* Blocked by:
* Blocking:
* Updates:
**Thanks to people helping out with WDQS stability issues!
**Working on exciting new things in the UI -> will request a service from
SRE at some point in the near future
**Several collegues from the team got deployment access rights -> thanks
for support to SRE and RelEng!
== German Technical Wishlist ==
* Blocked by:
* Blocking:
* Updates:
**
== Multi-Content Revisions ==
* Blocked by:
* Blocking:
* Updates:
** Enabled “read new” on all wikis
** Approved RFC: Spec for representing multiple content objects per
revision (MCR) in XML dumps
** Working on backend support for editing slots other than main slot
Reminder: Technical Advice IRC meeting again **Wednesday at 4-5 pm UTC AND
11-12 pm UTC** on #wikimedia-tech.
The Technical Advice IRC Meeting (TAIM) is a weekly support event for
volunteer developers. Every Wednesday, two full-time developers are
available to help you with all your questions about Mediawiki, gadgets,
tools and more! This can be anything from "how to get started" over "who
would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting:
https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
Hope to see you there!
Michi (for the Technical Advice IRC Meeting crew)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
The Search Platform Team
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> will be holding
office hours the first Wednesday of each month, starting next week. Come
ask us anything about Wikimedia search!
We’re particularly interested in:
* Opportunities for collaboration—internally or externally to the Wikimedia
Foundation
* Challenges you have with on-wiki search, in any of the languages we
support
But we're happy to talk about anything search-related.
Details for our next meeting:
Date: Wednesday, November 7th, 2018
Time: 16:00 GMT / 08:00 PST / 11:00 EST / 17:00 CET
Google Meet link: https://meet.google.com/vyc-jvgq-dww
*N.B.:* Google Meet System Requirements
<https://support.google.com/meet/answer/7317473>
Hope to see you there!
—Trey
Trey Jones
Sr. Software Engineer, Search Platform
Wikimedia Foundation
