-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi,
CI now supports generating PHP test coverage reports for extensions
after a commit is merged. It's already running for some extensions:
<https://doc.wikimedia.org/cover-extensions/>
Hopefully these reports will encourage people to increase test
coverage :-)
I've also written documentation at
<https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Generat
ing_PHP_test_coverage_for_a_MediaWiki_extension>
which explains how to generate code coverage results locally, and then
write a patch for the CI configuration to have jenkins run it.
The bug for this was <https://phabricator.wikimedia.org/T71685>,
thanks to the people who helped out with testing this over the past
few weeks.
- -- Kunal / Legoktm
-----BEGIN PGP SIGNATURE-----
iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAlpnx7cXHGxlZ29rdG1A
bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KLd1A//S2TsTz7Ius1MVnsMv74q1h9O
z+zTrZ/LyqhVButL3/sFqwqpaDS25gA+ugyKpT9+h8bymUX5APHC4LPa3IZyNI/P
Y/lLNU7HiJsB0RUg2Qq/LrKmeX5yuQbrA7zeCpDzvCmiOEpd7VkMLLSnIao0nJ7h
tf3J7WersuZQBsn9mkuNCsGrTl3n7/Fo3HiL7LXpLlwpTYDLTEV+M231dKvGMFdx
7ui0LFWprpzGWO9eTxuJvVGy6mYonN4GRieFmzi6vyvr65xPEtCXXYx7dfTNUDml
C2o4WGWbYvPxwhSeFpf1W6cdGd0dkC1gqYYCCKlqXDveQNIvpkLcsr0S5/hWZKyx
djgQWR+Rue4Bq3EQH1AjqzrP6HQNElP7a2KjELHBA7ihGlnXaYeSO5QNC5DJ+nvy
/062SVYANbCRIjjjSvl+mPeaXwoBoIX8z4Z2qm/0nCaUrGo2HeQ1O7Mw7Ocf0+od
nzjIjI0CHcW1i5afz0f8Kc8XByRvhccWKRDcFsw1m64xVP6Aazn6keR3ZJtGkQS7
hMRC0eOsDEtc8uNW2fVYReyFZBo1xiAprQRd2y46fv0JDqTRGLfGihIVLPFQiVqN
BE+kcro/i1TCAVOVTEu56kYhOE+HCtwHVRZA/P46Z4d5+/3RLDxS39/w4peRgd2K
+IEOgVfACbz2Mi8Xg/Y=
=l2yo
-----END PGP SIGNATURE-----
TL;DR
-----
On January 31, 2018, on ru.wp, sv.wp, fi.wp and he.wp, we are going to
turn off Tidy and switch to the Remex HTML5 parsing library.
Besides those, another 200+ wikis will also be switched away from Tidy
on that day. You can find the list of such wikis at T184656 [1].
Do any of you belong to or know someone active in these communities?
While we've also announced this on Tech News, based on our previous
experience, since we don't anticipate the change to be ground-breaking
for these communities, we think that "spamming" the village pumps
may be not so effective and so we'd appreciate your help in assuring
these wikis that they can contact us @ mw:Help_talk:Extension:Linter [2]
if needed, and that there's plenty of documentation to help with
Linter fixes at mw:Help:Extension:Linter [3].
Thanks!
Background
----------
In July 2017, we announced [4] our intention to replace Tidy with a
HTML5-based solution on the Wikimedia cluster by the end of June 2018
at the latest. Please refer to that original posting for specifics of
the project and why we are replacing Tidy.
Status of Tidy replacement
--------------------------
Over the last 3 months, we have now replaced Tidy with RemexHTML
on mediawiki, testwiki, nowiki, fawiki, itwiki, dewiki and 170 other
small wikis. [5]
We have approached ruwiki, svwiki, fiwiki, hewiki for replacement this
month based on remaining linter errors and progress those wikis have
been making. We expect to approach other medium and large wikis for
replacement next month.
In addition, for any wiki that has < 10 linter errors in any high-priority
category, we will be replacing Tidy with RemexHTML. T184656 [1] has the
list of wikis that will see this change (this list includes wikis that
have already had Tidy replaced in December).
To be clear, if we notice problems (or if the wiki requests it), we will
revert the change after identifying the source of the problem. If you
notice any incorrect rendering, you can use ?action=parsermigration-edit
to identify if the switch from Tidy actually caused it.
Status of linter fixes
----------------------
We have been publishing weekly stats [6] of changes to linter counts
which shows how wikis have been progressing with making linter fixes.
Based on what we've observed, of the 38 largest wikis, besides the one
that have Tidy replaced already or will get it replaced this month, most
other wikis seem to be making progress, albeit at different rates.
idwiki, viwiki, jawiki, and rowiki haven't seen a lot of activity yet.
Results from pixel diff tests
-----------------------------
We have also been doing weekly test runs to calculate pixel diffs on
about 70K pages which we have sampled from over 50 wikis. To do this,
we generate a screenshot of a page with Tidy and one with RemexHTML,
and compare the renderings while ignoring vertical whitespace shifts.
We generate a numeric score for the diff that tries to be reflective
of the magnitude of differences we are seeing.
Thanks to fixes to pages and our testing infrastructure to more
accurately detect differences, between July 2017 and January 2018,
the percentage of pages that rendered with only vertical whitespace
shifts increased from 91.9% to 94.6%. Similarly, the percentage of
pages that rendered with pixel perfect accuracy went up from
63.2% to 68.3%. For technical reasons related to the testing setup
that I will skip here, 100% for either metric is not achievable.
Summary
-------
Overall, at the end of January, about 400 of Wikimedia's wikis will
have replaced Tidy. This includes 7 of the largest wikis.
Linter fixes are also happening on lots of wikis, but some large wikis
could pick up the pace.
We still expect to replace Tidy on all wikis by end of June 2018,
and your cooperation and help with fixing pages identified by the
Linter tool is greatly appreciated.
Subbu,
Manager and Technical Lead,
Parsing Team @ the WMF.
1. https://phabricator.wikimedia.org/T184656
2. https://www.mediawiki.org/wiki/Help_talk:Extension:Linter
3. https://www.mediawiki.org/wiki/Help:Extension:Linter
4. https://www.mediawiki.org/wiki/Talk:Parsing/Replacing_Tidy/FAQ
5. https://phabricator.wikimedia.org/T175706
6. https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/Linter/Stats
Hii i am interested to work on education dashboard project . I am from
IIT Jammu CSE and have experience in reactjs javascript and nodejs .So
please can anyone tell how should i proceed on this
Hello all!
Welcome to the TechCom Radar email, Heathrow Airport edition.
Here are the minutes from this week's meeting:
* Developer Summit coming up! Most TechCom members will be there. The goal is to
provide input to the strategy process from a technical perspective. The session
on “Evolving the MediaWiki architecture” strongly aligned with TechCom’s scope.
It is being prepared on phabricator: <https://phabricator.wikimedia.org/P6622>,
comments welcome!
* HtmlCacheUpdate jobs migrated to Kafka based ChangeProp mechanism. Looking good!
* Ongoing issue: Stack overflow when redis is down
<https://phabricator.wikimedia.org/T185055>. Probably caused by upstream issue
in HHVM’s redis extension.
* Approved after Last Call: Introduce PlatformSettings.php
<https://phabricator.wikimedia.org/T182020>
* Not approved after Last Call, due to ongoing discussion about scalability and
usability issues: Page Creation Log <https://phabricator.wikimedia.org/T12331>
* New RFC: Don’t log autopatrol events <https://phabricator.wikimedia.org/T184485>
* Active discussion: insecure anon edit token
<https://phabricator.wikimedia.org/T40417>
* No RFC discussion on January 24th due to everyone being at the Summit or
All-Hands. Discussion topic for January 31st to be announced. Tentative:
MediaWiki support for Composer equivalent for JavaScript packages
<https://phabricator.wikimedia.org/T107561>
You can also find our meeting minutes at
<https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes>
See also the TechCom RFC board
<https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>.
--
Daniel Kinzler
Principal Platform Engineer
Wikimedia Deutschland
Gesellschaft zur Förderung Freien Wissens e.V.
Hi! We have a video to share from the December 2017 Readers monthly
meeting. Dmitry Brant from the Wikimedia Foundation's Apps team shows
updates to the Wikipedia for Android app on Feed customization, the
Randomizer, and a Black theme for AMOLED devices.
https://www.youtube.com/watch?v=Yeh8n4asU8Mhttps://commons.wikimedia.org/wiki/File:Readers-monthly-december-2017-wikip…
*About the Randomizer*
The team redesigned the Randomizer function to be easier and more fun to
interact with. We hope it will give users hours of enjoyment and help them
discover Wikipedia content they might otherwise never have known existed.
Credit to design, engineering, and product management!
-Adam
At the Dev Summit, Birgit Müller and I will run a session on Growing the
MediaWiki Technical Community. If you're attending, we hope you will
consider joining us.
Everyone (attending the Dev Summit or not) is welcome and encouraged to
participate at https://phabricator.wikimedia.org/T183318 (please comment
there, rather than by email).
We are discussing the following questions:
* What would allow you to develop and plan your software more efficiently?
* What would make software development more fun for you?
* What other Open Source communities do we share interests with?
* How can we change our processes to take technical debt more seriously?
"Develop" means any kind of work on a software system, including design,
documentation, etc.
Our topics are:
* Better processes and project management practices, integrating all
developers and allowing them to work more efficiently
* Building partnerships with other Open Source communities on shared
interests (e.g. translation, audio, video)
* Reducing technical debt
Matt Flaschen
Howdy Wikitechnorati,
(And thank you for patience with me cross-posting if you're on other lists.)
I'm writing to invite your input on the following Phabricator task ahead of
next week's Wikimedia Developer Summit 2018 [1] session.
Knowledge as a Service
https://phabricator.wikimedia.org/T183315
The purpose [2] of the Wikimedia Developer Summit 2018 sessions is to
provide guidance for Phase 2 of the Movement Strategic Direction [3] on
buildout of technology capabilities. We'd really love your thoughts to help
set context for our session next week, as Knowledge as a Service is a
primary consideration in the Movement Strategic Direction.
What is Knowledge as a Service? Its essence is about information
architecture approaches and the necessary software that will ultimately
allow content consumption and creation to radiate to new and different
types of interfaces and devices in addition to browser-based approaches. As
you review position papers from attendees [4] you'll notice that the way
they (myself included) think about best solving this is through a heavy
emphasis on technology that makes it easier to better structure information
and its metadata for re-use, remixing, and querying.
What might this mean? Does it mean we should build Wikimedia software in an
API- and metadata-first manner following industry standards compatible with
content structuration? Does it mean weaving our existing structured and
semi-structured data technologies together? How do we build technology that
can ensure successful collaboration between communities on increasingly
structured and interdependent information sources? And how can we ensure
the tech will bolster growth of multilingual and multimedia content
creation and consumption?
I've copied some of the essential material from the Movement Strategic
Direction concerning Knowledge as a Service so you have it here. We would
appreciate your input and hope you will subscribe to the Phabricator task
to contribute and follow along as we explore this topic.
https://phabricator.wikimedia.org/T183315
The following content is copied from
https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/2017/Direction :
Knowledge as a service: To serve our users, we will become a platform that
serves open knowledge to the world across interfaces and communities. We
will build tools for allies and partners to organize and exchange free
knowledge beyond Wikimedia. Our infrastructure will enable us and others to
collect and use different forms of free, trusted knowledge.
...
As technology spreads through every aspect of our lives, Wikimedia's
infrastructure needs to be able to communicate easily with other connected
systems.
...
As a platform, we need to transform our structures to support new formats,
new interfaces, and new types of knowledge. We have a strategic opportunity
to go further and offer this platform as a service to other institutions,
beyond Wikimedia. In a world that is becoming more and more connected,
building the infrastructure for knowledge gives others a vested interest in
our success. It is how we ensure our place in the larger network of
knowledge, and become an essential part of it. As a service to users, we
need to build the platform for knowledge or, in jargon, provide knowledge
as a service.
...
Knowledge as a service: A platform that serves open knowledge to the world
across interfaces and communities
Our openness will ensure that our decisions are fair, that we are
accountable to one another, and that we act in the public interest. Our
systems will follow the evolution of technology. We will transform our
platform to work across digital formats, devices, and interfaces. The
distributed structure of our network will help us adapt to local contexts.
...
We will build tools for allies and partners to organize and exchange free
knowledge beyond Wikimedia.
We will continue to build the infrastructure for free knowledge for our
communities. We will go further by offering it as a service to others in
the network of knowledge. We will continue to build the partnerships that
enable us to develop knowledge we can't create ourselves.
...
Our infrastructure will enable us and others to collect and use different
forms of free, trusted knowledge.
We will build the technical infrastructures that enable us to collect free
knowledge in all forms and languages. We will use our position as a leader
in the ecosystem of knowledge to advance our ideals of freedom and
fairness. We will build the technical structures and the social agreements
that enable us to trust the new knowledge we compile. We will focus on
highly structured information to facilitate its exchange and reuse in
multiple contexts.
Thank you.
-Adam
[1] https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2018
[2]
https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2018/Purpose_and_…
[3]
https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/2017/Direction
[4] https://wikifarm.wmflabs.org/devsummit/index.php/Session:10
https://www.mediawiki.org/wiki/Scrum_of_scrums/2018-01-17#Reading_Web
= *2018-**0**1-17* =
== Callouts ==
* Grafana will be migrated to support native LDAP on Feb 12 2018. This
means the eventual demise for grafana-admin.wikimedia.org. Announcement to
be posted to engineering@ and wikitech@.
https://phabricator.wikimedia.org/T170150
== Audiences ==
=== Readers ===
==== iOS native app ====
* Blocked by:
* Blocking:
* Updates:
**Continuing work on 5.8 - synced reading lists
==== Android native app ====
* Blocked by: weird RESTBase cache behavior:
https://phabricator.wikimedia.org/T184833
* Blocking:
* Updates:
** On track to release update with performance enhancements for Reading
Lists (and "default" list, for feature parity with iOS)
** Continuing testing of reading list syncing.
==== Reading Web ====
* Blocked by:
** Release engineering - need to setup some CI on pdf service
https://phabricator.wikimedia.org/T179552
Services - soon we will be looking at how to apply different styles to
pdf service (https://phabricator.wikimedia.org/T181680) and how to use
firejail to limit resources used by chromium render service? (
https://phabricator.wikimedia.org/T180626) - we may need some guidance.
* Blocking:
* Updates:
**We will be releasing a revamp of the mobile settings page. It also brings
structure to the mobile beta meaning it will inform the user what features
are in beta at any given time. (see
https://phabricator.wikimedia.org/T182217)
**Added instrumentation to print to pdf button to understand our users
better (https://phabricator.wikimedia.org/T181297)
-
==== Reading Infrastructure ====
* Blocked by:
** Services on code review for
https://github.com/wikimedia/restbase/pull/944
* Blocking:
* Updates:
**Switchover to MCS summary implementation is delayed until after Dev
Summit week.
===== Maps =====
* Updates: None
==== Multimedia ====
* Updates: None
==== Discovery ====
* Blocked by:
* Blocking:
* Updates:
=== Community Tech ===
* Blocked by: Security for GlobalPreferences review
* Blocking:
* Updates: analysing top 10 proposals
=== Contributors ===
==== Editing ====
* Blocked by:
* Blocking:
* Updates:
==== Parsing ====
* Blocked by: None
* Blocking: None
* Updates:
** https://gerrit.wikimedia.org/r/#/c/402455/ has some changes to how
Parsoid handles the interaction between templates and responsive wrappers.
Heads up to Parsoid clients to take a look at that patch and +1 if it
doesn't affect you
** We will roll out changes this week to replace <span> with <sup> for ref
linkbacks ( https://phabricator.wikimedia.org/T45094)
==== Global Collaboration ====
* Blocked by: Security on 5, but waiting for all-hands to talk
* Blocking: Ops on Flow dumps probably; haven't checked in with Matt/Ariel
about this recently
* Updates:
** Flow respects robot policies (noindex) now
==== UI Standardization ====
** OOUI v0.25.1 released
https://phabricator.wikimedia.org/diffusion/GOJU/browse/master/History.md
*** Renaming to unified “OOUI” in code documentation and comments
*** Remove buggy `translateZ` hack on scrollable PanelLayouts – keep an eye
on scrolling perf in Blink/Webkit if you use PanelLayout
* Ongoing:
** OOUI & based products:
*** icons: Unify, refine and align to WikimediaUI Style Guide
https://phabricator.wikimedia.org/T177432 – first patches in:
https://gerrit.wikimedia.org/r/#/c/402757/
== Technology ==
=== Analytics ===
* Blocked by:
* Blocking:
* Updates:
** Found some Event data missing for January 3 through January 7. If your
reports/dashboards look weird, rerun them:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater#Re-runs
** Clickstream blogpost:
https://blog.wikimedia.org/2018/01/16/wikipedia-rabbit-hole-clickstream/
** working on new APIs to report pageviews per project per country (sorting
out ISO codes and data shape)
** testing our java 1.8 upgrade in labs
** Working with Brandon on TLS
** Meltdown reboots continue
=== Cloud Services ===
* Blocked by:
* Blocking:
* Updates:
=== Fundraising Tech ===
* Blocked by:
* Blocking:
* Updates:
** Various CiviCRM improvements
** Making our Amazon Pay SDK fork support TCP proxy
** Re-starting work on new API for our main credit card processor
** Stats projects:
*** Andrew Green's druid banner impressions lib:
https://github.com/AndrewGreen/centralnotice_analytics
*** talked with Analytics, planning to use EventLogging for banner and
donatewiki stats
=== MediaWiki Platform ===
* Blocked by:
* Blocking:
* Updates:
* Triaged/updated RFCs
* Multi-Content Revisions:
** Patches under review:
*** https://gerrit.wikimedia.org/r/#/c/380669/
*** https://gerrit.wikimedia.org/r/#/c/402932/
*** https://gerrit.wikimedia.org/r/#/c/393929/
* cleanupUsersWithNoId: Wikibase blocker has movement.
* Comment table: Schema change looks even closer to done.
* A decision has been reached on ExternalStore de-PHP-serialization; patch
still needs review
* Third-party developer support Phab tasks coordinated with Tech
Collaboration:
** T184606: Evaluate and set up a test instance of FOSS persistent chat
software as a companion to Q&A system for communication with third-party
developers
** T184648: Create and publish a multi-tiered support level system for
MediaWiki extensions frequently used by third parties
* Dev Summit planning
* Audiences Technology Working Group participation
=== Performance ===
* Blocked by: N/A
* Blocking: N/A
* Updates:
**Team offsite this week, will not be attending
**Prep for monitoring/evaluating Singapore at go-live
**Fixes for database "Domain ID" logic -
https://gerrit.wikimedia.org/r/404060 and
https://gerrit.wikimedia.org/r/#/c/404056/
**Docs on running performance testing on an Android phone -
https://wikitech.wikimedia.org/wiki/Measure_Performance#Testing_performance…
**Added alerts for NavigationTiming report rates -
https://phabricator.wikimedia.org/T179555#3887559
=== Release Engineering ===
* Blocking
** None?
* Blocked
** "Stack overflow when Redis is down" -
https://phabricator.wikimedia.org/T185055
*** Need help from Operations and/or Performance
* Updates
** Catching up the train this week and rolling out the last version before
DevSummit/All Hands and RelEng team offsite weeks. [wiki[email]]
**** **https://phabricator.wikimedia.org/T180749#3897321*
<https://phabricator.wikimedia.org/T180749#3897321>
** We moved Wednesday morning’s SWAT window 1 hour earlier (to 10am) to
give us an hour break before the new MW version rolls to second set of
wikis (all non-wikipedias) which was a follow-up from a recent post-mortem.
[wiki][email]
**** *
*https://lists.wikimedia.org/pipermail/wikitech-l/2018-January/089404.html*
<https://lists.wikimedia.org/pipermail/wikitech-l/2018-January/089404.html>
**** **https://phabricator.wikimedia.org/T182733*
<https://phabricator.wikimedia.org/T182733>
** We broke git-fat deploy repos in scap (old config no longer valid),
workaround/fix available in all relevant repos.
**** **https://phabricator.wikimedia.org/T184882#3899710*
<https://phabricator.wikimedia.org/T184882#3899710>
*** (Yes, we’re re-doing how the CI for scap is done, see:
*https://phabricator.wikimedia.org/T184628*
<https://phabricator.wikimedia.org/T184628> )
** Updated the Debian packaging for Zuul (CI task scheduler) and released
2.5.0-8-gcbc7f62-wmf6, unblocking an upgrade of Gerrit.
**** **https://phabricator.wikimedia.org/T158243*
<https://phabricator.wikimedia.org/T158243>
** Converted our home-grown docker image builder to `docker-pkg` from
Giuseppe
**** **https://phabricator.wikimedia.org/T177276*
<https://phabricator.wikimedia.org/T177276>
** Getting started with the basics of planning our team offsite pre
Barcelona Hackathon. Submitted travel request form and let eng-admin@ know.
** Working on browser tests with Search (“selenium-CirrusSearch-jessie
daily Jenkins job”).
=== Research ===
* Blocked by:
* Blocking:
* Updates:
=== Scoring Platform ===
* Blocked by:
* Blocking:
* Updates:
=== Search Platform ===
* Blocked by:
* Blocking:
* Updates:
* Improving LTR training mechanisms
https://phabricator.wikimedia.org/T184547
* Working on fixes for completion suggester & redirects namespaces
https://phabricator.wikimedia.org/T115756
* Investigating ElasticSearch phonetic search
https://phabricator.wikimedia.org/T182708
* Working on Serbian analysis plugins for ES
https://phabricator.wikimedia.org/T183015
* Working on refactoring search profiles to make them more config-like
https://phabricator.wikimedia.org/T183279
* Finished test for machine-learning ranking on Hebrew wiki
https://phabricator.wikimedia.org/T182616, result analysis next
* Working on enabling WDQS-driven deep category search
https://phabricator.wikimedia.org/T181549
=== Security ===
* Blocked by:
* Blocking:
* Updates:
=== Services ===
* Blocked by: none
* Blocking: none
* Updates:
** All REST traffic is served from Cassandra 3 cluster
** Remaining Cassandra 2 nodes are being moved to Cassandra 3 cluster
** 50% of htmlCacheUpdate jobs are processed on kafka
*** All except enwiki, commons and wikidata
=== Technical Operations ===
* Blocked by:
** Global collaboration on the flow dumps.
https://phabricator.wikimedia.org/T164262
* Blocking:
** None
* Updates:
** Team restructuring happening. Don't expect much fallout, but we are SREs
these days.
** Grafana will be migrated to support native LDAP on Feb 12 2018. This
means the eventual demise for grafana-admin.wikimedia.org. Announcement to
be posted to engineering@ and wikitech@.
https://phabricator.wikimedia.org/T170150
== Wikidata ==
* Blocked by:
* Blocking:
* Updates:
== German Technical Wishlist ==
* Blocked by:
* Blocking:
* Updates:
== SoS Meeting Bookkeeping ==
* Updates: