Hi there,
I'm a final year Mathematics student at the University of Bristol, and I'm
studying Wikipedia as a graph for my project.
I'd like to get data regarding the number of outgoing links on each page,
and the number of pages with links to each page. I have already
inquired about this with the Analytics Team mailing list, who gave me a few
suggestions.
One of these was to run the code at this link https://quarry.wmflabs.org/
query/25400
with these instructions:
"You will have to fork it and remove the "LIMIT 10" to get it to run on
all the English Wikipedia articles. It may take too long or produce
too much data, in which case please ask on this list for someone who
can run it for you."
I ran the code as instructed, but the query was killed as it took longer
than 30 minutes to run. I asked if anyone on the mailing list could run it
for me, but no one replied saying they could. The guy who wrote the code
suggested I try this mailing list to see if anyone can help.
I'm a beginner in programming and coding etc., so any and all help you can
give me would be greatly appreciated.
Many thanks,
Nick Bell
University of Bristol
What ways are there to include user-edited JavaScript in a wiki page?
I ask because someone put this revision in (which is now deleted):
https://fa.wikipedia.org/w/index.php?title=%D9%85%D8%AF%DB%8C%D8%A7%D9%88%D…
You can't see it now, but it was someone including a JavaScript
cryptocurrency miner in common.js!
Obviously this is not going to be a common thing, and common.js is
closely watched. (The above edit was reverted in 7 minutes, and the
user banned.)
But what are the ways to get user-edited JavaScript running on a
MediaWiki, outside one's own personal usage? And what permissions are
needed? I ask with threats like this in mind.
- d.
Hi,
Sometimes I find adding assert() calls in my code very handy for various
reasons:
- failures in development mode on some complex code where exposing all the
details to unit tests is sometimes hard and/or pointless
- readability of the code
But I worry about the perf implications of these lines of code. I don't
want these assertions to be used to track errors in production mode.
PHP7 introduced expectations which permit to have zero-cost assert() [1]
Looking at the MW codebase we don't seem to use assert frequently (only 26
files [2] ).
Are there some discussions about this?
Is assert() a good practice for the MW code base?
If yes would it make sense to benefit from zero-cost assertions in WMF
appservers?
Thanks!
[1]
http://php.net/manual/en/function.assert.php#function.assert.expectations
[2]
https://codesearch.wmflabs.org/search/?q=%5Ctassert%5C(&i=nope&files=php%24…
[x-posted announcement]
Hello,
Wikimedia Foundation’s Language team would like to invite you for an online
office hour session scheduled for Wednesday, March 21st, 2018 at 13:00 UTC.
This will be an open session to talk about our work, and in particular the
changes to interlanguage links, which were recently rolled-out on the
English Wikipedia.
The new option shows a list of up to 9 languages instead of a long list
that can have more than 200 items, and a panel with all the links that can
be looked up in any language using a search box. The purpose of this
feature is to make articles in all languages easier to find. We recently
published a blog post about this feature and the thoughts behind the
development:
https://blog.wikimedia.org/2018/03/08/compact-language-links-launch.
This session is going to be an online discussion over Google
Hangouts/Youtube with a simultaneous IRC conversation. Due to the
limitation of Google Hangouts, only a limited number of participation slots
are available. Hence, do please let us know in advance if you would like to
join in the Hangout. The IRC channel will be open for interactions during
the session.
Please read below for the event details, including local time, youtube
session links and do let us know if you have any questions.
Thank you
Runa
== Details ==
# Event: Wikimedia Foundation Language office hour session
# When: March 21st, 2018 (Wednesday) at 13:00 UTC (check local time
http://www.timeanddate.com/worldclock/fixedtime.html?iso=20180321T1300)
# Where: and on IRC #wikimedia-office (Freenode) and
https://www.youtube.com/watch?v=RmZcL6zVcTA
# Agenda:
Discussion about Compact Language Links, and Q & A.
--
Engineering Manager, Language (Contributors)
Wikimedia Foundation
Hi all,
TL;DR: When using X-Wikimedia-Debug to profile web requests on Wikimedia
wikis, the generated profile information will now include details from
"w/index.php", and MWMultiVersion, and things like
wmf-config/CommonSettings.php. Details at
https://phabricator.wikimedia.org/T180183.
-
The debug profiler provided on Wikimedia production wikis[1] previously
could not cover the code that executes before MediaWiki core instantiates
ProfilerXhprof, which was in charge of calling `xhprof_enable`. This
normally happens within core's Setup.php.
While that point in Setup.php is before any important MediaWiki core logic,
it misses out on two other chunks of code:
1. Initialisation of MediaWiki core – This includes entry point code (eg.
index.php, PHPVersionCheck), but also the first steps of Setup before
Profiler. Such as AutoLoader, vendor, and LocalSettings.php. At WMF,
LocalSettings.php loads wmf-config/InitialiseSettings.php and
wmf-config/CommonSettings.php.
2. Wrapping of MediaWiki entrypoint – At Wikimedia, the index.php
entrypoint is itself further wrapped in something called "multiversion".
Multiversion is what determines the wiki ID (eg. "enwiki") and MediaWiki
branch (eg. "1.31.0-wmf.25") associated with the current domain (eg. "
en.wikipedia.org").
Over the past weeks, I've been refactoring MediaWiki core, wmf-config and
Wikimedia's HHVM settings to make we can instrument the above code as part
of our performance profiles.
This change happened in three phases:
## 1. Update wmf-config/StartProfiler to.. actually start the profiler!
The file name is somewhat deceptive because traditionally this is (and can)
only be used to *configure* the profiler, by assigning $wgProfiler. It
makes sense that we cannot instantiate the Profiler subclass from this
file, because the classes and run-time configuration are not and cannot be
available this early.
However, we don't the Profiler class to record data. The Profiler classes
typically obtain their data from native PHP. The one used at WMF is XHProf.
Previously, we would assign $wgProfiler['class'] = 'ProfilerXhprof', and
then later MediaWiki core instantiates ProfilerXhprof, which then calls
xhprof_enable. We now xhprof_enable directly from StartProfiler.php.
This change enabled coverage of code in Setup.php between 'include
StartProfiler' and 'Profiler::instance()'. – Mainly: vendor, LocalSettings,
wmf-config.
## 2. Update MediaWiki core to include StartProfiler earlier.
It is now the first thing included by Setup.php.
This change enabled coverage of code in Setup.php that previously was
before 'include StartProfiler'. – Namely: AutoLoader.php, Defines.php.
## 3. Configure WMF's PHP engine to use auto_prepend_file
This is the big one, and requires a PHP ini setting change. Third parties
can follow the same pattern
in order to get the same benefits:
* Put `xhprof_enable( $flags )`, along with any sampling/conditional
logic, in a separate file.
* Use it from two places:
** In StartProfiler.php, include using require_once.
** In php.ini, set auto_prepend_file=path/to/profiler.php.
This change enabled coverage of all remaining code. – Namely: multiversion,
w/index.php and things like PHPVersionCheck.
## Example
Using cURL:
$ curl -H 'X-Wikimedia-Debug: 1' 'https://en.wikipedia.beta.
wmflabs.org/w/load.php?debug=false&modules=startup&only=
scripts&forceprofile=1'
Output now includes:
- main() # resembles the wrapper at
operations/mediawiki-config.git:/w/load.php
-- run_init::/srv/mediawiki/multiversion/MWMultiVersion.php
-- MWMultiVersion::getMediaWiki
-- run_init:/srv/mediawiki/php-../load.php
-- run_init::/srv/mediawiki/php-../includes/Setup.php
-- run_init::/srv/mediawiki/php-../LocalSettings.php
-- run_init::/srv/mediawiki/wmf-config/CommonSettings.php
--
run_init::/srv/mediawiki/php-../extensions/Wikibase/client/WikibaseClient.php
Greetings,
-- Timo Tijhof
[0] https://phabricator.wikimedia.org/T180183
[1] https://wikitech.wikimedia.org/wiki/X-Wikimedia-Debug
*On 14 March 2018, evidence of cryptocurrency mining software was
discovered on Persian Wikipedia. It was identified by the community and
removed within 10 minutes of being added to the site. Additionally, the
rights of the user responsible have been revoked and their account has been
globally locked. At this time there is no evidence of any user's computer
or account being compromised or otherwise affected. However, we encourage
everyone to take some routine steps to maintain a secure computer and
account - including regularly changing your passwords, actively running
antivirus software on your systems, and keeping your system software up to
date. The Wikimedia Foundation's Security team is investigating this
incident as well as potential improvements to prevent future incidents. If
you have any questions, please contact the Security team
(security-team{{(a)}}wikimedia.org <http://wikimedia.org/>). Apologies for
only posting in English, translating and reposting in Fārsi would be
greatly appreciated.Thanks,John Bennett*
https://www.mediawiki.org/wiki/Scrum_of_scrums/2018-03-14
= 2018-03-14 =
= date =
== Callouts ==
* Fundraising campaigns
https://meta.wikimedia.org/wiki/CentralNotice/Calendar - next up:
Netherlands 2018-04-03 through 2018-05-01
* [Performance] Editing: Someone familiar with VisualEditor should probably
take a look at https://phabricator.wikimedia.org/T189229-- perf issue, but
not something that we can easily address
* [Language] SRE: DBA to look/review:
https://phabricator.wikimedia.org/T187986
* [SRE] Started talking about next Quarter's goals, please DO LET us know
if you have dependencies on us, don't just mark them on the wiki.
* Tidy has been replaced on another 100 wikis this week. About 300 more to
go before the end June deadline.
* https://gerrit.wikimedia.org/r/#/c/415770/ is almost ready to go that
implements RFC to trim whitespace in headings, lists, and table cells. Very
likely to go out next week unless there are objections raised in the last
call period (ending today, I think).
* Parsing team has been working on the heredocs syntax RFC (
https://phabricator.wikimedia.org/T114432 ) and has a WIP patch @
https://gerrit.wikimedia.org/r/#/c/418198/ if someone wants to test and
play with it on your local dev wikis.
* Wikistats 2 has now a canary site - https://wikistats-canary.wmflabs.org/
* 1.31.0-wmf.25 going out this week, if you see blockers:
https://phabricator.wikimedia.org/T183964
* Scoring platform team is restarting work on the promotional editing
model, want to collaborate?
== Audiences ==
=== Readers ===
==== iOS native app ====
* Blocked by:
* Blocking:
* Updates:
**
==== Android native app ====
* Blocked by:
* Blocking:
* Updates:
** Continuing rollout of Reading List syncing to Beta audience. Wrapping up
next maintenance/bugfix release.
==== Readers Web ====
* Blocked by:
Language engineering/core platform(?) -
https://gerrit.wikimedia.org/r/#/c/415187/ We'd like to make RTL info of
languages available cheaply either via JavaScript/PHP array or API.
Guidance needed
* Blocking:
* Updates:
Refreshing some old bugs relating to the Nearby overlay
*Quarterly goal dependency update:
** Increase learning by lowering the cost of exploration
*** On final stretch. Aiming to launch previews within next 3 weeks. Now we
have the new summary endpoint from RI we are addressing some legacy code on
client.
*** Reading Web/Performance
** Continue improving the ways that users can download articles of interest
for later consumption
*** Reading Web: Tech Ops/RelEng
==== Readers Infrastructure ====
* Blocked by:
* Blocking:
* Updates:
** Filter for /page/summary endpoint is enabled, which means all reads
should get the latest version. If you find any issues with the summary
content please file a task.
** Wikidata overrides will be enabled on beta enwiki today (Wed), testwiki
early next week
===== Maps =====
* Blocked by:
* Blocking:
* Updates:
**
==== Multimedia ====
* Updates
** 3D work closing soon
** Work on file page prototype for structured data is progressing (slowly),
may be bothering WMDE soon about some questions
** Search work seems like it's going smoothly, thanks Search for being
available and helpful
** UploadWizard prototype changes for multi-lingual captions
Quarterly goal dependency update
** Prepare backend infrastructure for structured data search
*** SDC: Search PF/Multimedia/Wikidata
** Wikimedia Technology/Goals/2017-18 Q3#Segment 2: Search integration and
exposure
*** SDC: Research/Multimedia
** Prepare backend infrastructure for structured data search
*** SDC: Search PF/Multimedia/Wikidata
=== Contributors ===
==== Community Tech ====
* Blocked by: SRE on GlobalPreferences schema T184666 - quarterly goal
* Blocking:
* Updates:
** Working on thanking for log entries and squashing some CodeMirror bugs
before promoting it from beta feature.
==== Anti-Harassment Tools ====
* Blocked by: None
* Blocking: None
* Updates:
** Compmleted Interaction Timeline v1
** Started work on blocking tools
** Research for reporting system
==== Editing ====
* Blocked by:
* Blocking:
** Updates:
==== Parsing ====
* Blocked by:
* Blocking:
* Updates:
** We've fallen behind responding to all the Linter feature requests --
since we don't have sufficient time to handle all of it, but we'll try to
address some in the coming couple weeks.
** Tidy has been replaced on another 100 wikis this week. About 300 more to
go before the end June deadline.
** https://gerrit.wikimedia.org/r/#/c/415770/ is almost ready to go that
implements RFC to trim whitespace in headings, lists, and table cells. Very
likely to go out next week unless there are objections raised in the last
call period (ending today, I think).
** Arlo has been working on the heredocs syntax RFC (
https://phabricator.wikimedia.org/T114432 ) and has a WIP patch @
https://gerrit.wikimedia.org/r/#/c/418198/ if someone wants to test and
play with it on your local dev wikis.
** Scott has been adding language variant conversion support to Parsoid --
a WIP patch in place for Serbian.
** Arlo would appreciate Android app folks taking a look at
https://gerrit.wikimedia.org/r/#/c/411493/ -- this is not directly related
to Parsing work.
*Quarterly goal dependency update:
** Support work towards unifying MediaWiki's parser implementations, in
liaison with Technology's MediaWiki team
*** Parsing:Mediawiki PF/Services
*** No new updates or blockers.
==== Collaboration ====
* Blocked by: nobody
* Blocking: nobody
* Updates:
** Working on a test setup for the maps services (Kartotherian and
Tilerator) in beta labs, to replace the maps-test servers
==== Language ====
* Blocked by: See callouts.
* Blocking:
* Updates:
** CX2 (ie ContentTranslation version 2 with VE) work continue.
=== Audiences Design ===
* Blocked by:
* Blocking:
* Updates:
**
* Quarterly goal dependency update:
** Audiences DesignStandardise our user interfaces to match user
expectation of quality from our products
*** Audiences Design: Ops
==== UI Standardization ====
* Blocked by: Security review for Style Guide
https://phabricator.wikimedia.org/T188698
* Blocking:
* Updates:
** OOUI – no release this week, in prep for v0.26.0 breaking change release
with (currently planned)
*** WikimediaUI theme's new icon set
https://gerrit.wikimedia.org/r/#/c/400091/
*** Aligning VE and Special:Pages interface base font-size to each other
https://phabricator.wikimedia.org/T97631
*** Switch to grunt-svgmin for SVG opt accomplished
** Style Guide
*** Continued work to accomplish v1 goals
https://phabricator.wikimedia.org/tag/wikimediaui_style_guide/
== Technology ==
=== Analytics ===
* Blocked by:
* Blocking:
* Updates:
** Eventlogging Analytics migrates to Kafka Jumbo -
https://phabricator.wikimedia.org/T183297
** Eventlogging is now on eventlog1002 and running Debian/systemd
** Deployed sanitization efforts on EventLogging Hive
https://phabricator.wikimedia.org/T181064
** Wikistats 2 has now a canary site - https://wikistats-canary.wmflabs.org/
** Rolling out responsive site soon
* Quarterly goal dependency update:
**Improve, adjust, or create features geared at the needs identified in New
Editors research project.
*** New Editors Experience:Analytics
=== Cloud Services ===
* Blocked by:
* Blocking:
* Updates:
** Wikitech moved to new labweb hardware
https://phabricator.wikimedia.org/T168470
** Updates to Horizon and Toolsadmin coming up on Wednesday, Mar 15
*** Horizon UI will look different - test it out at
https://newhorizon.wikimedia.org/
** Dumps dataset server migration being planned for first week of April
*** See https://phabricator.wikimedia.org/T168486#4033572 for timeline
=== Fundraising Tech ===
* Blocked by:
* Blocking:
* Updates:
** CentralNotice: deployed and announced security patches, now deploying
EventLogging for impressions. Reviewing a couple of contributed patches.
** CiviCRM: working on anti-fraud tools and custom imports
** More work for main card processor's new API
** Working to also use EventLogging for fundraiser landing page hits
=== MediaWiki Platform ===
* Blocked by:
* Blocking:
* Updates:
** Edit summary display-side truncation
https://gerrit.wikimedia.org/r/#/c/417593/
** Sifting through resumes
** Coordinating with Federal MediaWiki user group and User:Econterms on VPAT
*** User:Econterms has started
https://www.mediawiki.org/wiki/VPAT_for_MediaWiki
** Published aggregate pingback data at https://pingback.wmflabs.org
** Started working on patches to extensions to use Actor table
** RemexHtml parser tests merged https://gerrit.wikimedia.org/r/#/c/409432/
** PHP 7.0 MediaWiki unit tests now voting
https://gerrit.wikimedia.org/r/#/c/417343/
*Quarterly goal dependency update:
** Support work towards unifying MediaWiki's parser implementations, in
liaison with Technology's MediaWiki team
*** Parsing:Mediawiki PF/Services
**** No new updates or blockers
*** Parsing/Mediawiki PF
**** No new updates or blockers
*** SDC: Mediawiki PF/Wikidata
**** No new updates or blockers
=== Performance ===
* Blocked by:
** n/a
* Blocking:
** n/a
* Updates:
** Issues with the kafka library that we used for upgrading coal (one of
our utilities), need to re-do
** We will start working on safemode for Visual Editor this week
** Attempting to remove a few query-master-on-GET issues
** Updated our MediaWiki profiling to start earlier, will catch more
performance issues on startup
** Beginning to work on packaging Dynomite
** Oversampling of performance data in Singapore is live
=== Release Engineering ===
* Blocking
** Scoring platform release of scap 3.8 (I think mukunda tagged that, but
I'll double check)
* Blocked
** SRE: Minikube packaging stuff https://phabricator.wikimedia.org/T184457
* Updates
** 1.31.0-wmf.25 going out this week, if you see blockers:
https://phabricator.wikimedia.org/T183964
** greg is out this week, FYI
*Quarterly goal dependency update:
** Continue improving the ways that users can download articles of interest
for later consumption
*** Reading Web: Tech Ops/RelEng
** Update:
*
=== Research ===
* Blocked by:
* Blocking:
* Updates:
*Quarterly goal dependency update:
** Wikimedia Technology/Goals/2017-18 Q3#Segment 2: Search integration and
exposure
*** SDC: Research/Multimedia
** Update:
*
=== Scoring Platform ===
* Blocked by: RelEng on release of scap 3.8 and git-lfs
* Blocking:
* Updates:
**Extension:JADE is live on the beta cluster :D
** wp10 model support in ORES ext. is almost finished. Expected to have it
enabled in enwiki next week
**Restarting work on the promotional editing model, want to collaborate?
***Just about to publish a dataset of paid editors
=== Search Platform ===
* Blocked by:
* Blocking:
* Updates:
** Testing “query explorer” feature of LTR plugin:
https://phabricator.wikimedia.org/T187148
** TermLookup implementation for ElasticSearch ready for review:
https://phabricator.wikimedia.org/T143706
** Working on query parsing refactoring:
https://phabricator.wikimedia.org/T185108
** Serbian analyzer merged, will be enabled after Elastic plugin is
deployed: https://phabricator.wikimedia.org/T183015
** Next is Slovak analysis: https://phabricator.wikimedia.org/T178929
** Discussing Lexeme search:
https://www.wikidata.org/wiki/User:Smalyshev_(WMF)/Lexeme_search
** Stability issues with wdqs1004 solved (thanks RobH and all who helped!),
WDQS cluster back at full capacity
** Kafka poller has been having problems, had to turn off for now. Will be
back when jumbo mirroring is stable -
https://phabricator.wikimedia.org/T189458
*Quarterly goal dependency update:
** Prepare backend infrastructure for structured data search
*** SDC: Search PF/Multimedia/Wikidata
**** Working with Cormac on MediaInfo/File indexing patch
****Started design discussions on Lexeme search -
https://www.wikidata.org/wiki/User:Smalyshev_(WMF)/Lexeme_search
** Update:
*
=== Security ===
* Blocked by:
* Blocking:
* Updates:
**
=== Services ===
* Blocked by: none
* Blocking: none
* Updates:
** Summary migration completely complete
** Working on issues with MySQL connection spiking from refreshLinks jobs an
d kafka rebalancing issues
=== Site Reliability Engineering ===
* Blocked by:
** None
* Blocking:
** None
* Updates:
* Started talking about next Quarter's goals, please DO LET us know if you
have dependencies on us, don't just mark them on the wiki.
**
*Quarterly goal dependency update:
** Continue improving the ways that users can download articles of interest
for later consumption
*** Reading Web: Tech Ops/RelEng
** Update: No updates
*
** Audiences DesignStandardise our user interfaces to match user
expectation of quality from our products
*** Audiences Design: Ops
** Update: No updates on our side, ball is not in our field.
== Wikidata ==
* Blocked by: nothing
* Blocking: not aware of any blocks
* Updates: re-org happened, we are now 2 teams, so didn't achieve much as
far as production in the last week, Team 1 will focus on delivering Lexeme
and team 2 are working on Scalaibity issues and data quality at this point
**
*Quarterly goal dependency update:
**1.1 It is possible to store structured data within wiki pages, in
particular on media file pages on Commons. We will enable the MediaWiki
storage layer to correctly store and process structured data elements
within wiki pages.
*** SDC: Mediawiki PF/Wikidata
** Update:
*
** Prepare backend infrastructure for structured data search
*** SDC: Search PF/Multimedia/Wikidata
** Update:
*
== German Technical Wishlist ==
* Blocked by:
** Review on https://gerrit.wikimedia.org/r/#/c/404293/
* Blocking:
** no
* Updates:
** continue working on FileImporter
** Finished work on Wikidiff2 (Reviewww!!)
** Continue working on AdvancedSearch
== SoS Meeting Bookkeeping ==
* Updates:
**