Wikitech-l August 2020

wikitech-l@lists.wikimedia.org

67 participants
58 discussions

Upgrading mailman (the software behind mailing lists)
by Amir Sarabadani 24 Aug '20

24 Aug '20

Hey, Mailman, the software that powers our mailing lists, is extremely old, by looking at https://lists.wikimedia.org/ you can guess how old it is. I would really like to upgrade it to mailman 3 which has these benefits: * Much better security (including but not limited to https://phabricator.wikimedia.org/T181803) * Much better UI and UX * Much easier moderation and maintaining mailing lists * Ability to send mail from the web * Ability to search in archives. * Ability to like/dislike an email * List admins will be able to delete emails, merge threads, and much more. * Admins won't need to store passwords for each mailing list separately, they just login as their account everywhere. * The current mailman stores everything as files (even mailing list settings), mailman3 actually uses a proper database for everything meaning proper backup and recovery, high availability and much more. I have already put up a test setup and humbly ask you (specially list admins) to test it (and its admin interface), if you want to become a list admin, drop me a message. Keep in mind that we don't maintain the software so the most I can do is to change configuration and can't handle a feature request or solve a bug (you are more than welcome to file it against upstream though) Here's the test setup: * https://lists.wmcloud.org Here's a mailing list: * https://lists.wmcloud.org/postorius/lists/test.lists.wmcloud.org/ Here's an archive post: * https://lists.wmcloud.org/hyperkitty/list/test@lists.wmcloud.org/thread/RMQ… Issues that I haven't figured out yet: * This system has profile picture support but it's only gravatar which we can't enable due to our privacy policy but when you disable it, it shows empty squares and looks bad. Reported upstream [1] but also we can have a gravatar proxy in production. And in the worst case scenario we can just inject "$('.gravatar').remove();" and remove them. Feel free to chime in in the phabricator ticket in this regard: https://phabricator.wikimedia.org/T256541 * Upgrade will break archive links, making it work forever is not trivial (you need write apache rewrite rule) (You can read about it in https://docs.mailman3.org/en/latest/migration.html#other-considerations) * Mailman allows us to upgrade mailing list by mailing list, that's good but we haven't found a way to keep the old version and the new ones in sync (archives, etc.). Maybe we migrate a mailing list and the archives for the old version will stop getting updated. Would that work for you? Feel free to chime in: https://phabricator.wikimedia.org/T256539 * We don't know what would be the size of the database after upgrade because these two versions are so inherently different, one idea was to check the size of a fully public mailing list, then move the files to the test setup, upgrade it to the new version and check how it changes, then extrapolate the size of the final database. The discussion around the database is happening in https://phabricator.wikimedia.org/T256538 If you want to help in the upgrade (like puppetzining its configuration, etc.) just let me know and I add you to the project! It uses a stand-alone puppetmaster so you don't need to get your puppet patches merged to see its effects. The main ticket about the upgrade: https://phabricator.wikimedia.org/T52864 [1] https://gitlab.com/mailman/hyperkitty/-/issues/303#note_365162201 Hope that'll be useful for you :) -- Amir (he/him)

8 12

No train next week; no deployments next Tuesday
by Tyler Cipriani 24 Aug '20

24 Aug '20

Hi all There is a planned switchover to our secondary datacenter scheduled for Tuesday, September 2nd 2020. To avoid creating problems for our SREs we'll be skipping the train for next week -- the week of 2020-08-31 -- and not doing any deployments the day of the switchover -- 2020-09-01. The deployment calendar is up-to-date[1]. Thanks! -- Tyler [0]: <https://meta.wikimedia.org/wiki/Tech/Server_switch_2020> [1]: <https://wikitech.wikimedia.org/wiki/Deployments#Week_of_August_31>

1 0

New member
by Rahul Pawar 24 Aug '20

24 Aug '20

Hello Guys!! I'm Rahul Pawar a B.tech second-year student at Visvesvarya National Institute of Technology, Nagpur, India. I am a newbie but have tried frontend mentor challenge, 2-star coder in Codechef, Gold medal in C, Cpp, Python in hackerRank. I'm interested in working with this organization and even participate in GSOC 2021. SO any mentor out there to guide me. Thanks in advance. Regards Rahul Pawar

2 1

Scrum of scrums/2020-08-19
by Deb Tankersley 24 Aug '20

24 Aug '20

Hello, This email contains updates for last week - August 19, 2020, with apologies for the lateness. For the HTML versions, see: https://www.mediawiki.org/wiki/Scrum_of_scrums/2020-08-19 Cheers, Deb ------------------------ *= 2020-08-19 =* == Product == === Web === * Updates: ** '''Summary''': Vue.js-focused week. ** [[Reading/Web/Desktop Improvements|Desktop Improvements Project (Vector / DIP)]]: *** [[phab:T258493|[Spike 8hrs] "Use Legacy Vector" is not working as a global preference]] *** [[phab:T258465|[Bug] Sidebar Expand/Collapse button impacted by Narrow Width]] *** [[phab:T255727|Make collapsible sidebar persistent for logged-in users]] *** [[phab:T249363|Move the existing search to the header in preparation for Vue.js search development]] *** [[phab:T251817|[Dev] Allow extensions to update the footer without resorting to SkinTemplateOutputPageBeforeExec hook]] *** [[phab:T244392|Vue.js search case study]]: **** See [[Reading/Web/Desktop Improvements/Vue.js case study/Status log|weekly status updates]]. ** Mobile website (MinervaNeue / MobileFrontend): *** [[phab:T260155|PHP Fatal error: Uncaught Error: Call to undefined method TitleValue::isSubpage() in /srv/mediawiki/php-1.36.0-wmf.4/skins/MinervaNeue/includes/Skins/SkinUserPageHelper.php:55 in /srv/mediawiki/php-1.36.0-wmf.4/skins/MinervaNeue/includes/Skins/SkinUserPageHelper.php on line 55]] *** [[phab:T231160|HtmlFormatter incorrectly removes partial classname matches in "xenomobile" or "not-an-navbox"]] *** [[phab:T214647|[EPIC] Re-define the contract for displaying drawers and overlays in MobileFrontend]] *** [[phab:T212465|[EPIC] None of our View's should exhibit 2 levels of inheritance]] *** [[phab:T258096|Regression: Nested references do not open if user clicks on [ or ] (which are wrapped in span)]] *** [[phab:T240622|[Technical debt payoff] Remove InlineDiffFormatter and InlineDifferenceEngine from MobileFrontend]] ** Standardization *** [[phab:T256520|Consider 'normalize' stylesheet RL module]] *** [[phab:T259906|Add 'volumeUp' icon in OOUI]] ** Miscellaneous *** [[phab:T224985|Rename Special:ElectronPdf to Special:Download as Pdf or Special:Export as Pdf]] == Technology == === Search Platform === * Blocked by: * Blocking: * Updates ''(long list of updates, we were lagging behind in reporting those)'': ** Integrate Wikimedia Event Utilities with discovery-parent-pom - https://phabricator.wikimedia.org/T260375 ** Search does not return exact title match - https://phabricator.wikimedia.org/T257922 ** Querying WCQS should allow me to use prefixes for MediaInfo items - https://phabricator.wikimedia.org/T258625 ** Gather statistics on head and tail query distribution on Commons - https://phabricator.wikimedia.org/T252544 ** Slow indexing of Lexemes for wbsearchentities - https://phabricator.wikimedia.org/T240328 ** Create spreadsheet of last 90 days of Commons search queries - https://phabricator.wikimedia.org/T257361 ** Validate and fix TTL dumps of SDoC - https://phabricator.wikimedia.org/T251496 ** Create a Examples Page for WCQS - https://phabricator.wikimedia.org/T258489 ** v: prefix not correctly prefixed in Wikibase when using entitysource config and extra prefixes -https://phabricator.wikimedia.org/T258507 ** CirrusSearch throws an error on several wikis when searching for "intitle:/regex/" -https://phabricator.wikimedia.org/T250810 ** Access restriction for SPARQL Endpoint for Commons - https://phabricator.wikimedia.org/T251498 ** UI for SPARQL Endpoint for Commons - https://phabricator.wikimedia.org/T251514 -- deb tankersley (she/her) sr program manager, engineering Wikimedia Foundation

1 0

MediaWiki 1.35.0-rc.2 is ready for testing
by Sam Reed 21 Aug '20

21 Aug '20

I'm pleased to announce the immediate availability of MediaWiki 1.35.0-rc.2, the third release candidate for 1.35.x, the next LTS version to replace 1.31 which is due to go end of life in June 2021. Download links at the end of the e-mail. The tag has been signed and pushed to Git. Please note that the PHP version requirement has been raised from 7.2.9 in MediaWiki 1.34 (and 7.0 in MediaWiki 1.31), to 7.2.22 (this may change further, see below). This is not a final release and should not be used for production websites. Known issues are tracked in Phabricator on the release workboard [1]. As always please do try out the release candidate in a test environment and report any issues that you discover. Please use the #MW-1.35-Release [2] tag in Phabricator when reporting issues specific to this release. It is expected that MediaWiki 1.35 will become final in late August 2020, and will be supported for 3 years after that. Known/outstanding issues/things to test: * It has been proposed to require PHP 7.3 for MediaWiki 1.35, please discuss at <https://phabricator.wikimedia.org/T257879>. * Both the Vector skin and the underlying skin infrastructure are undergoing numerous changes, so there might be things broken that are already fixed in master and as such need backporting. * VisualEditor and Parsoid are now bundled in the tarball and no longer need a separate nodejs service. The documentation for this still needs to be updated, and in some cases, users are reporting HTTP 500 errors from RestBase <https://phabricator.wikimedia.org/T259693>. * Watchlist expiry (behind the $wgWatchlistExpiry flag) is currently experimental. It should be finished for the 1.35.0 final release. * If you're on Windows and use 7zip and had issues in the previous release candidates (and the last round of security releases) extracting the tarball, this should be fixed for this release. * While the rc.2 tarballs are smaller than the rc.1 tarballs, the patch for rc.2 is much larger than usual, due to the removal of Gruntfile.js and package-lock.json files from the tarball. Changes since 1.35.0-rc.1: * (T259693) uuid: Fix filenames on Windows. * Remove Gruntfile.js and package-lock.json from the tarball. * firejail: Strengthen by copying from Wikimedia's profile. * (T260059) ResourceLoaderOOUIImageModule: loadOOUIDefinition() may return false. * (T30162, T245387) The installer supports using a Postgres server running on a custom port other than 5432. * (T260201) Support private wikis in Parsoid zero configuration mode. * Fix bad use of `|=` PHP bit operation where `= … ||` bool is intended. * (T259212) SpecialBlock: Show error if a block could not be inserted or found. * (T255842) UserOptionsManager: fix options reset. * (T258649) WatchAction: avoid unnecessary UPDATEs when expiry is unchanged. * (T250851) Allow skins to override mediawiki.page.ready initialisation. * (T250851) mediawiki.page.ready: Allow skins to disable search lazy load. * (T253135, T255632) Update language in watchlist expiry. * Use IPset in MWRestrictions::checkIP. * (T259564) Fix race condition on edit page. * (T260759) Hide watchlist expiry label in edit form. * mime: Fix docs of MIME_EXTENSIONS, they're arrays, not space-seperated. * (T260031) Add application/font-sfnt to MimeMap for ttf files. * (T259379) WatchedItemStore: Cache single WatchedItems with preexisting expiry. * Add a maintenance script to create bot passwords. * (T201269) Add Traditional Chinese zh-hant as fallback for Amis (ami). * Improve wfParseUrl docs. * (T251038) Add multi index fields in ImageListPager for unique paginate. * (T259916) Guard against 'Widget not found' error. Preliminary release notes: https://phabricator.wikimedia.org/source/mediawiki/browse/REL1_35/RELEASE-N… https://www.mediawiki.org/wiki/Release_notes/1.35 Open Bugs: [1] https://phabricator.wikimedia.org/project/board/4035/ Bug report form: [2] https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?tags=MW-1.35-… ********************************************************************** Download: https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.0-rc.2.tar.gz Download without bundled extensions: https://releases.wikimedia.org/mediawiki/1.35/mediawiki-core-1.35.0-rc.2.ta… Patch to previous version (1.35.0-rc.0): https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.0-rc.2.patch.gz GPG signatures: https://releases.wikimedia.org/mediawiki/1.35/mediawiki-core-1.35.0-rc.2.ta… https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.0-rc.2.tar.gz.… Public keys: https://www.mediawiki.org/keys/keys.html

1 0

realtime notifications disabled in Phabricator
by Daniel Zahn 17 Aug '20

17 Aug '20

Phabricator users, this is to let you know that the "aphlict" service has been disabled on Phabricator (for now) because it caused stability issues. This means you will not get realtime (pop-up) notifications on Phabricator. (If you had those enabled in the first place). Regular notifications (that do not pop-up) and emails are not affected by this. https://phabricator.wikimedia.org/T238593 -- Daniel Zahn <dzahn(a)wikimedia.org> Operations Engineer

8 15

[FYI] Identifying candidates for Gerrit CR+2 permissions
by Andre Klapper 17 Aug '20

17 Aug '20

I've been wondering for a while [1] about a way to get a list of quite active reviewers in Wikimedia Gerrit who have not +2'ed any proposed changes in Gerrit, implying that these developers don't have +2 permissions [2] and could be candidates to receive +2 permissions. Last weekend I created a dashboard at https://wikimedia.biterg.io/app/kibana#/dashboard/ffd01840-cdf9-11ea-8358-4… It's not very performant but it works. On the left you can filter on a specific reviewer (hover over the name and select the "+" magnifier icon). After enough patience you can see on the right the code repositories in which they have been reviewing proposed patches in the timeframe defined in the upper right corner. https://www.mediawiki.org/wiki/Community_metrics#List_of_the_most_active_re… is the corresponding documentation. Please use the discussion page if you have any comments or questions. Thanks! andre [1] https://phabricator.wikimedia.org/T199385 [2] https://www.mediawiki.org/wiki/Gerrit/Privilege_policy -- Andre Klapper (he/him) | Bugwrangler / Developer Advocate https://blogs.gnome.org/aklapper/

2 1

WikiContrib 2.0 released!
by Raymond Olisaemeka 13 Aug '20

13 Aug '20

Hello Wikimedians, We are proud to announce the release of the second version of the WikiContrib tool: <https://wikicontrib.toolforge.org/>. WikiContrib is an easy-to-use developer metrics tool for visualizing the technical contributions to Wikimedia projects on Phabricator, Gerrit, and Github. You can view fellow Wikimedians' contributions or share with them a link to your contribution activity on WikiContrib. Most relevant changes in this version include: - Feature to visualize contributions to Wikimedia repositories on GitHub. - Unique shareable URL for contributions. - Caching feature to reduce query latency. - Fixes to long-standing critical bugs. - User experience improvements. I worked on these recent changes as part of my Outreachy internship project with guidance and support from my mentors Srishti Sethi < https://phabricator.wikimedia.org/p/srishakatux/>, Rammanoj Potla <https://phabricator.wikimedia.org/p/rammanojpotla/>, and Suchakra Sharma <https://phabricator.wikimedia.org/p/Tuxology/>. Relevant Links: - Tool: https://wikicontrib.toolforge.org - Source code: https://github.com/wikimedia/wikicontrib - Proposal: https://phabricator.wikimedia.org/T249175 - Documentation: https://wikicontrib.readthedocs.io/ We invite you to try the tool, and if you encounter any bugs, it would be helpful if you could report them on the GitHub repository <https://github.com/wikimedia/WikiContrib>. We would also love to hear what more you would like to see in the tool, how you would want to use it, and any other ideas you might have on the Tool’s talk page: https://meta.wikimedia.org/wiki/Talk:WikiContrib. Thanks, Raymond Ndibe

3 3

TechCom Radar 2020-08-12
by Alexandra Paskulin 13 Aug '20

13 Aug '20

Hi all, Here are the minutes from this week's TechCom meeting: == RFC: Drop support for database upgrade older than two LTS releases == <https://phabricator.wikimedia.org/T259771> * New RFC * Moved from P1: Define to P2: Resource * TT: The “UPGRADE” documentation file has a warning about upgrading from certain older versions being unsupported. Maybe formalizing that in some way would be useful. Interesting that the install is mentioned as slow since it’s less than a second in CI as far as I know. * DK: It’s hard to reason about database updates, so there’s mental load involved. Unclear what to do with patches when changing database fields. There was an idea that maintenance scripts would run after the schema changes, but it doesn’t work in all cases, so now there are scripts that run at different points in the process, which introduces additional complexity. * TT: Right, running it mid-migration makes the state easier to reason about, but falls apart if it runs dynamic MW code that can’t/shouldn’t support running on an old MW schema. * DK: This is a question of MediaWiki as a product, which doesn’t have a clear owner. Who has the authority to make this decision? * TS: The people maintaining it are the most impacted, so it makes sense for them to make the decision since it is related to developer productivity. * TT: The RFC can help bring together different points of view on this, have teams weigh in on the costs from their perspectives. * TS: Two LTS releases sounds like a pretty short time compared to the current policy. * DK: I think two is reasonable (four years). One cost is testability; it’s hard to test this update logic. == Stable Interface Policy: Timeline for removing obsolete code == <https://www.mediawiki.org/wiki/Topic:Vrwr9aloe6y1bi2v> * Regarding a proposal to change wording on the talk page * DK: The stable interface policy didn’t change the deprecation policy. Things that aren’t used within the MediaWiki ecosystem don’t have to use the deprecation policy. The intention was to make it easy to remove obsolete code that isn’t used. So the question is if you make code obsolete and then remove usages, can you remove it? * TT: Hard deprecation is fine, not the same thing. * DK: Depending on the interpretation of the policy, we could remove anything if we remove all usages of it. * TT: It’s meant to include a requirement to email wikitech-l with a notice and justification. I think this made sense when there was an assumption of stable by default. If we intentionally create an extension for public use, I’m not sure why we’d need to bypass deprecation. * DK: There should be a clear, easy path for getting something into the ecosystem. * TT: Codesearch goes through external repositories as well. The process for adding things to this isn’t easy or transparent, which needs to be improved. The policy incentivizes the Wikimedia Foundation to make a choice to help third party users migrate or continue support for some time. As phrased now, the policy requires the code to no longer be used in the default branch, allowing for the code manager to reject the change. * TT: We can resolve within TechCom, but we should send a message to wikitech-l. == Introduce Authority objects to represent the user performing a given action == <https://phabricator.wikimedia.org/T231930> * DK: Strawman design for managing permission checks with an experimental patch. <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/582889/> There may be resourcing available to start working on this in the next few weeks. We could try to come up with a comprehensive design and submit an RFC or do it the experimental way and use it in one corner of the codebase. * TT: This is about making sure we can’t accidentally bypass permission checks for non-current users. Experimenting seems fine, if there’s a commitment to actively limit it to a narrow scope and not allow other features to adopt it. * DK: The authority is something that actually encapsulates the checks themselves, as opposed to a user. We’re missing a clean place to store state for IP addresses when checking for IP-address blocks. Also applies to OAuth sessions. This new idea partially supersedes permission manager, so there’s some sunken cost there. * TT: You might also want to talk to the AHT team about their experience with Block Manager. Similar issues. * DK: The tradeoff is more flexibility now and a commitment to clean it up later. Thinking about this as example-driven development. Trying to think through everything in advance tends to bring up important edge cases and problems, but it tends to change again during implementation. The goal is to provide more room for exploration and experimentation without losing the feedback process. * TT: Needs an RFC to be used outside of the narrowly-defined, experimental scope == Next week IRC office hours == No IRC discussion scheduled for next week You can also find our meeting minutes at <https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes> See also the TechCom RFC board <https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>. If you prefer you can subscribe to our newsletter here <https://www.mediawiki.org/wiki/Newsletter:TechCom_Radar> -- Alex Paskulin Technical Writer Wikimedia Foundation

1 0

List of all articles about cities?
by Thomas Güttler Lists 13 Aug '20

13 Aug '20

Hi, I would like to download all articles about cities and then do some machine learning fun with the text For example: https://en.wikipedia.org/wiki/New_York_City https://en.wikipedia.org/wiki/Rio_de_Janeiro ... Do you have an idea how I can get a list of articles which are about a city? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines

4 4

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2020