TL;DR: I've decided to sunset my "snapshots" tool.
The Snapshots tool created TAR-archives of MediaWiki core branches, fresh
from Gerrit, every hour.
I created it in 2012 on the Toolserver,[1] to make it easier for site
admins to try out the alpha version of MediaWiki (or a WMF branch), using
the same format as our official stable releases. The snapshots were
generated using a PHP script and git-cli commands, scheduled with a cronjob
onto the Toolforge Grid. [2] [3]
### *Lessons* learned
Maintaining this tool has mostly been an exercise in learning how hard it
is to keep a Git repository functional over a large period of time.
I learned about the numerous ways that a Git repository can become
corrupted or unusable when commands are terminated in unforeseen ways. For
example, what happens to the state of a Git repository when it's a clone of
MediaWiki core, put on NFS, and you try to switch from current master to a
release branch from a decade ago – while lots of users are also working on
that same NFS mount, and do this for every branch – every hour? [4]
I learned how poor Git can be at forgetting which files are part of a
branch and which aren't, so that even when there aren't any errors, if you
switch to an old branch with directories or submodules that a newer branch
doesn't have and then switch back, old files could stay and be seen as
untracked files. These then cause failed checkouts later on due to
conflicting changes when another branch does have the file in question.
(This was improved a lot around 2015 with later releases of Git 2.x.)
I learned that, apparently, when Git's garbage collector kicks in from time
to time, it doesn't know how much memory it is allowed to use, and will
"efficiently" use larger and larger blocks to speed up the process until it
gets killed by the grid engine, at which point it will eventually start
again and make the same mistake, until someone comes in and manually runs
git-gc outside the grid. (This was initially worked around by disabling
git-gc. I later found a way to re-enable with more constrained settings,
see [5]).
### *Sunsetting*
The tool hasn't seen much use (to my knowledge) – apart from web crawlers
of search engines. I haven't heard much complaining over the years (if at
all) whenever it got stuck for long periods at a time.
Last week, I noticed it once again got stuck, and apparently had been for
several months. Rather than fixing it, I decided to shut it down this time.
The source code is available on GitHub for anyone interested in picking it
up again.[3] My recommendation would be to *not* try to maintain a local
Git repository like I did. Instead, have everything be ephemeral. That is,
whenever you run the script, create for each branch you're interested in, a
temporary clone with limited depth and just that branch, then create an
archive and get rid of the clone (also before beginning, in case something
was left behind). This will make it a bit slower, and less elegant, but
presumably much more stable. Actually, given how slow branch switches can
be, it might even be faster! There is also support in newer versions of Git
to invoke git-archive directly on a remote URL, which would remove the need
for local clones entirely. [6]
### *Recipe* for (mostly stable) creation of tarballs from Git
My final hourly recipe looked like this:
1. Blindly delete any ".git/index.lock" file.
2. Run "git clean -q -d -x -f'", deletes unknown files with extra force.
3. Run "git reset -q --hard", deletes any locally staged state, with extra
force. (While nothing does any staging in this script, Git would sometimes
magically think a file was staged. If I recall correctly, this related to
extension submodules.)
4. Get name of remote.
5. Run "git fetch origin".
6. Run "git remote prune origin".
7. Get list of branches (then filter by pattern). Then, for each branch:
8. Get $head of tree for branch via "git rev-parse --verify $branch".
9. Check if you've already got an archive for that. If so, continue with
the next branch instead. If not, go on:
10. Repeat steps 1-3 to reset the repo. [4]
11. Run "git checkout -q -f $branch", checks out the branch, with extra
force.
12. Run "git rev-parse --verify HEAD", and confirm it matches $head,
because sometimes checkout command succeeded, but not really.
13. Run "git archive HEAD --format='tar' | gzip > mediawiki-$branch-$
head.tar.tz", which creates the actual archive.
14. If for "master" branch, update the Mediawiki-latest.tar.gz symlink.
15. (End of for-each branch). Delete older tar files for branches that we
created a new one for just now.
It lived at https://tools.wmflabs.org/snapshots, which now redirects to
https://www.mediawiki.org/wiki/Snapshots instead.
Best,
-- Krinkle
[1] Toolserver. – https://en.wikipedia.org/wiki/Wikipedia:Toolserver
[2] Toolforge Grid. –
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid
[3] The script. –
https://github.com/Krinkle/mw-tool-snapshots/blob/b33d479cb9/scripts/update…
[4] The reset. –
https://github.com/Krinkle/toollabs-base/blob/v1.0.2/src/GlobalFunctions.ph…
[5] More about Git GC and memory configuration management. –
https://github.com/Krinkle/mw-tool-snapshots#git-memory
[6] Use git-archive on a remote repo, without a local clone. –
https://git-scm.com/docs/git-archive/2.18.0#git-archive---remoteltrepogt
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi,
It's time to work on upgrading phan, the PHP static analysis tool we
use! The version of phan we were using was released over two years ago
(0.8.0) - we're moving to 1.2.4 that was released less than a week
ago. And the CI infrastructure is now in place to facilitate easy
upgrades in the future.
The new version comes with lots of upstream bug fixes and feature
requests. We're also able to get rid of our custom wrapper scripts
that hacked around limitations in phan as well.
I've filed bugs for every extension in Gerrit to upgrade phan[1],
including links to migration steps. For some extensions the upgrade
will be trivial, but for others it will be rather involved. I do
expect that we will likely find some bugs or missing features in phan,
but now we should be able to file upstream issues in those cases since
we're no longer running a super out of date version.
In the second extension I ran it against, it was able to find actual
bugs (e.g. [2]). So I'm pretty hopeful that this will be an overall
improvement.
The CI tutorial[3] has also been updated for the new-style phan setup.
Let me know if you have any questions/comments/etc.
[1] https://phabricator.wikimedia.org/maniphest/query/KxjdNDM65iNM/
[2]
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/4925
27
[3]
https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Add_phan
_to_a_MediaWiki_extension
- -- Legoktm
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE2MtZ8F27ngU4xIGd8QX4EBsFJpsFAlxyUCIACgkQ8QX4EBsF
JptcIQ//TmSWriBhI1tBJVELo9Cqb+vuSyUMVFG7X+dVIYxkF6pyci1NiTuIUfxx
+U+LYxjo+9tVBez4UXnIuyuV8JK6EdhBqpaMHqnMNJnzA11/hL7xc8Dc5hm0Oc2j
WIUwrfEyolO/HzlYtRAXRw+q9uDVoMot5IvG0rLbOgeMaDqsuhihfhtOqxbxIjzI
UVY4IpiHQQlJpx5MS80AffxkFtxcs/E+oLrHOAYRut76PWcvoX/q2pO4uzLV/YK+
aQov+fg9/6JQLn9wCEZjh0Vn/KRobJPntOMErJQoGzqdBx1ysqDQ/3aqWqgp2w+E
QV79QxjH5SVLV41pumkH5mf9ADm9b61ocNXP3c7SY7TO9FyqEwrn6qI5DVndC82B
CriWWLrFYL5JSb4/7ORJ+G/8QPk8AT9Z/3iW2qmdYu4tGnNNqBcP+k2yNwY0xazG
sRgn2BsgpNhGZmqCdcnlHtBq8Dn0BgtBGylz90H03MNOol5ABfqqTW60esNWmBnZ
/ZV67nS0B9bUFk6FYx8DP99+8jO/fBeDXaQvA29e2F4VGAR9w/pI3uCteTbmY0LU
DJx2nAwyOJlcJ9CfScFYUc3BY9dOuewAQJxzZwLrzh8QXRjhk5XMrAV+S+BKcFxG
U63HVxVKeqlLANQErhyPViPYI1cXAwUwKhgwvk3reauz8kJw3vY=
=JfvQ
-----END PGP SIGNATURE-----
Hi folks,
Based on comments that I received on Wikimedia-l, I would like to invite
people to a casual online meetup one hour before the monthly WMF Metrics
and Activities Meeting.
There will be no set agenda. You can come with questions or ideas that you
would like to discuss. Please be willing to listen to questions and ideas
from other Wikimedians.
I will host the meeting with the Zoom software. You can join with software
or by using your phone. If you join by phone then your phone number will be
visible to other participants.
The primary language of the meeting will be English, but if people would
like to communicate in diverse languages then that is okay too. We can
facilitate translation by text chat. Many Wikimedians, myself included, are
multilingual in varying degrees, so we might try to have live
interpretation also.
Here is information about how to connect:
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/136978210
Or iPhone one-tap :
Argentina: +543415122188,,136978210#
Or Telephone:
Dial (for higher quality, dial a number based on your current
location):
Argentina: +54 341 512 2188
Australia: +61 (0) 2 8015 2088 or +61 (0) 8 7150 1149
Canada: +1 647 558 0588
Hong Kong, China: +852 5808 6088
France: +33 (0) 1 8288 0188 or +33 (0) 7 5678 4048
Germany: +49 (0) 30 3080 6188 or +49 (0) 30 5679 5800
Israel: +972 (0) 3 978 6688
Italy: +39 069 480 6488
Japan: +81 (0) 3 4578 1488 or +81 524 564 439
Mexico: +52 229 910 0061 or +52 554 161 4288
Spain: +34 84 368 5025 or +34 91 198 0188
Sweden: +46 (0) 7 6692 0434 or +46 (0) 8 4468 2488
Russia: +7 495 283 9788
United Kingdom: +44 (0) 20 3051 2874 or +44 (0) 20 3695 0088
US: +1 408 638 0986 or +1 646 558 8665
Meeting ID: 136 978 210
International numbers available: https://zoom.us/u/ekaPibJIy
The first "Wikimedia Café" meetup will be on 30 August 2018, at 17:00 UTC /
10:00 Pacific.
Let me emphasize that the environment won't be like this
<https://en.wikipedia.org/wiki/File:West_Hartford,_Connecticut_health_care_r…>,
so please don't feel intimated if you are nervous about public speaking.
(If a conversation feels to me like it is becoming uncivil or intimidating,
then I will ask the debaters to quiet themselves or to move to somewhere
else.) The meeting will generally have an environment that is more like this
<https://en.wikipedia.org/wiki/File:Caf%C3%A9_M%C3%A9lange,_Wien.jpg> or
this
<https://en.wikipedia.org/wiki/File:Takamatsu-Castle-Building-Interior-M3488…>.
I anticipate that few people will come, which is okay. I hope that if you
come then you will enjoy the environment and conversation.
Until next time,
Pine
( https://meta.wikimedia.org/wiki/User:Pine )
I am pleased to announce that the Security Team has a new Security Reviews
process.
"Wikimedia Security Team/Standard Operating Procedure/Security Readiness
Reviews" [0] replaces the former "Wikimedia Security Team - Security
Reviews" [1].
Please note: one of the new requirements is that there is *30-day*
pre-deployment date submission.
If you have any questions after reviewing the Standard Operating Procedure
(SOP), please contact the Security Team at security-team(a)wikimedia.org.
Thank you,
Charlotte Portero
Project Manager, Security
[0]
https://www.mediawiki.org/wiki/Wikimedia_Security_Team/Standard_Operating_P…
Security_Readiness_Reviews
[1]
https://www.mediawiki.org/w/index.php?title=Wikimedia_Security_Team/Securit…
Hello all!
I have not yet started the 1.33.0-wmf.18 train; however, at the end of
last week, I noticed some errors that (AFAICT) are regressions in
1.33.0-wmf.17.
There were two new errors that started showing up in 1.33.0-wmf.17:
1. ErrorException from includes/HeaderCallback.php: PHP Notice:
Undefined offset: 1[0]
2. includes/specials/pagers/ActiveUsersPager.php: PHP Notice: Undefined
index: dir[1]
Neither of these errors were happening at a high enough rate, or with
enough of a user impact to trigger a rollback of 1.33.0-wmf.17; however,
I added them as blockers for wmf.18 in the hopes that we could address
regressions caused from wmf.17 before rolling out wmf.18.
If folks could take a look at these tasks and help me resolve these
regressions before we start rollout of a new version that'd be great!
Thanks in advance for your help and attention!
-- Tyler
[0]. <https://phabricator.wikimedia.org/T216086>
[1]. <https://phabricator.wikimedia.org/T216200>
Reminder: Technical Advice IRC meeting this week **(Wednesday) 4-5 pm UTC**
on #wikimedia-tech.
Question can be asked in English, Bulgarian & Hungarian.
The Technical Advice IRC Meeting is a weekly support event for volunteer
developers. Every Wednesday, two full-time developers are available to help
you with all your questions about Mediawiki, gadgets, tools and more! This
can be anything from "how to get started" over "who would be the best
contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting:
https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
Hope to see you there!
Michi (for the Technical Advice IRC Meeting crew)
--
Michael F. Schönitzer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
https://wikimedia.de
Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit
teilhaben, es nutzen und mehren können. Helfen Sie uns dabei!
https://spenden.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hi, i am pleased to announce that the zuul-status plugin is now available for gerrit 2.16+.
This plugin integrates the zuul status json response into gerrit displaying the jobs lined up and running. It's similar to how you would see it on https://integration.wikimedia.org/zuul/ apart from it will only show for your change only. This will improve the experience as you won't need to leave your change to get a live feed of your change as it progresses through checks.
You can see the demo at https://imgur.com/a/uBk2oxQ
Plugin at https://gerrit-review.googlesource.com/admin/repos/plugins/zuul-status
First commit: https://gerrit-review.googlesource.com/c/plugins/zuul-status/+/212103
The plugin uses PolyGerrit's ui rather then GWTUI as GWTUI has been removed upstream (from gerrit 3.0) and because PolyGerrit provides XSS protection.
