Hey all,
A note that WMF Release Engineering are spending a sprint focusing as a
team on GitLab migration efforts. *As a consequence, we're pausing the
deployment train for the week of **M**ay 9th*so we can work on the
problem as a group without losing 1-2 people to train duties for most of
the week.
Unlike our recent experiment at running multiple trains in a week, this
/shouldn't/have much immediate impact on your work, but if you're
interested in GitLab adoption, you might want to follow along on the
sprint workboard:
https://phabricator.wikimedia.org/project/board/5873/
Goal for this sprint:***Migration of blubber / blubberoid to GitLab*
* https://phabricator.wikimedia.org/T301168
* What that means;
* Project exists on GitLab
* CI runs on commits
* We can publish an image from GitLab
<3
-- Brennen Bearnes, forRelease Engineering
Since the beginning of the year, the Wikimedia Language team has enabled
translation backports for MediaWiki core, extensions and skins hosted on
Gerrit. On a weekly schedule compatible translations from master branch are
backpored to all the supported release branches. Currently supported
branches are 1.35–1.38.
Translation backports partially replace the purpose of the
LocalisationUpdate extension. Wikimedia sites no longer use the extension,
and to our knowledge only a few other users of the extension exist, because
it needs manual setup to use.
We, the Language team, think that maintaining the LocalisationUpdate
extension is no longer a good use of our time. We are asking for your
feedback about the future of this extension.
We are planning to:
* Remove LocalisationUpdate from the MediaWiki Language Extension Bundle
starting from version 2022.07
* Remove us as maintainers of the extension
Additionally, based on the feedback, we are planning to either mark the
extension as unmaintained, transfer maintenance to a new maintainer, or
request the extension to be archived and removed from the list of
extensions bundled with MediaWiki core if there is no indication that
anyone uses this extension.
We request your feedback and welcome discussion on
https://phabricator.wikimedia.org/T300498. Please let us know if you are
using this extension and whether you would be interested in maintaining it.
*Anticipated questions*
Q: What about Wikimedia sites: does this mean they will not get frequent
translation updates as they used to have?
A: We still think this is important, but we do not think the previous
solution can be restored. We would like to collaborate on new solutions.
One solution could be more frequent deployments.
-Niklas
Hi everyone,
I just realized I probably sent my original message (see below) to the
wrong list. So here goes again, this time with a deeper understanding
what is going on, but also still much doubt and confusion.
tl;dr on the original post: Users are being randomly disconnected from
our mediawiki instances.
I would like to check with you if what I found is a bug or not and if
it's not, then I'm looking to understand why it behaves this way. I am
also looking for information on what triggers garbage collection of PHP
sessions.
Since I originally posted, I have found out what is deleting sessions
from the objectcache table in the database. I have been able to
reproduce this both on a remote server and locally. Both times, my
set-up is running in a docker environment.
The call-stack that does this is the following:
#0 SqlBagOStuff->deleteServerObjectsExpiringBefore() called at
[/var/www/html/includes/objectcache/SqlBagOStuff.php:668]
#1 SqlBagOStuff->deleteObjectsExpiringBefore() called at
[/var/www/html/includes/libs/objectcache/CachedBagOStuff.php:148]
#2 CachedBagOStuff->deleteObjectsExpiringBefore() called at
[/var/www/html/includes/session/PHPSessionHandler.php:376]
#3 MediaWiki\Session\PHPSessionHandler->gc()
#4 session_start() called at [/var/www/html/includes/Setup.php:757]
#5 require_once(/var/www/html/includes/Setup.php) called at
[/var/www/html/includes/WebStart.php:89]
#6 require(/var/www/html/includes/WebStart.php) called at
[/var/www/html/index.php:44]
Usually, SqlBagOStuff::deleteServerObjectsExpiringBefore is called
fromMWCallableUpdate->doUpdate() and passed a Unix timestamp. This
timestamp is a UTC value and handled as such.
However, sometimes, as can be seen from the backtrace above, it is
called from the Garbage Collector in PHPSessionHandler:
public function gc( $maxlifetime ) {
if ( self::$instance !== $this ) {
throw new \UnexpectedValueException( __METHOD__ . ': Wrong
instance called!' );
}
$before = date( 'YmdHis', time() );
$this->store->deleteObjectsExpiringBefore( $before );
return true;
}
The problem here is that – probably for historical reasons - the
function is passed a date string here instead of a Unix timestamp. And
*the actual problem with that* is that the date() function applies
timezone information (if $wgLocaltimezone is set). The resulting
timestamp will thus be in the future (or be local time, interpreted as
UTC). With a value of something equivalent to GMT+2, gc() will cause all
current sessions to be deleted since they have a standard expiry time of
1h, stored as UTC value in the database.
If this happens, and the session is not declared persistent
($session->isPersistent(), "Keep me logged in"), then all users are
immediately logged out of mediawiki.
Thus, my first question: Is this a bug? Since the $timestamp parameter
is passed to ConvertibleTimestamp::convert( TS_UNIX, $timestamp ) in
SqlBagOStuff::deleteServerObjectsExpiringBefore anyway, I do not see any
reason to pass a date string. My suggestion for a fix would be to change
the gc method as follows:
public function gc( $maxlifetime ) {
if ( self::$instance !== $this ) {
throw new \UnexpectedValueException( __METHOD__ . ': Wrong
instance called!' );
}
$this->store->deleteObjectsExpiringBefore( microtime(true) ); //
time() would be sufficient
return true;
}
My second question would be: What controls when PHPSessionHandler::gc is
called? Is it indeed the session configuration for the PHP process (i.e.
session.gc_probability, session.gc_divisor and session.gc_maxlifetime) ?
Thanks,
David
> Hi all,
>
>
> I'm writing to the list in the hope of receiving some feedback,
> questions, maybe answers to help us solve a conundrum with sessions we
> have on our Wikibase/Mediawiki installation.
>
>
> We're running two different docker-based wikibase installations (one
> staging, one production) on two separate virtual machines. Both use
> the SimpleSaml extension to connect to a SAML implementation, but we
> have found that the random session deletion happens both with and
> without the SAML extension used.
>
>
> The symptoms that we've seen are that out of the blue, users are
> disconnected from mediawiki. Since we use SAML, it's enough to click
> the Login link to again be connected.
>
>
> The duration or number of requests until a session is deleted seems
> random. It does appear though, that the more (or more frequent)
> requests are made (and thus background jobs run) the quicker a session
> is deleted.
>
>
> We have even tried setting $wgObjectCacheSessionExpiry = 7200; in
> order to exclude any TimeZone related issues which was our first
> suspicion. However, changing this from the default 1h session expiry
> does not change the behavior we're seeing.
>
>
> Sessions are deleted regardless of their nature. E.g. sessions
> established for oauth connections are deleted in the same way and at
> the same time as other sessions. E.g. using wikidataintegrator we're
> able to run a number of requests until the CSRF token expires (with
> the number of requests that succeed before this happens being random):
>
> […]
> Created Item Q379 from Classhttp://www.cidoc-crm.org/cidoc-crm/E34_Inscription.
> Created Item Q380 from Classhttp://www.cidoc-crm.org/cidoc-crm/E53_Place.
> Errorwhile writing to Wikidata
> ERROR creating class referencefor http://www.cidoc-crm.org/cidoc-crm/E35_Title: {'error': {'code':'badtoken','info':'Invalid CSRF token.','*':'See https://saf-dev.bnl.lu/w/api.php for API usage. Subscribe to the
> mediawiki-api-announce mailing list at
> <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce>
> for notice of API deprecations and breaking changes.'}}
> Errorwhile writing to Wikidata
> ERROR creating object property referencefor http://www.cidoc-crm.org/cidoc-crm/P16_used_specific_object: {'error': {'code':'badtoken','info':'Invalid CSRF token.','*':'See https://saf-dev.bnl.lu/w/api.php for API usage. Subscribe to the
> mediawiki-api-announce mailing list at
> <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce>
> for notice of API deprecations and breaking changes.'}}
>
>
>
> If sessions are not randomly deleted, then session renewal seems to
> work as planned with them being renewed after half their expiry time:
>
> |[DBQuery] SqlBagOStuff::fetchBlobMulti [0s] db.svc:3306: SELECT
> keyname,value,exptime FROM `objectcache` WHE RE keyname =
> 'mediawiki:MWSession:n9krnsfa303qk9c2osp6c81tp1k2bp5b' [session]
> SessionBackend
> "require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend-
> >renew" metadata dirty for renew():
> require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\Sessi
> onBackend->renew [session] SessionBackend
> "n9krnsfa303qk9c2osp6c81tp1k2bp5b" force-persist for renew():
> require/require_once/Media
> Wiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew
> [session] SessionBackend "n9krnsfa303qk9c2osp6c81tp1k2bp5b" save:
> dataDirty=0 metaDirty=1 forcePersist=1 [cookie] setcookie:
> "mediawiki_session", "n9krnsfa303qk9c2osp6c81tp1k2bp5b", "0", "/", "",
> "1", "1", "" [cookie] setcookie: "mediawikiUserID", "13",
> "1648233666", "/", "", "1", "1", "" [cookie] setcookie:
> "mediawikiUserName", "Ibl676", "1648233666", "/", "", "1", "1", ""
> [cookie] already deleted setcookie: "mediawikiToken", "",
> "1614105666", "/", "", "1", "1", "" [cookie] already deleted
> setcookie: "forceHTTPS", "", "1614105666", "/", "", "", "1", ""
> [session] SessionBackend "n9krnsfa303qk9c2osp6c81tp1k2bp5b" Taking
> over PHP session [session] SessionBackend
> "n9krnsfa303qk9c2osp6c81tp1k2bp5b" save: dataDirty=0 metaDirty=1
> forcePersist=1 [cookie] already set setcookie: "mediawiki_session",
> "n9krnsfa303qk9c2osp6c81tp1k2bp5b", "0", "/", "", "1", "1", ""
> [cookie] already set setcookie: "mediawikiUserID", "13", "1648233666",
> "/", "", "1", "1", "" [cookie] already set setcookie:
> "mediawikiUserName", "Ibl676", "1648233666", "/", "", "1", "1", ""
> [cookie] already deleted setcookie: "mediawikiToken", "",
> "1614105666", "/", "", "1", "1", "" [cookie] already deleted
> setcookie: "forceHTTPS", "", "1614105666", "/", "", "", "1", ""
> [DBQuery] SqlBagOStuff::updateTable [0.002s] db.svc:3306: REPLACE INTO
> `objectcache` (keyname,value,exptime) VALU ES
> ('mediawiki:MWSession:n9krnsfa303qk9c2osp6c81tp1k2bp5b','...\0','20220223194106')|
>
> Here is one example where the session got deleted by whatever feels
> responsible to do so:
>
> MariaDB [mediawiki]> select keyname, exptimefrom objectcachewhere keynamelike 'mediawiki:MWSession%';
> +------------------------------------------------------+---------------------+
> | keyname | exptime |
> +------------------------------------------------------+---------------------+
> | mediawiki:MWSession:t7d5lms2nrma6k627c6usrgc6289en3a |2022-03-24 17:34:34 |
> +------------------------------------------------------+---------------------+
> 1 rowin set (0.001 sec)
>
> MariaDB [mediawiki]> select keyname, exptimefrom objectcachewhere keynamelike 'mediawiki:MWSession%';
> Emptyset (0.001 sec)
>
> MariaDB [mediawiki]> select now();
> +---------------------+
> | now() |
> +---------------------+
> |2022-03-24 17:37:15 |
> +---------------------+
> 1 rowin set (0.024 sec)
>
>
> The only instance of me finding something remotely related to randomly
> deleting a session would be these log entries:
>
>
> |2022-03-24 17:03:29 51290acae35a mediawiki: SessionManager using
> store SqlBagOStuff 2022-03-24 17:03:29 51290acae35a mediawiki: Saving
> all sessions on shutdown 2022-03-24 17:03:29 51290acae35a mediawiki:
> SessionManager using store SqlBagOStuff 2022-03-24 17:03:29
> 51290acae35a mediawiki: Session
> "[30]MediaWiki\Session\CookieSessionProvider<-:13:Ibl676>8favvi92d08njut82j66un60r601titn":
> Unverified user provided and no metadata to auth it 2022-03-24
> 17:03:29 51290acae35a mediawiki: setcookie: "mediawiki_session", "",
> "1616605409", "/", "", "1", "1", "" 2022-03-24 17:03:29 51290acae35a
> mediawiki: setcookie: "mediawikiUserID", "", "1616605409", "/", "",
> "1", "1", "" 2022-03-24 17:03:29 51290acae35a mediawiki: already
> deleted setcookie: "mediawikiToken", "", "1616605409", "/", "", "1",
> "1", "" 2022-03-24 17:03:29 51290acae35a mediawiki: already deleted
> setcookie: "forceHTTPS", "", "1616605409", "/", "", "", "1", ""
> 2022-03-24 17:03:29 51290acae35a mediawiki: SessionBackend
> "ujf6rcheseb51lpaojrqggb4c23s1flr" is unsaved, marking dirty in
> constructor 2022-03-24 17:03:29 51290acae35a mediawiki: SessionBackend
> "ujf6rcheseb51lpaojrqggb4c23s1flr" save: dataDirty=1 metaDirty=1
> forcePersist=0 2022-03-24 17:03:29 51290acae35a mediawiki: already
> deleted setcookie: "mediawiki_session", "", "1616605409", "/", "",
> "1", "1", "" 2022-03-24 17:03:29 51290acae35a mediawiki: already
> deleted setcookie: "mediawikiUserID", "", "1616605409", "/", "", "1",
> "1", "" 2022-03-24 17:03:29 51290acae35a mediawiki: already deleted
> setcookie: "mediawikiToken", "", "1616605409", "/", "", "1", "1", ""
> 2022-03-24 17:03:29 51290acae35a mediawiki: already deleted setcookie:
> "forceHTTPS", "", "1616605409", "/", "", "", "1", "" 2022-03-24
> 17:03:29 51290acae35a mediawiki: Saving all sessions on shutdown |
>
> IBL676 is my SAML ID. I'm not sure why the user is unverified and what
> metadata would be used/required to verify it?
>
>
>
> Looking forward to any advice on how to further investigate what is
> going on here.
>
--
*TenTwentyFour S.à r.l.*
www.tentwentyfour.lu <https://www.tentwentyfour.lu>
*T*: +352 20 211 1024
*F*: +352 20 211 1023
1 place de l'Hôtel de Ville
4138 Esch-sur-Alzette
Hello,
The 1.39.0-wmf.10 version of MediaWiki is blocked[0].
The new version is deployed to group0 but can proceed no further until
these issue is resolved:
* wbsearchentities produces no results -
https://phabricator.wikimedia.org/T307586
It has been reported by Lucas Werkmeister from WMDE and I have
immediately rolled back group1 wikis.
Once the issue is resolved train can resume. If the issue is resolved on
Friday the train will resume Monday.
Thank you for your help resolving these issues!
-- Your humble train toiler
[0]. https://phabricator.wikimedia.org/T305216
[1]. https://versions.toolforge.org/
The Search Platform Team
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds an
open meeting on the first Wednesday of each month. Come talk to us about
anything related to Wikimedia search, Wikidata Query Service (WDQS),
Wikimedia Commons Query Service (WCQS), etc.!
Feel free to add your items to the Etherpad Agenda for the next meeting.
Details for our next meeting:
Date: Wednesday, May 4th, 2022
Time: 15:00-16:00 GMT / 08:00-09:00 PDT / 11:00-12:00 EDT / 16:00-17:00 WAT
/ 17:00-18:00 CEST
Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours
Google Meet link: https://meet.google.com/vgj-bbeb-uyi
Join by phone: https://tel.meet/vgj-bbeb-uyi?pin=8118110806927
Hope to talk to you next week!
—Trey
Trey Jones
Staff Computational Linguist, Search Platform
Wikimedia Foundation
UTC–4 / EDT
Hi Community Metrics team,
This is your automatic monthly Phabricator statistics mail.
Accounts created in (2022-04): 360
Active Maniphest users (any activity) in (2022-04): 1063
Task authors in (2022-04): 572
Users who have closed tasks in (2022-04): 295
Projects which had at least one task moved from one column to another on
their workboard in (2022-04): 300
Tasks created in (2022-04): 2072
Tasks closed in (2022-04): 1772
Open and stalled tasks in total: 49426
* Only open tasks in total: 48514
* Only stalled tasks in total: 912
Median age in days of open tasks by priority:
Unbreak now: 36
Needs Triage: 764
High: 1156
Normal: 1648
Low: 2324
Lowest: 2403
(How long tasks have been open, not how long they have had that priority)
Active Differential users (any activity) in (2022-04): 7
To see the names of the most active task authors:
* Go to https://wikimedia.biterg.io/
* Choose "Phabricator > Overview" from the top bar
* Adjust the time frame in the upper right corner to your needs
* See the author names in the "Submitters" panel
TODO: Numbers which refer to closed tasks might not be correct, as
described in https://phabricator.wikimedia.org/T1003 .
Yours sincerely,
Fab Rick Aytor
(via community_metrics.sh on phab1001 at Sun 01 May 2022 12:00:21 AM UTC)