I lately switched Commons coordinate template https://commons.wikimedia.org/wiki/Template:Location from using dozens of subtemplates to much cleaner Lua code. Everything works great, except for Special:WhatLinksHere links. The no longer used subtemplates like https://commons.wikimedia.org/wiki/Template:Location/layout still show over 3M files transcluding it and files, like https://commons.wikimedia.org/wiki/File:Estadio_Pacaembu.jpg which still claims to transclude [[Template:Location/layout]] ("Templates used on this page:" section in edit mode). The transclusion dependencies do not correct themselves even with a Purge, but page edit fixes it. Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
In the past when I run into this kind of trouble I waited, but in some cases it took over half a year for the database to update itself. In other cases those issues never clear (or I did not wait long enough), like with Category:Pages with malformed coordinate tags https://commons.wikimedia.org/wiki/Category:Pages_with_malformed_coordi nate_tags , which due to a long fixed bug in {{#coordinates: https://www.mediawiki.org/wiki/Extension:GeoData }} was filled with 40k files which did not have any issues. Then the only solution I know is Pywikibot/touch.py (https://www.mediawiki.org/wiki/Manual:Pywikipediabot/touch.py ), which I recently used to clear {{#coordinates: https://www.mediawiki.org/wiki/Extension:GeoData }} issues.
Is there a better way to synchronize database with reality?
Jarek T.
(user:jarekt http://commons.wikimedia.org/wiki/User:Jarekt )
On Mon, Jan 27, 2014 at 2:12 PM, Tuszynski, Jaroslaw W. JAROSLAW.W.TUSZYNSKI@leidos.com wrote:
I lately switched Commons coordinate template https://commons.wikimedia.org/wiki/Template:Location from using dozens of subtemplates to much cleaner Lua code. Everything works great, except for Special:WhatLinksHere links. The no longer used subtemplates like https://commons.wikimedia.org/wiki/Template:Location/layout still show over 3M files transcluding it and files, like https://commons.wikimedia.org/wiki/File:Estadio_Pacaembu.jpg which still claims to transclude [[Template:Location/layout]] ("Templates used on this page:" section in edit mode). The transclusion dependencies do not correct themselves even with a Purge, but page edit fixes it. Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
In the past when I run into this kind of trouble I waited, but in some cases it took over half a year for the database to update itself. In other cases those issues never clear (or I did not wait long enough), like with Category:Pages with malformed coordinate tags https://commons.wikimedia.org/wiki/Category:Pages_with_malformed_coordi nate_tags , which due to a long fixed bug in {{#coordinates: https://www.mediawiki.org/wiki/Extension:GeoData }} was filled with 40k files which did not have any issues. Then the only solution I know is Pywikibot/touch.py (https://www.mediawiki.org/wiki/Manual:Pywikipediabot/touch.py ), which I recently used to clear {{#coordinates: https://www.mediawiki.org/wiki/Extension:GeoData }} issues.
Is there a better way to synchronize database with reality?
Jarek T.
(user:jarekt http://commons.wikimedia.org/wiki/User:Jarekt )
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
It takes a lot of work to reparse 4 million pages. If it makes you feel better, the toolserver transclusion count you linked to is about 20000 pages higher than the real number of transclusions.
but in some cases it took over half a year for the database to update itself
If something is taking 6 months, it probably means that some sort of bug happened, and the pages aren't being refreshed at all.
The transclusion dependencies do not correct themselves even with a Purge, but page edit fixes it.
Yes, null edits update links tables (e.g. Categories, whatlinkshere, Special:LinkSearch, etc) purges do not.
Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
That shouldn't happen. What sort of things were you doing when the error happened?
--bawolff
So I guess everything works as expected, it just takes a long time.
I guess I do not understand the logic of what triggers the update of the links tables (e.g. Categories, whatlinkshere, Special:LinkSearch, etc) of pages: purge does not, null edit does, but how about edits to transcluded templates or modules? Based on many years of working with Commons high-use templates, it seems like in case of templates with few uses, if changes to the template changes links tables for pages transcluding it than those changes are done in a timely manner. However, high-use templates (which up to about a year ago always showed an error message after each save), are either much slower or might not be triggering the update of the links tables at all. It also seems to depend on how deep on the call-stack is the template which was edited. Finally if the update of the links tables is due to changes in extensions, like Extension:GeoData, etc. than changes to the extension code do not seem to trigger the updates at all.
BTW, the internal database errors when using Special:WhatLinksHere do not seem to be repeatable. They just happen occasionally.
Also I am curious, is there some sort of process that refreshes occasionally all pages that were not refreshed for some period of time?
Jarek T. (user:jarekt)
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of bawolff Sent: Monday, January 27, 2014 6:50 PM To: Wikimedia developers Subject: Re: [Wikitech-l] Template transclusion on Wikimedia Commons
On Mon, Jan 27, 2014 at 2:12 PM, Tuszynski, Jaroslaw W. JAROSLAW.W.TUSZYNSKI@leidos.com wrote:
I lately switched Commons coordinate template https://commons.wikimedia.org/wiki/Template:Location from using dozens of subtemplates to much cleaner Lua code. Everything works great, except for Special:WhatLinksHere links. The no longer used subtemplates like https://commons.wikimedia.org/wiki/Template:Location/layout still show over 3M files transcluding it and files, like https://commons.wikimedia.org/wiki/File:Estadio_Pacaembu.jpg which still claims to transclude [[Template:Location/layout]] ("Templates used on this page:" section in edit mode). The transclusion dependencies do not correct themselves even with a Purge, but page edit fixes it. Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
In the past when I run into this kind of trouble I waited, but in some cases it took over half a year for the database to update itself. In other cases those issues never clear (or I did not wait long enough), like with Category:Pages with malformed coordinate tags <https://commons.wikimedia.org/wiki/Category:Pages_with_malformed_coor di nate_tags> , which due to a long fixed bug in {{#coordinates: https://www.mediawiki.org/wiki/Extension:GeoData }} was filled with 40k files which did not have any issues. Then the only solution I know is Pywikibot/touch.py (https://www.mediawiki.org/wiki/Manual:Pywikipediabot/touch.py ), which I recently used to clear {{#coordinates: https://www.mediawiki.org/wiki/Extension:GeoData }} issues.
Is there a better way to synchronize database with reality?
Jarek T.
(user:jarekt http://commons.wikimedia.org/wiki/User:Jarekt )
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
It takes a lot of work to reparse 4 million pages. If it makes you feel better, the toolserver transclusion count you linked to is about 20000 pages higher than the real number of transclusions.
but in some cases it took over half a year for the database to update itself
If something is taking 6 months, it probably means that some sort of bug happened, and the pages aren't being refreshed at all.
The transclusion dependencies do not correct themselves even with a Purge, but page edit fixes it.
Yes, null edits update links tables (e.g. Categories, whatlinkshere, Special:LinkSearch, etc) purges do not.
Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
That shouldn't happen. What sort of things were you doing when the error happened?
--bawolff
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, 28 Jan 2014 17:04:23 +0100, Tuszynski, Jaroslaw W. JAROSLAW.W.TUSZYNSKI@leidos.com wrote:
Also I am curious, is there some sort of process that refreshes occasionally all pages that were not refreshed for some period of time?
In a way: as far as I know, pages are cached for no more than 30 days – if anybody (anonymous or registered) tries to view a page that was last viewed more than 30 days ago, it's parsed and generated from scratch for them. This doesn't happen if the page is not viewed, thought.
(I might be wrong, in that case hopefully someone will correct me.)
But I assume that is equivalent to a Purge and does not trigger update of the links tables.
Jarek
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Bartosz Dziewonski Sent: Wednesday, January 29, 2014 9:52 AM To: Wikimedia developers Subject: Re: [Wikitech-l] Template transclusion on Wikimedia Commons
On Tue, 28 Jan 2014 17:04:23 +0100, Tuszynski, Jaroslaw W. JAROSLAW.W.TUSZYNSKI@leidos.com wrote:
Also I am curious, is there some sort of process that refreshes occasionally all pages that were not refreshed for some period of time?
In a way: as far as I know, pages are cached for no more than 30 days – if anybody (anonymous or registered) tries to view a page that was last viewed more than 30 days ago, it's parsed and generated from scratch for them. This doesn't happen if the page is not viewed, thought.
(I might be wrong, in that case hopefully someone will correct me.)
Yes that is correct. A parser cache expiry does does not update the links tables.
-bawolff
On Jan 29, 2014 11:03 AM, "Tuszynski, Jaroslaw W." < JAROSLAW.W.TUSZYNSKI@leidos.com> wrote:
But I assume that is equivalent to a Purge and does not trigger update of
the links tables.
Jarek
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:
wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Bartosz Dziewonski
Sent: Wednesday, January 29, 2014 9:52 AM To: Wikimedia developers Subject: Re: [Wikitech-l] Template transclusion on Wikimedia Commons
On Tue, 28 Jan 2014 17:04:23 +0100, Tuszynski, Jaroslaw W. <
JAROSLAW.W.TUSZYNSKI@leidos.com> wrote:
Also I am curious, is there some sort of process that refreshes
occasionally all pages that were not refreshed for some period of time?
In a way: as far as I know, pages are cached for no more than 30 days -
if anybody (anonymous or registered) tries to view a page that was last viewed more than 30 days ago, it's parsed and generated from scratch for them. This doesn't happen if the page is not viewed, thought.
(I might be wrong, in that case hopefully someone will correct me.)
-- Matma Rex
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
That shouldn't happen. What sort of things were you doing when the error happened?
I found how to reproduce the database errors I mentioned earlier. Both
https://commons.wikimedia.org/w/index.php?title=Special%3AWhatLinksHere&... and
https://commons.wikimedia.org/w/index.php?title=Special%3AWhatLinksHere&... gave me
Database error
A database query error has occurred. This may indicate a bug in the software.
Function: SpecialWhatLinksHere::showIndirectLinks
Error: 0
Jarek T.
(user:jarekt http://commons.wikimedia.org/wiki/User:Jarekt )
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of bawolff Sent: Monday, January 27, 2014 6:50 PM To: Wikimedia developers Subject: Re: [Wikitech-l] Template transclusion on Wikimedia Commons
On Mon, Jan 27, 2014 at 2:12 PM, Tuszynski, Jaroslaw W.
<JAROSLAW.W.TUSZYNSKI@leidos.com mailto:JAROSLAW.W.TUSZYNSKI@leidos.com > wrote:
I lately switched Commons coordinate template
https://commons.wikimedia.org/wiki/Template:Location https://commons.wikimedia.org/wiki/Template:Location from using dozens
of subtemplates to much cleaner Lua code. Everything works great,
except for Special:WhatLinksHere links. The no longer used
subtemplates like
https://commons.wikimedia.org/wiki/Template:Location/layout https://commons.wikimedia.org/wiki/Template:Location/layout still show
over 3M files transcluding it and files, like
https://commons.wikimedia.org/wiki/File:Estadio_Pacaembu.jpg https://commons.wikimedia.org/wiki/File:Estadio_Pacaembu.jpg which
still claims to transclude [[Template:Location/layout]] ("Templates
used on this page:" section in edit mode). The transclusion
dependencies do not correct themselves even with a Purge, but page
edit fixes it. Also several times I got some sort of internal database errors when using Special:WhatLinksHere.
In the past when I run into this kind of trouble I waited, but in some
cases it took over half a year for the database to update itself. In
other cases those issues never clear (or I did not wait long enough),
like with Category:Pages with malformed coordinate tags
<https://commons.wikimedia.org/wiki/Category:Pages_with_malformed_coor
di nate_tags> , which due to a long fixed bug in {{#coordinates:
<https://www.mediawiki.org/wiki/Extension:GeoData https://www.mediawiki.org/wiki/Extension:GeoData > }} was filled with
40k files which did not have any issues. Then the only solution I know
is Pywikibot/touch.py
(https://www.mediawiki.org/wiki/Manual:Pywikipediabot/touch.py https://www.mediawiki.org/wiki/Manual:Pywikipediabot/touch.py ),
which I recently used to clear {{#coordinates:
<https://www.mediawiki.org/wiki/Extension:GeoData https://www.mediawiki.org/wiki/Extension:GeoData > }} issues.
Is there a better way to synchronize database with reality?
Jarek T.
(user:jarekt <http://commons.wikimedia.org/wiki/User:Jarekt http://commons.wikimedia.org/wiki/User:Jarekt > )
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org mailto:Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l https://lists.wikimedia.org/mailman/listinfo/wikitech-l
It takes a lot of work to reparse 4 million pages. If it makes you feel better, the toolserver transclusion count you linked to is about
20000 pages higher than the real number of transclusions.
but in some
cases it took over half a year for the database to update itself
If something is taking 6 months, it probably means that some sort of bug happened, and the pages aren't being refreshed at all.
The transclusion dependencies do
not correct themselves even with a Purge, but page edit fixes it.
Yes, null edits update links tables (e.g. Categories, whatlinkshere, Special:LinkSearch, etc) purges do not.
Also
several times I got some sort of internal database errors when using
Special:WhatLinksHere.
That shouldn't happen. What sort of things were you doing when the error happened?
--bawolff
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org mailto:Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 30/01/14 14:39, Tuszynski, Jaroslaw W. a écrit :
I found how to reproduce the database errors I mentioned earlier. Both
https://commons.wikimedia.org/w/index.php?title=Special%3AWhatLinksHere&... and
https://commons.wikimedia.org/w/index.php?title=Special%3AWhatLinksHere&... gave me
Database error
A database query error has occurred. This may indicate a bug in the software. Function: SpecialWhatLinksHere::showIndirectLinks Error: 0
Hello,
The first URL worked for me, the second throws a database error.
Thu Jan 30 14:11:19 UTC 2014 /* query */ Thu Jan 30 14:19:33 UTC 2014 /* query */ Connection lost and reconnected after 59.754s
Can you please fill a bug about it?
Done. See https://bugzilla.wikimedia.org/show_bug.cgi?id=60618
Jarek T. (user:jarekt)
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Antoine Musso Sent: Thursday, January 30, 2014 9:23 AM To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Template transclusion on Wikimedia Commons
Le 30/01/14 14:39, Tuszynski, Jaroslaw W. a écrit :
I found how to reproduce the database errors I mentioned earlier. Both
https://commons.wikimedia.org/w/index.php?title=Special%3AWhatLinksHer e&target=Template%3ALocation%2Flayout&namespace=10 and
https://commons.wikimedia.org/w/index.php?title=Special%3AWhatLinksHer e&target=Module%3AFallbacklist&namespace=828 gave me
Database error
A database query error has occurred. This may indicate a bug in the software. Function: SpecialWhatLinksHere::showIndirectLinks Error: 0
Hello,
The first URL worked for me, the second throws a database error.
Thu Jan 30 14:11:19 UTC 2014 /* query */ Thu Jan 30 14:19:33 UTC 2014 /* query */ Connection lost and reconnected after 59.754s
Can you please fill a bug about it?
-- Antoine "hashar" Musso
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org