Hi,
I have launched an upload of 3000 images using the GWToolset − actually two, one yesterday night with the first 1200 files, another one this morning with the 3K files minus the 100 uploaded in the night
Here are the logs : < https://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=gwtoo...
(batches are dated in the summary)
Two things: * uploads get stalled. Nothing has happened in the past hour, even though the expected time laps between is around 5 minutes. I had assumed yesterday job had crashed because of too many errors (bad source) but today’s one is stalled too and... * ...uploads may resume. My batch of yesterday was blocked in the night ; before resuming today.
Is this normal behaviour? Anything I can do to make it easier for the toolset?
Thanks,
this is a tough one to troubleshoot. the issue seems to be that there was no additional "metadata job created" entry after 14:57, 28 November 2014. a couple of thoughts:
1. the job runner picked up the job that creates additional media file jobs, but something went wrong - is there someone who could investigate what may have happened with the job runner?
2. something went wrong when the job that creates additional media file jobs ran. there should be an exception message in the php log - is there someone who could investigate that?
3. later, i can look into possibly adding further special:log entries that might help further, but i think i did cover those scenarios already; still i can look into it a bit later when i get a chance.
with kind regards, dan
On Nov 28, 2014, at 14:20 , Jean-Frédéric jeanfrederic.wiki@gmail.com wrote:
Hi,
I have launched an upload of 3000 images using the GWToolset − actually two, one yesterday night with the first 1200 files, another one this morning with the 3K files minus the 100 uploaded in the night
Here are the logs : https://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=gwtoolset&user=Jean-Fr%C3%A9d%C3%A9ric
(batches are dated in the summary)
Two things:
- uploads get stalled. Nothing has happened in the past hour, even though the expected time laps between is around 5 minutes. I had assumed yesterday job had crashed because of too many errors (bad source) but today’s one is stalled too and...
- ...uploads may resume. My batch of yesterday was blocked in the night ; before resuming today.
Is this normal behaviour? Anything I can do to make it easier for the toolset?
Thanks,
Jean-Frédéric
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
Hey Dan,
Thanks for the answer.
Since it was urgent I have completed the uploads on Friday using Pywikibot.
Looking at the logs today, it seems that the jobs were unstuck today and yesterday − of course, they failed with « already existing files » errors.
I am tempted to assume that the problem did not lie on the GWT side but on MediaWiki / Wikimedia Commons side.
2014-11-28 15:43 GMT+00:00 dan entous dan.entous.wikimedia@gmail.com:
this is a tough one to troubleshoot. the issue seems to be that there was no additional "metadata job created" entry after 14:57, 28 November 2014. a couple of thoughts:
- the job runner picked up the job that creates additional media file
jobs, but something went wrong - is there someone who could investigate what may have happened with the job runner?
- something went wrong when the job that creates additional media file
jobs ran. there should be an exception message in the php log - is there someone who could investigate that?
Should we open a Phabricator task for this ? Not sure this list is monitored by the relevant folks.
On 1 December 2014 at 19:03, Jean-Frédéric jeanfrederic.wiki@gmail.com wrote: ...
Should we open a Phabricator task for this ? Not sure this list is monitored by the relevant folks.
Please do. This is a tiny list for users, and stuff that pops up we ought to log so that it can be planned for. It's unfair to rely on Dan picking up fixes, even though he does such a good job of it. :-)
Fae
There were job queue issues. https://wikitech.wikimedia.org/wiki/Server_Admin_Log#November_30
Nemo
thanks both.
so it looks like it'd be good to contact the ops team next time this happens via irc or email and/or open a bug for them regarding the job queue.
o dan
On Dec 1, 2014, at 21:21 , Federico Leva (Nemo) nemowiki@gmail.com wrote:
There were job queue issues. https://wikitech.wikimedia.org/wiki/Server_Admin_Log#November_30
Nemo
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
2014-12-01 19:47 GMT+00:00 Fæ faewik@gmail.com:
On 1 December 2014 at 19:03, Jean-Frédéric jeanfrederic.wiki@gmail.com wrote: ...
Should we open a Phabricator task for this ? Not sure this list is
monitored
by the relevant folks.
Please do. This is a tiny list for users, and stuff that pops up we ought to log so that it can be planned for. It's unfair to rely on Dan picking up fixes, even though he does such a good job of it. :-)
Indeed :)
See https://phabricator.wikimedia.org/T76451
Hi,
It seems we have again a similar case here with the Swiss National Library. I have open a bug: https://phabricator.wikimedia.org/T87040
Regards Emmanuel
On 28.11.2014 14:20, Jean-Frédéric wrote:
Hi,
I have launched an upload of 3000 images using the GWToolset − actually two, one yesterday night with the first 1200 files, another one this morning with the 3K files minus the 100 uploaded in the night
Here are the logs : https://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=gwtoolset&user=Jean-Fr%C3%A9d%C3%A9ric
(batches are dated in the summary)
Two things:
- uploads get stalled. Nothing has happened in the past hour, even
though the expected time laps between is around 5 minutes. I had assumed yesterday job had crashed because of too many errors (bad source) but today’s one is stalled too and...
- ...uploads may resume. My batch of yesterday was blocked in the night
; before resuming today.
Is this normal behaviour? Anything I can do to make it easier for the toolset?
Thanks,
Jean-Frédéric
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools