wouldn't it be better to throttle the application/tool that generates thumbnails so that it doesn't try to produce too many thumbnails at once?

Hi Dan, wouldn’t it be better to throttle the application/tool that generates thumbnails so that it doesn’t try to produce too many thumbnails at once? The issue is that there is no application generating thumbnails at a given rate. Thumbnails are being generated on demand when people view a thumbnail that doesn't exist. And since Special:NewFiles exists, and is visited every few seconds by bots, that means all new uploads have their thumbnails generated almost on the spot. Thus, we can't slow down that part. We have several long-term tasks to improve this issue, but they will take months to implement. Our only option at the moment is to try and avoid having GWToolset make too many massive images appear on Common's Special:NewFiles in a short period of time. Over 500 of the tiff images were greater than 50 megapixels and as a consequence Commons fails to render any thumbnails Indeed, it seems like some thumbnail generation requests timed out due to the size of these images. There are limits on the image scalers in regards to how long a thumbnailing job can take and these were going over the limit. To make matters worse, the current retry mechanism means that they were being retried 5 times, and thus using 5 times the resources. I would advise against trying to upload those enormous images for now, we should try to focus on a solution for the smaller images. It would be great if the next upload attempt leaves the images that are too large aside. I think the safest option to proceed forward is to lower the appropriate GWToolset throttles in production and then schedule a time for Fae to try the upload process again. By scheduling a specific day and time for the next attempt, we can make sure that engineers and ops have eyes on the servers to watch the load. Then if things go well, we can tweak the throttles back to higher values. http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle…, The throttle documentation doesn't have any unit. I understand that it's "per background job run", but how often do these background jobs run?

I couldn't find configuration values for these throttles on Commons. Dan, can you confirm that Commons is using the default values?

On Mon, Apr 28, 2014 at 11:17 AM, dan entous <d_entous(a)yahoo.com> wrote: GWToolset already has several throttles in place, http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle…, that limit how many background uploads are picked up with each background job run, and how many total GWToolset background jobs can exist in the entire job queue. on the beta cluster the background job seemed to vary in regards to how often it ran for GWToolset varying between 7-30. that seems like enough time for additional images to get processed in-between GWToolset images. wouldn’t it be better to throttle the application/tool that generates thumbnails so that it doesn’t try to produce too many thumbnails at once? with kind regards, dan On Apr 25, 2014, at 20:41 , Gergo Tisza <gtisza(a)wikimedia.org> wrote:

what do you mean by unit? each config key in that section shows a default I want to figure out how many background job runs we end up with per minute/per hour in practice. So I meant units such as X/minute, Y/hour. I know that it's dependent on how the background jobs are configured, but this throttle figures section of the documentation doesn't help figure that out. Makes it hard for anyone to pick a figure, because it's hard to know what the number represents. hopefully, we could instead make it clear to the uploader that if their Guidelines sound like a good idea. If I'm following this logic correctly, though, doesn't that mean that there's also a risk that separate users might "step on each other's toes" in terms of resources, if they happen to be uploading content at the same time? Basically, if a given user sets a threshold which is a fine value for isolated use, isn't the risk that the threshold ends up being too high if more than one GWToolset user is uploading to Commons at the same time? At first I thought that the limit was on the Commons server side, but your remark seems to suggest that this is configured on the uploader's side. CCing Aaron. On Tue, Apr 29, 2014 at 4:19 PM, dan entous <d_entous(a)yahoo.com> wrote: > On Apr 29, 2014, at 15:10 , Gilles Dubuc <gilles(a)wikimedia.org> wrote: > > > Hi Dan, > > > > wouldn’t it be better to throttle the application/tool that generates > thumbnails so that it doesn’t try to produce too many thumbnails at once? > > > > The issue is that there is no application generating thumbnails at a > given rate. Thumbnails are being generated on demand when people view a > thumbnail that doesn't exist. And since Special:NewFiles exists, and is > visited every few seconds by bots, that means all new uploads have their > thumbnails generated almost on the spot. Thus, we can't slow down that > part. We have several long-term tasks to improve this issue, but they will > take months to implement. Our only option at the moment is to try and avoid > having GWToolset make too many massive images appear on Common's > Special:NewFiles in a short period of time. > > > > Over 500 of the tiff images were greater than 50 megapixels and as a > consequence Commons fails to render any thumbnails > > > > Indeed, it seems like some thumbnail generation requests timed out due > to the size of these images. There are limits on the image scalers in > regards to how long a thumbnailing job can take and these were going over > the limit. To make matters worse, the current retry mechanism means that > they were being retried 5 times, and thus using 5 times the resources. I > would advise against trying to upload those enormous images for now, we > should try to focus on a solution for the smaller images. It would be great > if the next upload attempt leaves the images that are too large aside. > > > > I think the safest option to proceed forward is to lower the > appropriate GWToolset throttles in production and then schedule a time for > Fae to try the upload process again. By scheduling a specific day and time > for the next attempt, we can make sure that engineers and ops have eyes on > the servers to watch the load. Then if things go well, we can tweak the > throttles back to higher values. > > > > > http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle… > , > > > > The throttle documentation doesn't have any unit. I understand that > it's "per background job run", but how often do these background jobs run? > > what do you mean by unit? each config key in that section shows a default > > > I couldn't find configuration values for these throttles on Commons. > Dan, can you confirm that Commons is using the default values? > > > throttle config values > ---------------------- > the throttle configuration values are in the extension itself, conhttp:// > git.wikimedia.org/blob/mediawiki%2Fextensions%2FGWToolset.git/d27991ca8168e…, > and can be overridden in the > http://git.wikimedia.org/tree/operations%2Fmediawiki-config.gitwmf-config/C… file in the if ( $wmgUseGWToolset ) { section. > > the config values to most likely change would be > $mediafile_job_throttle_default, which is currently set to 10 and > $mediafile_job_throttle_max, which is currently set to 20. > > at the moment, a user can set this throttle between 1-20. that means that > every time a GWToolset Metadata background job is run between 1-20 > GWToolset Mediafile jobs are added to the queue. we could change those > values, but that would be a pity for people uploading smaller file sizes. > hopefully, we could instead make it clear to the uploader that if their > GWToolset\Config::$mediafile_job_throttle_default = new_value > GWToolset\Config::$mediafile_job_throttle_max = new_value > > > > > > > > > > On Mon, Apr 28, 2014 at 11:17 AM, dan entous <d_entous(a)yahoo.com> > wrote: > > GWToolset already has several throttles in place, > http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle…, > that limit how many background uploads are picked up with each background > job run, and how many total GWToolset background jobs can exist in the > entire job queue. on the beta cluster the background job seemed to vary in > regards to how often it ran for GWToolset varying between 7-30. that seems > like enough time for additional images to get processed in-between > GWToolset images. > > > > wouldn’t it be better to throttle the application/tool that generates > thumbnails so that it doesn’t try to produce too many thumbnails at once? > > > > with kind regards, > > dan > > > > > > > > On Apr 25, 2014, at 20:41 , Gergo Tisza <gtisza(a)wikimedia.org> wrote: > > > > > On Fri, Apr 25, 2014 at 11:13 AM, Fæ <faewik(a)gmail.com> wrote: > > > With no obvious immediate fix/work-around on the table from WMF ops, I > > > have proposed to re-start my uploads for this project with an > > > effective throttle by using 2 threads (this is a setting on the first > > > screen of the GWToolset. In practice, having tried a run of a couple > > > of hundred, this means that the tool is uploading 100MB sized images > > > at a rate of 2 every 5 minutes. This seems to not be causing any > > > issues. > > > > > > The issue was not directly with the uploads; there is no thumbnail > rendering happening on upload, so GWToolset adding lots of large TIFFs > quickly would not cause problems in itself. The upload speed was > problematic because that meant GWToolset saturated pages like > Special:NewFiles, and when somebody looked at such pages, *that* triggered > lots of thumbnail renderings of huge TIFF files at the same time. If > GWToolset is slowed down and lots of miscellaneous files are uploaded > between the TIFFs, those special pages won't be problematic, but something > like a gallery or category of huge TIFF files could still be. > > > _______________________________________________ > > > Glamtools mailing list > > > Glamtools(a)lists.wikimedia.org > > > https://lists.wikimedia.org/mailman/listinfo/glamtools > > > > > >

Up to 16 jobs among all GWT job types can be picked at once (1 runner per Are they picked up at a steady frequency? Basically, if each job treats less images in one run, will it truly space out each image's treatment in time? Or will it have no effect because each GWT job will complete faster, with the next GWT job being picked up right after the previous one on a given server completes? On Tue, Apr 29, 2014 at 5:56 PM, Aaron Schulz <aschulz(a)wikimedia.org>wrote;wrote: > Up to 16 jobs among all GWT job types can be picked at once (1 runner per > > On Tue, Apr 29, 2014 at 8:41 AM, Gilles Dubuc <gilles(a)wikimedia.org>wrote;wrote: > >> what do you mean by unit? each config key in that section shows a >>> default value to the right of it. >>> >> >> I want to figure out how many background job runs we end up with per >> minute/per hour in practice. So I meant units such as X/minute, Y/hour. I >> know that it's dependent on how the background jobs are configured, but >> this throttle figures section of the documentation doesn't help figure that >> out. Makes it hard for anyone to pick a figure, because it's hard to know >> what the number represents. >> >> hopefully, we could instead make it clear to the uploader that if their >>> file sizes exceed Xmb then they should set that throttle to 1 and make sure >>> the engineers and ops are notified in advance about the upload. >>> >> >> Guidelines sound like a good idea. If I'm following this logic >> correctly, though, doesn't that mean that there's also a risk that separate >> users might "step on each other's toes" in terms of resources, if they >> happen to be uploading content at the same time? Basically, if a given user >> sets a threshold which is a fine value for isolated use, isn't the risk >> that the threshold ends up being too high if more than one GWToolset user >> is uploading to Commons at the same time? At first I thought that the limit >> was on the Commons server side, but your remark seems to suggest that this >> is configured on the uploader's side. >> >> >>> job run frequency >>> ----------------- >>> how often are the background jobs run? >>> is there a limit on how many GWToolset Mediafile background jobs are >>> picked up at once? >>> >>> i don’t know. aaron schultz would be the best person to ask. on the >>> beta cluster it seemed to vary between 7-30 minutes, but that may have been >>> because of testing or other activity on that server. >> >> >> CCing Aaron. >> >> >> On Tue, Apr 29, 2014 at 4:19 PM, dan entous <d_entous(a)yahoo.com> wrote: >> >>> On Apr 29, 2014, at 15:10 , Gilles Dubuc <gilles(a)wikimedia.org> wrote: >>> >>> > Hi Dan, >>> > >>> > wouldn’t it be better to throttle the application/tool that generates >>> thumbnails so that it doesn’t try to produce too many thumbnails at once? >>> > >>> > The issue is that there is no application generating thumbnails at a >>> given rate. Thumbnails are being generated on demand when people view a >>> thumbnail that doesn't exist. And since Special:NewFiles exists, and is >>> visited every few seconds by bots, that means all new uploads have their >>> thumbnails generated almost on the spot. Thus, we can't slow down that >>> part. We have several long-term tasks to improve this issue, but they will >>> take months to implement. Our only option at the moment is to try and avoid >>> having GWToolset make too many massive images appear on Common's >>> Special:NewFiles in a short period of time. >>> > >>> > Over 500 of the tiff images were greater than 50 megapixels and as a >>> consequence Commons fails to render any thumbnails >>> > >>> > Indeed, it seems like some thumbnail generation requests timed out >>> due to the size of these images. There are limits on the image scalers in >>> regards to how long a thumbnailing job can take and these were going over >>> the limit. To make matters worse, the current retry mechanism means that >>> they were being retried 5 times, and thus using 5 times the resources. I >>> would advise against trying to upload those enormous images for now, we >>> should try to focus on a solution for the smaller images. It would be great >>> if the next upload attempt leaves the images that are too large aside. >>> > >>> > I think the safest option to proceed forward is to lower the >>> appropriate GWToolset throttles in production and then schedule a time for >>> Fae to try the upload process again. By scheduling a specific day and time >>> for the next attempt, we can make sure that engineers and ops have eyes on >>> the servers to watch the load. Then if things go well, we can tweak the >>> throttles back to higher values. >>> > >>> > >>> http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle… >>> , >>> > >>> > The throttle documentation doesn't have any unit. I understand that >>> it's "per background job run", but how often do these background jobs run? >>> >>> what do you mean by unit? each config key in that section shows a >>> default value to the right of it. >>> >>> >>> > I couldn't find configuration values for these throttles on Commons. >>> Dan, can you confirm that Commons is using the default values? >>> >>> >>> throttle config values >>> ---------------------- >>> the throttle configuration values are in the extension itself, >>> conhttp:// >>> git.wikimedia.org/blob/mediawiki%2Fextensions%2FGWToolset.git/d27991ca8168e…, >>> and can be overridden in the >>> http://git.wikimedia.org/tree/operations%2Fmediawiki-config.gitwmf-config/C… file in the if ( $wmgUseGWToolset ) { section. >>> >>> the config values to most likely change would be >>> $mediafile_job_throttle_default, which is currently set to 10 and >>> $mediafile_job_throttle_max, which is currently set to 20. >>> >>> at the moment, a user can set this throttle between 1-20. that means >>> that every time a GWToolset Metadata background job is run between 1-20 >>> GWToolset Mediafile jobs are added to the queue. we could change those >>> values, but that would be a pity for people uploading smaller file sizes. >>> hopefully, we could instead make it clear to the uploader that if their >>> file sizes exceed Xmb then they should set that throttle to 1 and make sure >>> the engineers and ops are notified in advance about the upload. >>> >>> GWToolset\Config::$mediafile_job_throttle_default = new_value >>> GWToolset\Config::$mediafile_job_throttle_max = new_value >>> >>> >>> job run frequency >>> ----------------- >>> how often are the background jobs run? >>> is there a limit on how many GWToolset Mediafile background jobs are >>> picked up at once? >>> >>> i don’t know. aaron schultz would be the best person to ask. on the >>> beta cluster it seemed to vary between 7-30 minutes, but that may have been >>> because of testing or other activity on that server. >>> >>> >>> > >>> > >>> > On Mon, Apr 28, 2014 at 11:17 AM, dan entous <d_entous(a)yahoo.com> >>> wrote: >>> > GWToolset already has several throttles in place, >>> http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle…, >>> that limit how many background uploads are picked up with each background >>> job run, and how many total GWToolset background jobs can exist in the >>> entire job queue. on the beta cluster the background job seemed to vary in >>> regards to how often it ran for GWToolset varying between 7-30. that seems >>> like enough time for additional images to get processed in-between >>> GWToolset images. >>> > >>> > wouldn’t it be better to throttle the application/tool that generates >>> thumbnails so that it doesn’t try to produce too many thumbnails at once? >>> > >>> > with kind regards, >>> > dan >>> > >>> > >>> > >>> > On Apr 25, 2014, at 20:41 , Gergo Tisza <gtisza(a)wikimedia.org> wrote: >>> > >>> > > On Fri, Apr 25, 2014 at 11:13 AM, Fæ <faewik(a)gmail.com> wrote: >>> > > With no obvious immediate fix/work-around on the table from WMF >>> ops, I >>> > > have proposed to re-start my uploads for this project with an >>> > > effective throttle by using 2 threads (this is a setting on the >>> first >>> > > screen of the GWToolset. In practice, having tried a run of a couple >>> > > of hundred, this means that the tool is uploading 100MB sized images >>> > > at a rate of 2 every 5 minutes. This seems to not be causing any >>> > > issues. >>> > > >>> > > The issue was not directly with the uploads; there is no thumbnail >>> rendering happening on upload, so GWToolset adding lots of large TIFFs >>> quickly would not cause problems in itself. The upload speed was >>> problematic because that meant GWToolset saturated pages like >>> Special:NewFiles, and when somebody looked at such pages, *that* triggered >>> lots of thumbnail renderings of huge TIFF files at the same time. If >>> GWToolset is slowed down and lots of miscellaneous files are uploaded >>> between the TIFFs, those special pages won't be problematic, but something >>> like a gallery or category of huge TIFF files could still be. >>> > > _______________________________________________ >>> > > Glamtools mailing list >>> > > Glamtools(a)lists.wikimedia.org >>> > > https://lists.wikimedia.org/mailman/listinfo/glamtools >>> > >>> > >>> >>> >> > > > -- > -Aaron S >

is to run an series of API queries checking whether everything on my list of images exists yet or not, itself probably creating more stress on the servers.

That would mean setting $mediafile_job_throttle_max to 10 and setting $metadata_job_delay to 20 minutes

From the Ops & Multimedia mailing lists:

was investigated and NYPL maps were found to be the cause of the outage.

Uhh... let's give this another shot in the morning. I went through last day's upload logs; on average there are ~600 uploads an hour, the peak was 1900, the negative peak around 240. (The numbers are at http://pastebin.com/raw.php?i=wmBRJm1G in case anybody finds them useful.) So that's around 4 files per minute in worst case. If we are aiming for no more than 10% of Special:NewFiles to be taken up by GWToolset, that means 5 uploads per run of the control job (10% of the 50 slots at Special:NewFiles) - the upload jobs can't really be throttled, so we must make sure they come in small enough chunks, no matter how much delay there is between the chunks). Also, we want to keep below 10% of the total Commons upload rate - that means 24 images per hour, which is roughly five runs of the control job per hour. So the correct config is GWToolset\Config::$mediafile_job_throttle_default = 5; $wgJobBackoffThrottling['gwtoolsetUploadMetadataJob'] = 5 / 3600; I'm leaving the max throttle at 20 so that people who are uploading small, non-TIFF images can get a somewhat higher speed.

From the Ops & Multimedia mailing lists: We just had a brief imagescaler outage today at approx. 11:20 UTC that was investigated and NYPL maps were found to be the cause of the outage. As Gergo's unanswered recent message in this thread suggested, we're actively working on a number of changes to stabilize GWToolset and improve image scaler performance in order to avoid such outages. I assumed that since everyone involved is participating in this thread, that you were waiting for these changes to happen before restarting the GWToolset job that caused the previous outage a couple of weeks ago, or that you would warn us when that job would be run again. There seems to be a communication issue here. By running this job, you've taken down thumbnail generation on Commons (and all WMF wikis) and we were lucky that someone from Ops was around, noticed it and reacted quickly. This could have been easily avoided with better coordination, by at least scheduling a time to run your next attempt, with people from Ops watching servers at the time the job is run. Please make sure that this happens for the next batch of NYPL maps/massive files that you plan to upload with GWToolset. All it takes is scheduling a day and time for the next upload attempt. Gergo and I will keep replying to this thread to notify everyone when our related code changes are merged. On Wed, May 7, 2014 at 10:26 PM, Gergo Tisza <gtisza(a)wikimedia.org> wrote: Uhh... let's give this another shot in the morning. I went through last day's upload logs; on average there are ~600 uploads an hour, the peak was 1900, the negative peak around 240. (The numbers are at http://pastebin.com/raw.php?i=wmBRJm1G in case anybody finds them useful.) So that's around 4 files per minute in worst case. If we are aiming for no more than 10% of Special:NewFiles to be taken up by GWToolset, that means 5 uploads per run of the control job (10% of the 50 slots at Special:NewFiles) - the upload jobs can't really be throttled, so we must make sure they come in small enough chunks, no matter how much delay there is between the chunks). Also, we want to keep below 10% of the total Commons upload rate - that means 24 images per hour, which is roughly five runs of the control job per hour. So the correct config is GWToolset\Config::$mediafile_job_throttle_default = 5; $wgJobBackoffThrottling['gwtoolsetUploadMetadataJob'] = 5 / 3600; I'm leaving the max throttle at 20 so that people who are uploading small, non-TIFF images can get a somewhat higher speed. _______________________________________________ Glamtools mailing list Glamtools(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools

first, i suggest that we put off all large image uploads, > 10mb ( unless we have a concrete value that would work ), until we resolve the thumbnail issue. during the zürich hackathon i spoke with aaron schultz, faiden liambotis, and brion vibber regarding approaches to dealing with this issue. in summary, the idea aaron came up with is to create initial thumbnails on download of the original mediafile to the wiki. this should block the appearance of the title on the new files page and anywhere else until the thumbnails and title creation/edit have completed. aaron thought, and faidon and i agree, that further throttling of gwtoolset will not help resolve the issue. i am currently looking into implementing this approach. with kind regards, dan

Just a quick note that technical discussion about this is mostly happening on the multimedia list (recently, the "Brief image scalers outage, Mon Apr 21 03:12 UTC" and "Limiting bandwidth when reading from swift" threads). _______________________________________________ Glamtools mailing list Glamtools(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools

How large is that batch? We're still working on technical changes, nothing has been merged since the last outage.

They might be good as a speed test to blast through and to create the thumbnails. I don't know if this means overwriting the current files or if there is some trick to forcing the thumbnail recreation. If these work well, then I'll stop creating PNGs for the TIFF drawings I'm uploading (there are going to be something like another 20,000+ of these to come as part of HABS uploading).

or if there is some trick to forcing the thumbnail recreation. If these work well, then I'll stop creating PNGs for the TIFF drawings

Hi all, with bug 65691 <https://bugzilla.wikimedia.org/show_bug.cgi?id=65691> fixed (the last patch was deployed today), now might be a good time to test large TIFF uploads again (the patch is limited to TIFF files for now). I was thinking of the following schedule: - wait until Monday (no breaking the site on the weekends) - launch an upload with large TIFF files (preferably the same one that caused issues earlier, ie. Fae's NYPL maps project) - make sure that the images are initially not categorized to avoid someone triggering 200 new thumbnail requests in parallel (GWToolset could add an emtpy template instead, and that template can be replaced with the category later). - initially use the minimum speed allowed by GWToolset (a single thread), to make sure Special:NewFiles and co. will also not be the source of many concurrent requests. - after a bunch of images have been uploaded, generate a gallery with 10 thumbnails and monitor imagescaler load and Swift traffic in the process. Repeat with 20, 50 etc until we are satisfied that the scalers are resilient to many concurrent requests for large files. - if all works out, the upload project can continue with normal speed (20 threads or whatever), and we can also relax throttle limits on GWToolset a bit. Does this sound reasonable? Fae, are you interested in doing this?

files on Commons is about 5 files per minute (very unscientific number, I just looked at the recent changes page), so a full-speed GWToolset run would