Hi opsen,
I'm currently working on this changeset: https://gerrit.wikimedia.org/r/#/c/157157/ whose goal is to pre-render commonly used thumbnail sizes at upload time, in order to avoid the delay experienced by users who are the first to view a particular image at a given size (particularly in media viewer).
So far I've implemented it as a job (in the job queue sense of the term). Which implies that the server(s) picking up this job type would need to have the whole stack of image processing software installed on them. The idea being that we could the resources for this prerendering separately from the existing pool of on-demand image scalers. Does this approach make sense from an Ops perspective? Basically having one or more servers with the same software as image scalers installed on them, configured as job runners for that particular job type.
The alternative is that the job would cURL the thumbnail urls to hit the image scalers. I'm not sure that this is a desirable network path, it might not be the most future-proof thing to expect job runners to be able to hit our public-facing URLs. Not to mention this makes it a very WMF-specific solution, whereas the job type approach is more generic. Maybe there's a better way for a job runner to make the image scalers do something, though. That alternative approach of hitting thumbnail urls would imply the job running on the regular pool of job runners. And it would mean that we probably wouldn't be able to tell apart the resource usage of the prerendering compared to the regular on-demand thumbnailing that's happening at the moment.
hi Gilles,
On Mon, Sep 1, 2014 at 9:02 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi opsen,
I'm currently working on this changeset: https://gerrit.wikimedia.org/r/#/c/157157/ whose goal is to pre-render commonly used thumbnail sizes at upload time, in order to avoid the delay experienced by users who are the first to view a particular image at a given size (particularly in media viewer).
So far I've implemented it as a job (in the job queue sense of the term). Which implies that the server(s) picking up this job type would need to have the whole stack of image processing software installed on them. The idea being that we could the resources for this prerendering separately from the existing pool of on-demand image scalers. Does this approach make sense from an Ops perspective? Basically having one or more servers with the same software as image scalers installed on them, configured as job runners for that particular job type.
The alternative is that the job would cURL the thumbnail urls to hit the image scalers. I'm not sure that this is a desirable network path, it might not be the most future-proof thing to expect job runners to be able to hit our public-facing URLs. Not to mention this makes it a very WMF-specific solution, whereas the job type approach is more generic. Maybe there's a better way for a job runner to make the image scalers do something, though. That alternative approach of hitting thumbnail urls would imply the job running on the regular pool of job runners. And it would mean that we probably wouldn't be able to tell apart the resource usage of the prerendering compared to the regular on-demand thumbnailing that's happening at the moment.
Your analysis (job vs image scalers) is correct, I think it makes more sense to keep the two isolated. Perhaps the job can by default generate the thumbs and optionally hit an URL so it won't be WMF-specific (or the other way around).
Anyways the easiest I think would be to just hit the public URL from the job and let varnish/etc do the rest, on the basis that URL will be hit by the user anyway (that's the case I'm assuming, correct?). If you want to future-proof it further you could hit the internal upload endpoint (upload.svc.eqiad.wmnet) with the right Host: header.
re: bawolff's concerns regarding many uploads I think we have the ability to limit concurrent job types
BTW what sizes are you planning to start out with?
HTH, filippo
Hi Filippo,
I've updated the changeset to support both cURL hits and rendering inside the job. Sadly it seems like cURL hits also need to be a job, because I tried sending a fire-and-forget HEAD request (with a Connection: Close header) and that wasn't enough to trigger the thumbnail generation. Thus the cURL request has to wait for completion, and a result is inside a job in order to avoid the upload script hanging while the thumbs are being generated. The upside is that the job that sends the cURL requests could be rate-limited.
BTW what sizes are you planning to start out with?
Mostly the sizes used by Media Viewer and probably the most common thumbnail sizes found in articles as well. I'm open to suggestions regarding how many different sizes should be pregenerated and which ones would be reasonable to pick. I guess maybe it's best to start with a minimal set and then expand it later if the resource usage is reasonable?
On Mon, Sep 1, 2014 at 1:17 PM, Filippo Giunchedi fgiunchedi@wikimedia.org wrote:
hi Gilles,
On Mon, Sep 1, 2014 at 9:02 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi opsen,
I'm currently working on this changeset: https://gerrit.wikimedia.org/r/#/c/157157/ whose goal is to pre-render commonly used thumbnail sizes at upload time, in order to avoid the delay experienced by users who are the first to view a particular image at a given size (particularly in media viewer).
So far I've implemented it as a job (in the job queue sense of the term). Which implies that the server(s) picking up this job type would need to have the whole stack of image processing software installed on them. The idea being that we could the resources for this prerendering separately from the existing pool of on-demand image scalers. Does this approach make sense from an Ops perspective? Basically having one or more servers with the same software as image scalers installed on them, configured as job runners for that particular job type.
The alternative is that the job would cURL the thumbnail urls to hit the image scalers. I'm not sure that this is a desirable network path, it might not be the most future-proof thing to expect job runners to be able to hit our public-facing URLs. Not to mention this makes it a very WMF-specific solution, whereas the job type approach is more generic. Maybe there's a better way for a job runner to make the image scalers do something, though. That alternative approach of hitting thumbnail urls would imply the job running on the regular pool of job runners. And it would mean that we probably wouldn't be able to tell apart the resource usage of the prerendering compared to the regular on-demand thumbnailing that's happening at the moment.
Your analysis (job vs image scalers) is correct, I think it makes more sense to keep the two isolated. Perhaps the job can by default generate the thumbs and optionally hit an URL so it won't be WMF-specific (or the other way around).
Anyways the easiest I think would be to just hit the public URL from the job and let varnish/etc do the rest, on the basis that URL will be hit by the user anyway (that's the case I'm assuming, correct?). If you want to future-proof it further you could hit the internal upload endpoint (upload.svc.eqiad.wmnet) with the right Host: header.
re: bawolff's concerns regarding many uploads I think we have the ability to limit concurrent job types
BTW what sizes are you planning to start out with?
HTH, filippo
hey Gilles,
On Tue, Sep 2, 2014 at 5:25 PM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi Filippo,
I've updated the changeset to support both cURL hits and rendering inside the job. Sadly it seems like cURL hits also need to be a job, because I tried sending a fire-and-forget HEAD request (with a Connection: Close header) and that wasn't enough to trigger the thumbnail generation. Thus the cURL request has to wait for completion, and a result is inside a job in order to avoid the upload script hanging while the thumbs are being generated. The upside is that the job that sends the cURL requests could be rate-limited.
good to know! FWIW I think we are better off with the job anyways
BTW what sizes are you planning to start out with?
Mostly the sizes used by Media Viewer and probably the most common
thumbnail sizes found in articles as well. I'm open to suggestions regarding how many different sizes should be pregenerated and which ones would be reasonable to pick. I guess maybe it's best to start with a minimal set and then expand it later if the resource usage is reasonable?
+1 to the minimal set, perhaps the top 2/3 most common resolutions for devices using MV? let us know when you plan to deploy it so we can keep an eye on swift.
thanks! filippo
On 09/01/2014 04:17 AM, Filippo Giunchedi wrote:
Anyways the easiest I think would be to just hit the public URL from the job and let varnish/etc do the rest, on the basis that URL will be hit by the user anyway (that's the case I'm assuming, correct?). If you want to future-proof it further you could hit the internal upload endpoint (upload.svc.eqiad.wmnet) with the right Host: header.
+1 to hitting the regular image scaling end points rather than doing the scaling right in the job runner. This lets us isolate image scalers from other tasks, which is an advantage from a security and monitoring perspective. It should also keep the actual job fairly simple.
Arguably this approach is also more flexible and future-proof, as it doesn't make strong assumptions about how image scaling is implemented behind the scenes.
Gabriel
multimedia@lists.wikimedia.org