On Apr 20, 2016 10:45 PM, "Brion Vibber" bvibber@wikimedia.org wrote:
Over in TimedMediaHandler extension, we've had a number of cases where old code did things that were convenient in terms of squishing read-write operations into data getters, that got removed due to problems with long running transactions or needing to refactor things to support future-facing multi-DC work where we want requests to be able to more reliably distinguish between read-only and read-write. And we sometimes want to put some of those clever hacks back and add more. ;)
For instance in https://gerrit.wikimedia.org/r/284368 we'd like to remove transcode derivative files of types/resolutions that have been disabled automatically when we come across them. But I'm a bit unsure it's safe to do so.
Note that we could fire off a job queue background task to do the actual removal... But is it also safe to do that on a read-only request?
https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacen...
seems to indicate job queueing will be safe, but would like to confirm that. :)
I think this is the preferred method. My understanding is that the jobs will get shipped to the primary DC job queue.
Similarly in https://gerrit.wikimedia.org/r/#/c/284269/ we may wish to trigger missing transcodes to run on demand, similarly. The actual re encoding happens in a background job, but we have to fire it off, and we have to record that we fired it off so we don't duplicate it...
(This would require a second queue to do the high-priority state table update and queue the actual transcoding job; we can't put them in one
queue
because a backup of transcode jobs would prevent the high priority job
from
running in a timely fashion.)
The job queue can do deduplication, although you would have to check if that is active while the job is running and not only while queued. Might help?
A best practices document on future-proofing for multi DC would be pretty awesome! Maybe factor out some of the stuff from the RfC into a nice dev doc page...
-- brion _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l