On Sun, Aug 12, 2018 at 4:48 AM, Brion Vibber bvibber@wikimedia.org wrote:
While working on some maintenance scripts for TimedMediaHandler I've been trying to make it easier to do scripts that use multiple parallel processes to run through a large input set faster.
My proposal is a ForkableMaintenance class, with an underlying QueueingForkController which is a refactoring of the OrderedStreamingForkController used by (at least) some CirrusSearch maintenance scripts.
For what it's worth, when I saw ForkableMaintenance I thought of forking an open-source project, not Unix fork(). Something like ParallelMaintenance or ParallelizableMaintenance would better suggest the desired meaning for me.