[Labs-l] Simultaneous job limits?

[John]

Actually python is fairly easy to get muti-threaded, and with a little more
work you can even get muti-processing working. All you really need is a
master worker that monitors the number of workers and limits the spawn of
additional workers. I use a 6 worker multi-process when doing my database
dump scans, each worker processes 500 pages and then dies. When the master
process sees a sub-process die it grabs another 500 pages from the work
queue and launches a replacement process. With a little monitoring and
tweaking you can adjust both the amount of work and number of workers to
work withing a reasonable resource limit. This way the work can still get
done fairly quickly and you are only using one job queue slot. (Each slot
isnt limited to a single process, it just calculates total resource usage
and compares that to the limits set, and if a job exceeds the max set it
kills it.) Depending on what your bounding factor is either mut-threading
or muti-processing should work for you without too much headache in the
conversion.

On Mon, Mar 30, 2015 at 12:37 AM, Anthony Di Franco <di.franco at gmail.com>
wrote:

> The intent of lots of jobs is to have lots of redis queue workers making
> network requests in parallel, so it would probably be significant work with
> python's threading or twisted or the like to consolidate multiple requests
> into a single OS process, but I'll start looking into it, and the resource
> increase  seems to have us covered in the meantime. Thanks!
>
> On Sun, Mar 29, 2015 at 7:49 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>
>> Part of the reason was that they asked for trusty and we only had one
>> trusty node working for exec hosts, apparently. I have just added 5
>> more and that should help.
>>
>> However, +1 to what Coren said. Can you simplify your code to combine
>> the tasks together?
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150330/8c748b6c/attachment.html>