[Labs-l] Processing dumps with Wikimedia Utilities

[Emilio J. Rodríguez-Posada]

Yes sorry, it was me.


2014-05-18 18:55 GMT+02:00 Damian Zaremba <damian at damianzaremba.co.uk>:

>  Is this the currentevents tool? It looks like a huge export is currently
> running on tools-login:
>
>
> cp/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history1.xml.bz2/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history2.xml.bz2/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history3.xml.bz2/public/dumps/public/eswiki/20140509/eswiki-20140509-pages-meta-history4.xml.bz2
>
> Thanks to the cpu this is chewing on the box is rather laggy with a load
> of ~5. Please don't run this on tools-login, use the dev box or an
> execution node.
>
> - Damian
>
>
> On 18/05/2014 17:14, Emilio J. Rodríguez-Posada wrote:
>
>  Well, the dump is reachable from the machine. But my script do this:
>
> fp = subprocess.Popen('7za e -bd -so %s 2>/dev/null' % dumpfilename,
> shell=True, stdout=subprocess.PIPE, bufsize=65535)
>
>  Is this correct in a grid?
>
>
> 2014-05-18 17:57 GMT+02:00 Emilio J. Rodríguez-Posada <emijrp at gmail.com>:
>
>>  Yeah... I have done that, but now a new problem. I can't read the dump
>> in the destination machine.
>>
>>  Where do I have to copy the dump?
>>
>>
>>  2014-05-18 17:21 GMT+02:00 Jeremy Baron <jeremy at tuxmachine.com>:
>>
>>>   On May 18, 2014 11:16 AM, "Emilio J. Rodríguez-Posada" <
>>> emijrp at gmail.com> wrote:
>>> > I have created the virtualenv for Python3, installed
>>> mediawiki-utilities, etc. I can launch my script in that virtualenv and
>>> works fine, but when I do 'jsub', the destination machine obviously doesn't
>>> have that module:
>>>
>>> Sounds like you need to use the activate script for your virtualenv. You
>>> could do that in a wrapper shell script.
>>>
>>>
>>> http://virtualenv.readthedocs.org/en/latest/virtualenv.html#activate-script
>>>
>>> -Jeremy
>>>
>>>  _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>>
>>
>
>
> _______________________________________________
> Labs-l mailing listLabs-l at lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20140518/998a06bb/attachment.html>