I recently migrated most of pywikibot jobs from grid engine to k8s and to my surprise, it was actually quite easy. I have lots of tasks (been running bots since 2008). So much that it showed redaction in the total number of jobs in SGE.

One thing that helped me that I collapsed everything into a bash file, like hourly.sh and made that a job (it's better to have one pod doing a batch of work as overhead of creating a pod is bigger than a SGE job)

Also, make sure your bash file is readable by others (see the doc, chmod ug+x hourlysh), that would have saved me a bit of time.

Thank you for doing this. I use only the new fancy infrastructure from now on ^^

On Fri, Feb 18, 2022 at 8:28 PM Arturo Borrero Gonzalez <aborrero@wikimedia.org> wrote:

On 2/16/22 17:34, Russell Blau wrote:
> Also, it is not possible to load Pywikibot in the tf-python39 runtime because a required module (requests, fromhttps://python-requests.org) is not available. What is the process for requesting (no pun intended) that this (or any other resource) be added to the image?

See some documentation here:

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_jobs

I just created it, and may need some polishing, but it should work!

We will review pywikibot specific workflows and documents soon.


regards.
--
Arturo Borrero Gonzalez
Site Reliability Engineer
Wikimedia Cloud Services
Wikimedia Foundation
_______________________________________________
Cloud mailing list -- cloud@lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

--

Amir (he/him)

_______________________________________________
Cloud mailing list -- cloud@lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/