I too use bash files (see https://github.com/PersianWikipedia/fawikibot/tree/master/HujiBot/grid/jobs) and am planning to move them from grid to k8s in the next few weeks. I have found the process well-documented and am hoping for the transition to be easy, like Amir said.
Since at least two people are using this approach, once I am done with my transition I might go to Wikitech and add some helpful hints for other users.
On Thu, Mar 10, 2022 at 5:03 AM Amir Sarabadani ladsgroup@gmail.com wrote:
I recently migrated most of pywikibot jobs from grid engine to k8s and to my surprise, it was actually quite easy. I have lots of tasks (been running bots since 2008). So much that it showed redaction in the total number of jobs in SGE.
One thing that helped me that I collapsed everything into a bash file, like hourly.sh and made that a job (it's better to have one pod doing a batch of work as overhead of creating a pod is bigger than a SGE job)
Also, make sure your bash file is readable by others (see the doc, chmod ug+x hourlysh), that would have saved me a bit of time.
Thank you for doing this. I use only the new fancy infrastructure from now on ^^
On Fri, Feb 18, 2022 at 8:28 PM Arturo Borrero Gonzalez < aborrero@wikimedia.org> wrote:
On 2/16/22 17:34, Russell Blau wrote:
Also, it is not possible to load Pywikibot in the tf-python39 runtime
because a required module (requests, fromhttps://python-requests.org) is not available. What is the process for requesting (no pun intended) that this (or any other resource) be added to the image?
See some documentation here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_...
I just created it, and may need some polishing, but it should work!
We will review pywikibot specific workflows and documents soon.
regards.
Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- Amir (he/him)
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/