Hi everybody,As announced previously, I am cutting off access to mwmaint servers now. Your homedirs have been copied over one last time.Replacements are:
- deployment.eqiad.wmnet
- MediaWiki Periodic Jobs: https://wikitech.wikimedia.org/wiki/Mw-cron_jobs
- MediaWiki Maintenance Scripts: https://wikitech.wikimedia.org/wiki/Maintenance_scripts
- ldap-maint1001.eqiad.wmnet
Please do not hesitate to reach out to me or a member of the SRE ServiceOps team, in this email thread or on task, if you have any question, or use cases you feel have not been properly replicated.Thank you,On Wed, Jun 25, 2025 at 4:16 PM Clément Goubert <cgoubert@wikimedia.org> wrote:--Hi everybody,As MediaWiki periodic jobs (https://wikitech.wikimedia.org/wiki/Mw-cron_jobs) and all maintenance scripts use cases have been migrated off the maintenance servers, we are going to start the process of decommissioning them.The unsupported use cases for mwscript-k8s can be worked around by using the old mwscript wrapper from the deployment server (as described in https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Not_yet_supported).We have copied your home directories from mwmaint{1002,2002} to deploy1003 in the /srv/homedirs/mwmaint{1002,2002} directories, so you can either copy these files to your homedir there, or clean them up if they are not useful anymore (please clean them up if at all possible! :D).We will soon proceed with limiting ssh access to members of the ops group as a soft decommission, then we will move on to actually decommissioning the servers.You can follow this process in https://phabricator.wikimedia.org/T397017Thank you,Clément 'claime' Goubert (they/them)Senior SRE - Service OpsWikimedia FoundationOn Thu, Jun 5, 2025 at 5:55 PM Scott French <swfrench@wikimedia.org> wrote:Hi all -Following up on Reuven's message on Monday, 5th of May.As of today, maintenance scripts can no longer run on the mwmaint hosts, in advance of their decommissioning. This follows the earlier "hard" deprecation that began on the 12th of May.While the majority of maintenance script use cases have moved to Kubernetes, some are not yet supported (e.g., sql.php [0]) or involve additional complexity in that environment (e.g., large output files [1]).For these cases, as Reuven noted in his message on the 5th of May, users are directed to the active deployment host (i.e., available at deployment.eqiad.wmnet) to run their scripts on an interim basis.This interim solution will be phased out as support lands for the remaining use cases.Many thanks,Scott FrenchService Ops SRE_______________________________________________On Tue, May 6, 2025 at 1:40 PM Brooke Vibber <bvibber@wikimedia.org> wrote:_______________________________________________On Tue, May 6, 2025 at 1:17 PM Reuven Lazarus <rlazarus@wikimedia.org> wrote:Foreachwiki and friends are available in the Kubernetes container now, with a flag on the way to make it easy to invoke them via mwscript-k8s. (https://phabricator.wikimedia.org/T378479) We've also resolved the resource-consumption issues with the shell loop approach, so that option is viable too.Woohoo! Thanks! :D(That is, simultaneously launching that number of jobs will no longer overload the Kubernetes infrastructure itself; of course it's still possible to use up the available compute resources, and it's also possible -- as ever -- for aggressive maintenance scripts to overload shared resources like the databases. But it's back on the table; consider it remediated from the temporary guidance "don't do this, it's dangerous" to the indefinite guidance "this is powerful, do it cautiously.")*nods seriously*Interactive scripts like eval.php can be used with mwscript-k8s in --attach mode (which forwards stdin and stdout to your terminal). (https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Input_on_stdin) Separately, note there's also mw-debug-repl to get a shell in a mw-debug pod, rather than a mw-script one. (https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#MediaWiki_REPL)*success kid meme* yay! Eval lives. :DThat said, mwscript-k8s can't support {my,}sql.php specifically, because we don't have a mysql client in the MW image and shouldn't add one. For now, you can run `sql` from the deploy hosts instead of the mwmaint hosts. The long-term replacement probably won't be a maintenance script at all but a dbconfig-aware wrapper around a local mysql client. (https://phabricator.wikimedia.org/T375910)Awesome, that works fine for my needs.Thanks!-- brooke
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
--Clément 'claime' Goubert (they/them)Senior SREWikimedia Foundation