- Cloud - lists.wikimedia.org

Wikimedia remote hackathon on May 22/23!
by Birgit Müller 31 Mar '21

31 Mar '21

Hello All, Many of us were hopeful that we would be able to organise an onsite hackathon and meet in person in 2021. While this is sadly not the case, we still wanted to offer the opportunity for the technical community to get together virtually, work together on various projects, and discuss new ideas. This is why we’re happy to invite you all to join the ‘21 Wikimedia remote hackathon on May 22/23 <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021> - save the date, and bring your projects & ideas! Building on last year’s edition, this event is organized in a light mode, offering a lot of space for spontaneity, experimentation, and all kinds of projects. The ‘21 Wikimedia remote hackathon is built by and for its participants, and coordinated by a team of volunteers and staff - Amir, Birgit, Joaquin, Léa, Mohammed, Neslihan, Pavritha. On this page <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021> you will find all the relevant information. The planned framework consists of one main track of sessions that participants can follow, open rooms for informal discussions, workshops and social events, and to work together on projects. We will send out more information on how to schedule a session in the program soon! You can also add yourself to the participants list <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Participants>, and mention if you would like to help with tasks such as facilitation, or welcoming newcomers. We hope that this event can be a moment of fun, reconnecting with long-time-no-see Wikimedians as well as onboarding new people into the technical community. If you have any questions, feel free to reach out to us on the communication channels of the event <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Discussions>. Hope to see you there! The ‘21 remote hack coordination team -- Birgit Müller (she/her) Director of Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[Cloud-announce] PAWS ready to use the new DB replicas
by Joaquin Oltra Hernandez 30 Mar '21

30 Mar '21

TLDR: - PAWS can now connect to the new replicas, see News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> for more info. - Report issues here: T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> Hi! PAWS is now capable of connecting and using the new replicas. Here are some resources you can check: - News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> - Accessing the new replicas, changes from the previous cluster <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20the%20new…> - Using Wikireplicas from PAWS with Python <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20Wikirepli…> In summary, due to issues with mysql-proxy and the new architecture, connecting to the replicas will be more in line with the Toolforge approach. There is a credentials file in $HOME/.my.cnf that you can use when connecting, instead of the environment variables. For the host name, you can use the same ones you would use when connecting from Toolforge (" {wiki}.{analytics,web}.db.svc.wikimedia.cloud"). To update a notebook, here is an example of the couple of changes when connecting: - import os import pymysql conn = pymysql.connect( - host = os.environ['MYSQL_HOST'], + host = "eswiki.analytics.db.svc.wikimedia.cloud", - user = os.environ['MYSQL_USERNAME'], - password = os.environ['MYSQL_PASSWORD'], + read_default_file = ".my.cnf", database = "eswiki_p" ) Note you have to connect to the host name of the DB you are going to query against. Existing notebooks remain readable with the output cached, and we are working on updating the documentation. In two weeks -April 15- the old cluster will migrate the old cluster to utilize new replication hosts, at which point replication may stop and running PAWS notebooks connecting to the old cluster may get stale results. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster, and running notebooks connecting to MYSQL_HOST will not work and will need updating the credentials and DB host name. If you find any issues or problems or need help, please reach out via IRC, mailing list, or in the phabricator task T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] Cloud Services Survey 2020 results published
by Joaquin Oltra Hernandez 26 Mar '21

26 Mar '21

Hi! The 2020 survey collected feedback from Toolforge project members and Cloud VPS project administrators on how the services offered can be improved to help their development and maintenance needs. It was active between 2021-01-14 and 2021-02-05, and had 137 participants. The summarized results have been published on meta: https://meta.wikimedia.org/wiki/Research:Cloud_Services_Annual_Survey/2020 Thanks to everyone who participated and provided input and comments. They are very useful and instrumental in shaping the future and improvements for the cloud services. Have a nice day! -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Install emacs 25 on bastion hosts?
by Roy Smith 22 Mar '21

22 Mar '21

Not sure if this is the right place to ask, but would it be possible to install emacs-25 on the bastion hosts? Apparently, the python mode in emacs-25 knows about the new python-3.5 asyncio keywords (async def, etc), but emacs-24 doesn't. One of these looks like the right thing: emacs25-lucid/oldstable,oldstable 25.1+1-4+deb9u1 amd64 GNU Emacs editor (with Lucid GUI support) emacs25-nox/oldstable,oldstable 25.1+1-4+deb9u1 amd64 GNU Emacs editor (without GUI support)

3 3

[Cloud-announce] The future of storage and /srv on cloud-vps
by Andrew Bogott 17 Mar '21

17 Mar '21

With Cinder storage[0] now available and well-tested on cloud-vps, it's time to move forward to widespread adoption. As of today, use of lvm (e.g. via role::labs::lvm::srv) is deprecated in favor of cinder volumes for all local storage outside of the initial 20GB allocated for for standard OS functions. In support of this move we'll be making several technical changes to our infrastructure over the coming days and weeks: - Default per-project cinder usage quotas will be increased from 10GB to 80GB - Instance flavors with variable disk sizes will be removed. Going forward all flavors will have a default local storage size of 20GB regardless of associated cores or RAM. - The old lvm puppet roles will continue to function as before on legacy VMs. On new VMs the roles will detect the lack of available space for partitioning and report failures[1]. - Instance resizing will be enabled in the Horizon UI. This will allow you to adjust the allocated cores and RAM allocated to a given VM without rebuilding. Manual setup of Cinder volumes is documented here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins… If your workflow involves puppet roles that depend on labs_lvm, you will need to alter both your puppet code and your manual processes. I've provided some helper roles to assist. The simplest[2] is the new 'role::labs::cindermount::srv' which in most cases can be used as a drop-in replacement for role::labs::lvm::srv: once a cinder volume is attached to a VM that role will detect, format, and mount it as /srv. The base class cinderutils::ensure[3] can be reused for other variations on this theme. Please don't hesitate to contact me or other WMCS staff for assistance with puppet patches! In some cases the default quota of 80GB will be insufficient for a full migration to cinder from the old flavor model, especially if you have made use of the old 'large' or 'xlarge' sizes. When it comes time to to rebuild these larger VMs, please open a quota request ticket and detail what VMs (of what flavors) will be deleted and we will try to be quick and generous with quota expansions to assist in migration. [0] https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/ [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668567 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/669958 [3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668757 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] Use of Debian Stretch now discouraged
by Andrew Bogott 12 Mar '21

12 Mar '21

Greetings! The foundation has long since stopped using Debian Stretch for new projects, and will start actively removing it from the data centers this summer. In the meantime, I don't want any one who is on the fence to choose Stretch when they could choose Buster, so I'm removing it by default from the image options in Horizon. If you still need to use it for legacy reasons, feel free to contact a cloud admin or open a phabricator ticket; restoring access to a specific project is simple. I'm also leaving it enabled for the deployment-prep and integration projects as I've no doubt they'll still need to create Stretch VMs until it's fully phased out in production. -Andrew _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Problem on linking Wikimedia account and LDAP account
by Kirk Lu 06 Mar '21

06 Mar '21

Hello, I am KirkLU (with the same username in wikimedia, phab and wikitech). I find a problem while I am trying to link my Wikimedia account and LDAP account, and cannot resolve it. Everytime I try to *login using OAuth* in my setting of Toolforge admin console, I have been noticed that "*Wikimedia account 'KirkLU' is already attached to another LDAP account*". However, I cannot remember which LDAP account I have linked to. I tried to find it in the User List, but failed. I am wondering if there is a way to find the old account, unlink the old connection and attach my Wikimedia account (KirkLU) to the LDAP account (KirkLU). Additionally, my phab account<https://phabricator.wikimedia.org/p/KirkLU/> has been linked with both my LDAP account (KirkLU) and Wikimedia account (KirkLU). Really hope someone can help me resolve this issue. And thank you very much for your attention. Best regards, Kirk -- User:KirkLU | Chinese Wikimedian admin@zhwp, mOTRS@info-zh Email: kirklu.wm(a)gmail.com https://meta.wikimedia.org/wiki/User:KirkLU

3 6

Re: [Cloud] Cloud Digest, Vol 110, Issue 17
by maxime delzenne 28 Feb '21

28 Feb '21

Hi Merlijn, Thanks a lot! I can actually remember reading the PLEASE PREFIX YOUR KEYS in the doc (link below), but didn't know how to do it. I will read again ;). https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge#Secu… Psemdel Le dim. 28 févr. 2021 à 13:02, <cloud-request(a)lists.wikimedia.org> a écrit : > Send Cloud mailing list submissions to > cloud(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/cloud > or, via email, send a message with subject or body 'help' to > cloud-request(a)lists.wikimedia.org > > You can reach the person managing the list at > cloud-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Cloud digest..." > > > Today's Topics: > > 1. Re: Does redis has a sync problem? (Merlijn van Deen (valhallasw)) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 27 Feb 2021 13:56:48 +0100 > From: "Merlijn van Deen (valhallasw)" <valhallasw(a)arctus.nl> > To: Wikimedia Cloud Services general discussion and support > <cloud(a)lists.wikimedia.org> > Subject: Re: [Cloud] Does redis has a sync problem? > Message-ID: > < > CADJO0e+PvCeg5iX-KF5-8rOTigPkdaQ8Kor7KQzi4i8Zuz+JTw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Maxime, > > Looking at the error logs, it sounds like there may be a clash between > celery workers of different tools. Specifically, the log shows that there > is a 'celery(a)cycling-init-bot.worker' ready, but it's a 'celery@algo-news' > worker that is using a different time zone. As the Redis server is shared, > clients should use a prefix to prevent this from happening. > > I'm not too familiar with Celery, but it seems the underlying code allows > you to set a KEY_PREFIX through the KOMBU_REDIS_PREFIX environment > variable: > > https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/tran… > > Hope this helps! > Merlijn > > On Sat, 27 Feb 2021 at 08:32, maxime delzenne <maxime.delzenne(a)gmail.com> > wrote: > > > Hi, > > > > Since yesterday evening (round about 22h CET), my celery send me > following > > error: > > > > [2021-02-27 07:26:44,782: INFO/MainProcess] Connected to > > redis://tools-redis.svc.eqiad.wmflabs:6379/0 > > [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for > neighbors > > [2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes > > [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete > > [2021-02-27 07:26:45,983: INFO/MainProcess] > > celery(a)cycling-init-bot.worker-56ddc85977-9sf2v ready. > > [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from > > celery(a)algo-news.celery-6444f768f9-w89fb may mean clocks are out of > > sync. Current drift is > > 3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27 > > 08:26:47.740345] > > > > I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django > settings. > > > > Psemdel > > _______________________________________________ > > Wikimedia Cloud Services mailing list > > Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) > > https://lists.wikimedia.org/mailman/listinfo/cloud > > >

1 0

Does redis has a sync problem?
by maxime delzenne 27 Feb '21

27 Feb '21

Hi, Since yesterday evening (round about 22h CET), my celery send me following error: [2021-02-27 07:26:44,782: INFO/MainProcess] Connected to redis://tools-redis.svc.eqiad.wmflabs:6379/0 [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for neighbors [2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete [2021-02-27 07:26:45,983: INFO/MainProcess] celery(a)cycling-init-bot.worker-56ddc85977-9sf2v ready. [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from celery(a)algo-news.celery-6444f768f9-w89fb may mean clocks are out of sync. Current drift is 3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27 08:26:47.740345] I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django settings. Psemdel

2 1

[Cloud-announce] ssh session interruptions today
by Andrew Bogott 27 Feb '21

27 Feb '21

For security reasons we will be updating and rebooting many (possibly all) cloud-vps VMs today, over the weekend, and on Monday. The first step of this will be a reboot of all bastions (including the toolforge login hosts) in about an hour, at 18:30 UTC. That means that all ssh sessions to cloud-vps hosts will be interrupted as proxy sessions are terminated. After the tools bastion reboots, users of toolforge should not be further affected by these reboots. All other cloud-vps users can expect at least one more surprise reboot over the next few days. If you need that reboot to be scheduled at a particular time please contact me immediately. Sorry for the inconvenience! -Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud