- Cloud - lists.wikimedia.org

[Cloud-announce] REMINDER cloud-vps maintenance Thursday, 2019-12-12
by Andrew Bogott 12 Dec '19

12 Dec '19

Reminder: this maintenance is happening today, starting in about 45 minutes. We'll be upgrading the cloud services OpenStack install on Monday, beginning at 06:00 UTC The entire upgrade process may take an hour or two. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) There may also be brief network interruptions during the upgrade. Toolforge and existing VMs should be largely unaffected apart from possible network hiccups. This is the second upgrade since October; we are trying to speed up our upgrade schedule in order to catch up with the upstream releases. With luck there will be further such upgrades in the coming months. - Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

toolforge and jupyter?
by Roy Smith 08 Dec '19

08 Dec '19

I want to use jupyter to do some data visualization. I found PAWS, but I can't access my toolforge directory: > FileNotFoundError: [Errno 2] No such file or directory: '/home/roysmith/sock-classifier/jobs/last/job.out' Is there a way to access my toolforge directory from within PAWS? Alternately, is there a way to run jupyter within toolforge? I tried installing it myself but it won't run: > (env) temp $ jupyter notebook > Traceback (most recent call last): > File "/mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/bin/jupyter-notebook", line 11, in <module> > sys.exit(main()) > File "/mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/lib/python3.5/site-packages/jupyter_core/application.py", line 268, in launch_instance > return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs) > File "/mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/lib/python3.5/site-packages/traitlets/config/application.py", line 663, in launch_instance > app.initialize(argv) > File "</mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/lib/python3.5/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize > File "/mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error > return method(app, *args, **kwargs) > File "/mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/lib/python3.5/site-packages/notebook/notebookapp.py", line 1716, in initialize > self.init_resources() > File "/mnt/nfs/labstore-secondary-tools-home/roysmith/temp/env/lib/python3.5/site-packages/notebook/notebookapp.py", line 1420, in init_resources > resource.setrlimit(resource.RLIMIT_NOFILE, (soft, hard)) > ValueError: not allowed to raise maximum limit

2 3

Cassandra?
by Roy Smith 07 Dec '19

07 Dec '19

Is there a publicly-accessable Cassandra instance? https://wikitech.wikimedia.org/wiki/Cassandra#Getting_a_CQL_shell <https://wikitech.wikimedia.org/wiki/Cassandra#Getting_a_CQL_shell> implies that there is, but cqlsh doesn't seem to exist on tools-sgebastion-08: > $ cqlsh > -bash: cqlsh: command not found

3 3

[Cloud-announce] cloud-vps maintenance Thursday, 2019-12-12
by Andrew Bogott 05 Dec '19

05 Dec '19

We'll be upgrading the cloud services OpenStack install on Monday, beginning at 06:00 UTC The entire upgrade process may take an hour or two. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) There may also be brief network interruptions during the upgrade. Toolforge and existing VMs should be largely unaffected apart from possible network hiccups. This is the second upgrade since October; we are trying to speed up our upgrade schedule in order to catch up with the upstream releases. With luck there will be further such upgrades in the coming months. - Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Decoding log_params for blocks?
by Roy Smith 05 Dec '19

05 Dec '19

Is there a description somewhere of what the log_params mean in this query? MariaDB [enwiki_p]> select log_params from logging_logindex where log_namespace = 2 and log_title = 'RoySmith-testing' and log_type = 'block' order by log_timestamp asc; +---------------------------------------------------------------------------------------------------------+ | log_params | +---------------------------------------------------------------------------------------------------------+ | a:3:{s:11:"5::duration";s:9:"5 minutes";s:8:"6::flags";s:11:"noautoblock";s:8:"sitewide";b:1;} | | a:3:{s:11:"5::duration";s:9:"5 minutes";s:8:"6::flags";s:20:"nocreate,noautoblock";s:8:"sitewide";b:1;} | | a:0:{} | +---------------------------------------------------------------------------------------------------------+ 3 rows in set (0.00 sec)

3 6

/util/arch and /usrlbin//now missing?
by Roy Smith 30 Nov '19

30 Nov '19

When I run qstatus on tools-sgebastion-08, I get the following errors: (env) utils $ qstatus /usr/bin/qstatus: line 299: /util/arch: No such file or directory /usr/bin/qstatus: line 300: /utilbin//now: No such file or directory Waiting jobs for user: roysmith job-ID # name submit time -------------------------------------------------------- 2249533 1 run-socks.21924 11/29/2019 21:58:00

2 1

Intermittent hangs on tools-sgebastion-07?
by Roy Smith 29 Nov '19

29 Nov '19

The last couple of days, I've been having problems with interactive ssh into login.tools.wmflabs.org <http://login.tools.wmflabs.org/>. Every so often (multiple times an hour, at least), my connection will hang for a few seconds. Sometimes more like 10-15 seconds. I connect from my home MacOS box on broadband using: > ssh -t -i ~/.ssh/id_rsa_wikimedia roysmith(a)login.tools.wmflabs.org tmux attach -t work The load doesn't look unreasonable: > $ uptime > 18:26:34 up 35 days, 9:04, 39 users, load average: 0.74, 1.93, 1.80 and ping times look fine: > $ ping -v login.tools.wmflabs.org > PING login.tools.wmflabs.org (185.15.56.48): 56 data bytes > 64 bytes from 185.15.56.48: icmp_seq=0 ttl=51 time=24.233 ms > 64 bytes from 185.15.56.48: icmp_seq=1 ttl=51 time=27.086 ms > 64 bytes from 185.15.56.48: icmp_seq=2 ttl=51 time=22.121 ms > 64 bytes from 185.15.56.48: icmp_seq=3 ttl=51 time=22.726 ms > 64 bytes from 185.15.56.48: icmp_seq=4 ttl=51 time=24.497 ms > 64 bytes from 185.15.56.48: icmp_seq=5 ttl=51 time=24.809 ms > 64 bytes from 185.15.56.48: icmp_seq=6 ttl=51 time=23.913 ms > 64 bytes from 185.15.56.48: icmp_seq=7 ttl=51 time=25.811 ms > 64 bytes from 185.15.56.48: icmp_seq=8 ttl=51 time=25.266 ms > 64 bytes from 185.15.56.48: icmp_seq=9 ttl=51 time=22.865 ms > 64 bytes from 185.15.56.48: icmp_seq=10 ttl=51 time=32.076 ms > 64 bytes from 185.15.56.48: icmp_seq=11 ttl=51 time=26.069 ms > 64 bytes from 185.15.56.48: icmp_seq=12 ttl=51 time=27.947 ms > 64 bytes from 185.15.56.48: icmp_seq=13 ttl=51 time=27.088 ms > ^C > --- login.tools.wmflabs.org ping statistics --- > 14 packets transmitted, 14 packets received, 0.0% packet loss > round-trip min/avg/max/stddev = 22.121/25.465/32.076/2.484 ms I'm in New York City, and login.tools.wmflabs.org <http://login.tools.wmflabs.org/> looks like it's in Virginia, so that's pretty close. This seems to have started in the past few days. Anybody else seeing problems?

4 5

protobufs?
by Roy Smith 27 Nov '19

27 Nov '19

I'm starting to look at some machine learning projects I've wanted to do for a while (ex: sock-puppet detection). This quickly leads to having to make decisions about data storage formats, i.e. csv, json, protobufs, etc. Left to my own devices, I'd probably use protos, but I don't want to be swimming upstream. Are there any standards in wiki-land for how people store data? If there's some common way that "everybody does it", that's how I want to do it too. Or, does every project just do their own thing?

2 4

[Cloud-announce] Cloud VPS users, please claim your projects
by Andrew Bogott 25 Nov '19

25 Nov '19

Every year or so the Cloud Services team tries to identify and clean up unused projects and VMs. We do this via an opt-in process: anyone can mark a project as 'in use,' and that project will be preserved for another year. I've created a wiki page the lists all existing projects, here: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge If you are a VPS user, please visit that page and mark any projects that you use as {{Used}}. Note that it's not necessary for you to be a project admin to mark something -- if you know that you're currently using a resource and want to keep using it, go ahead and mark it accordingly. If you /are/ a project admin, please take a moment to mark which VMs are or aren't used in your projects. When December arrives, I will shut down and begin the process of reclaiming resources from unused projects. If you think you use a VPS project but aren't sure which, I encourage you to poke around on https://tools.wmflabs.org/openstack-browser/ to see what looks familiar. Worst case, just email cloud(a)lists.wikimedia.org with a description of your use case and we'll sort it out there. Exclusive toolforge users are free to ignore this task. Thank you! -Andrew and WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

2 4

Please help with optimizing a SQL query
by Martin Urbanec 24 Nov '19

24 Nov '19

Hello, could someone please help me with optimizing the following query? USE commonswiki_p; SELECT first_upload, uploads, username FROM ( SELECT MIN(log_timestamp) AS first_upload, MIN(log_id) AS first_upload_id, COUNT(log_timestamp) AS uploads, log_user_text AS username FROM logging_compat LEFT JOIN user ON user_id = log_user JOIN page ON log_page = page_id WHERE log_type = "upload" AND (log_action = "upload" OR log_action = "overwrite") AND user_registration > "20190101000000" GROUP BY log_user ) AS first_uploads JOIN change_tag ON ct_log_id = first_upload_id WHERE ct_tag_id=21; It takes over 30 minutes :/. I want to have a list of users whose first contrib to Wikimedia Commons is tagged with tag number 21. Thanks! Martin

2 3

2024

2023

2022

2021

2020

2019

2018

2017

Cloud