Cloud-admin March 2018

cloud-admin@lists.wikimedia.org

14 participants
23 discussions

Delete button on gerrit
by Jaime Crespo 06 Mar '18

06 Mar '18

I just been told there is now a delete button, DON'T USE IT- I just pressed it by mistake after entering in edit mode thinking it was a "discard patch started on web interface" (but it is very easy to pres it by mistake), and apparently it removes the entire CR. I was told by Paladox this is a new feature on gerrit, and I do not like it already. I managed to delete the work of a workmate. :-( I could restore everthing from the database backups, but as it also deletes the git files content, it doesn't work without it -it cannot be reverted- only text can be recover from the database in not the nicest formatting. Apologies for the damages caused. Should I file a ticket to propose to disable such a button from the UI? -- Jaime Crespo <http://wikimedia.org>

6 6

FYI: Another nodepool outage this morning -- now resolved
by Andrew Bogott 03 Mar '18

03 Mar '18

CI tests stopped working again a couple of hours ago; I got a ping on IRC from paladox when I first woke up and checked my phone. The issue this morning was different, but most likely fallout from last night's issue. In short, nova's records of quota usage were all goofy and nova was preventing VM creation due to perceived quota violation even though we weren't actually over quota. I did some manual database intervention to force nova to re-calculate usages and now all is well. Details, including the commands I used, in https://phabricator.wikimedia.org/T188820. I also added example snippets to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#…. -Andrew

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 01 Mar '18

01 Mar '18

2018-02-28 20:00:02,835 INFO force is enabled 2018-02-28 20:00:02,893 INFO removing misc-project-backup 2018-02-28 20:00:03,001 INFO removing misc-project-backup 2018-02-28 20:00:03,932 INFO creating misc-project-backup at 2T 2018-02-28 20:00:04,783 ERROR b' /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558073344: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558130688: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 0: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 4096: Input/output error\n' 2018-02-28 20:00:04,784 INFO force is enabled 2018-02-28 20:00:04,813 ERROR b' /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558073344: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558130688: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 0: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 4096: Input/output error\n' 2018-02-28 20:00:04,813 INFO removing misc-snap 2018-02-28 20:00:04,829 ERROR b' /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558073344: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558130688: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 0: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 4096: Input/output error\n' 2018-02-28 20:00:04,846 ERROR b' /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558073344: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558130688: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 0: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 4096: Input/output error\n' 2018-02-28 20:00:04,846 INFO removing misc-snap 2018-02-28 20:00:05,434 ERROR b' /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558073344: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 5497558130688: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 0: Input/output error\n /dev/misc/misc-snap: read failed after 0 of 4096 at 4096: Input/output error\n' 2018-02-28 20:00:05,497 INFO creating misc-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin March 2018