Hello,
Pretty much everyone who's dealt with creating views for new wikis on the
labs hosts have experienced issues with "Access denied" sometimes.
This was usually due to the MariaDB grant role being missed. We tried to
workaround this by including the grant addition on the maintain-views
script.
Unfortunately, we ran into very weird problems when doing so and this is an
example: https://phabricator.wikimedia.org/T193187#4273281
After lots of back and forth we decided to create a bug to MariaDB (
https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB
yesterday and pointed to a similar issue (
https://jira.mariadb.org/browse/MDEV-14732).
The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of
us.
So, for now, the workaround before adding new views is to manually add the
GRANT on the DB and then run the script:
GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser';
Hopefully with this email everyone is on the same page now.
Thanks everyone (specially Brooke for helping me out with the
troubleshooting!)
Manuel.
2018-07-25 20:00:02,513 INFO force is enabled
2018-07-25 20:00:02,577 INFO removing misc-project-backup
2018-07-25 20:00:02,702 INFO removing misc-project-backup
2018-07-25 20:00:03,166 INFO creating misc-project-backup at 2T
2018-07-25 20:00:04,139 INFO force is enabled
2018-07-25 20:00:04,189 INFO removing misc-snap
2018-07-25 20:00:04,241 INFO removing misc-snap
2018-07-25 20:00:04,907 INFO creating misc-snap at 1T
2018-07-24 20:00:02,546 INFO force is enabled
2018-07-24 20:00:02,577 INFO removing tools-project-backup
2018-07-24 20:00:02,627 INFO removing tools-project-backup
2018-07-24 20:00:03,152 INFO creating tools-project-backup at 2T
2018-07-24 20:00:04,022 INFO force is enabled
2018-07-24 20:00:04,075 INFO removing tools-snap
2018-07-24 20:00:04,125 INFO removing tools-snap
2018-07-24 20:00:06,502 INFO creating tools-snap at 1T
2018-07-18 20:00:03,153 INFO force is enabled
2018-07-18 20:00:03,207 INFO removing misc-project-backup
2018-07-18 20:00:03,341 INFO removing misc-project-backup
2018-07-18 20:00:03,973 INFO creating misc-project-backup at 2T
2018-07-18 20:00:04,834 INFO force is enabled
2018-07-18 20:00:04,869 INFO removing misc-snap
2018-07-18 20:00:04,913 INFO removing misc-snap
2018-07-18 20:00:05,448 INFO creating misc-snap at 1T
2018-07-17 20:00:02,609 INFO force is enabled
2018-07-17 20:00:02,651 INFO removing tools-project-backup
2018-07-17 20:00:02,725 INFO removing tools-project-backup
2018-07-17 20:00:03,292 INFO creating tools-project-backup at 2T
2018-07-17 20:00:04,122 INFO force is enabled
2018-07-17 20:00:04,179 INFO removing tools-snap
2018-07-17 20:00:04,251 INFO removing tools-snap
2018-07-17 20:00:06,387 INFO creating tools-snap at 1T
Since I will be spending my time soon working most directly on things other
than Cloud Services goals I have attempted to persist my understanding of
shared/best practice to a wiki page. This is less a declaration than it is
an attempt to formalize current expectations as I understand them. I know
this is imperfect but the lack of a page like this I think has been making
life difficult for awhile and we gotta start somewhere. This is mostly
as-is but there are sprinklings of hope-to-be with links to tasks where
changes are in progress or being tracked.
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/NetworkandPolicy
--
Chase Pettet
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
2018-07-11 20:00:02,778 INFO force is enabled
2018-07-11 20:00:02,830 INFO removing misc-project-backup
2018-07-11 20:00:02,922 INFO removing misc-project-backup
2018-07-11 20:00:03,521 INFO creating misc-project-backup at 2T
2018-07-11 20:00:04,552 INFO force is enabled
2018-07-11 20:00:04,587 INFO removing misc-snap
2018-07-11 20:00:04,656 INFO removing misc-snap
2018-07-11 20:00:05,060 INFO creating misc-snap at 1T
2018-07-10 20:00:02,797 INFO force is enabled
2018-07-10 20:00:02,825 INFO removing tools-project-backup
2018-07-10 20:00:02,867 INFO removing tools-project-backup
2018-07-10 20:00:03,410 INFO creating tools-project-backup at 2T
2018-07-10 20:00:04,392 INFO force is enabled
2018-07-10 20:00:04,444 INFO removing tools-snap
2018-07-10 20:00:04,511 INFO removing tools-snap
2018-07-10 20:00:06,795 INFO creating tools-snap at 1T
(I'm not sure if Brooke already had a chance to respond)
Right now things are in a bit of a failure mode for dumps hosting which may
be complicating the issue, it's hard to say what is sane at the moment. If
you could a) drop us a note in #wikimedia-cloud when you are going to kick
off grabbing the files, and b) download from the web interface. That could
be the sanest course of action. In general nothing going on should have
been a problem, but we have a situation on our hands otherwise that is
probably complicating issues.
Best,
Chase
> Begin forwarded message:
>
> *From: *Stas Malyshev <smalyshev(a)wikimedia.org>
> *Subject: **Re: heads up--we killed a copy job*
> *Date: *July 8, 2018 at 1:14:05 PM MST
> *To: *Brooke Storm <bstorm(a)wikimedia.org>
>
> Hi!
>
> On 7/8/18 1:00 PM, Brooke Storm wrote:
>
> Hello Stas,
> We killed your cp job on wdqs-test.wikidata-query.eqiad.wmflabs today at
> 19:30-ish UTC. We wanted to give you a heads up. Somehow, it was
> causing huge load on the server, which we are trying to fix with some
> new traffic shaping settings because that really shouldn’t cause that.
>
> Sure. I was planning to do some experiments with dump
> processing/compression, which involve moving some large amounts of data
> around. Most of these should happen on the wdqs-test, locally, but I
> needed the initial file of course. Please tell me if you have any
> objections to that - i.e. if I should postpone this, or not do it on
> labs at all, or any other limitation? Is there a best way/place to run
> such things? I can download the initial file from web storage instead of
> using nfs if that's a problem.
>
> Thanks,
> --
> Stas Malyshev
> smalyshev(a)wikimedia.org
>
>
>
--
Chase Pettet
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC