[Cloud-announce] Dropping user text columns from replica views 2019-05-27

List overview All Threads
Download

newer

older

Another SQL optimization request

Optimalizing a SQL query

Brooke Storm

18 May 2019 18 May '19

12:27 a.m.

Similar to the earlier removal of text fields from the wiki replicas for comment storage refactors in Mediawiki, we are going to remove “user text” columns from the views that are deprecated in the Mediawiki schema to prepare for when they will actually be removed upstream. The column drops are tracked and explained here https://phabricator.wikimedia.org/T223406 <https://phabricator.wikimedia.org/T223406>. The tables with names such as <tablename>_compat will not see a difference in structure. The change is scheduled for Monday, May 27th. The fields that are dropping from the views are: revision: rev_user and rev_user_text. archive: ar_user and ar_user_text. ipblocks: ipb_by and ipb_by_text. image: img_user and img_user_text. oldimage: oi_user and oi_user_text. filearchive: fa_user and fa_user_text. recentchanges: rc_user and rc_user_text. logging: log_user and log_user_text. Ideally, tools that connect to the replicas should gather the information from the appropriate entries in the actor table instead, again, this is similar to the change for the comment table. The data is already there for you to start using. The alternative is to try using the related <tablename>_compat table, which won’t be changing in a user-visible way at this time. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Attachments:

attachment.htm (text/html — 5.1 KB)

Show replies by date

Pietro De Nicolao

19 May 19 May

3:07 p.m.

New subject: [Cloud-announce] Dropping user text columns from replica views 2019-05-27

Hi, I tried to update one of my tools (intersect-contribs) to use the `actor` table instead of the `revision.rev_user_text` field. Joining the `revision` and `actor` tables on `revision.rev_actor = actor.actor_id` is painfully slow and makes my tool unusable. I guess that the reason is that there is no index on the `revision.rev_actor` column (which is nullable, by the way). Would it be possible to have a view like `revision_userindex` (which has index on `rev_user_text`) with an index on the `rev_actor` column instead? In this way, this very common join would be way faster. Best, --- Pietro De Nicolao

...

Il giorno 18 mag 2019, alle ore 00:27, Brooke Storm <bstorm(a)wikimedia.org> ha scritto: Similar to the earlier removal of text fields from the wiki replicas for comment storage refactors in Mediawiki, we are going to remove “user text” columns from the views that are deprecated in the Mediawiki schema to prepare for when they will actually be removed upstream. The column drops are tracked and explained here https://phabricator.wikimedia.org/T223406 <https://phabricator.wikimedia.org/T223406>. The tables with names such as <tablename>_compat will not see a difference in structure. The change is scheduled for Monday, May 27th. The fields that are dropping from the views are: revision: rev_user and rev_user_text. archive: ar_user and ar_user_text. ipblocks: ipb_by and ipb_by_text. image: img_user and img_user_text. oldimage: oi_user and oi_user_text. filearchive: fa_user and fa_user_text. recentchanges: rc_user and rc_user_text. logging: log_user and log_user_text. Ideally, tools that connect to the replicas should gather the information from the appropriate entries in the actor table instead, again, this is similar to the change for the comment table. The data is already there for you to start using. The alternative is to try using the related <tablename>_compat table, which won’t be changing in a user-visible way at this time. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Jaime Crespo

3:39 p.m.

New subject: [Cloud-announce] Dropping user text columns from replica views 2019-05-27

On Sun, May 19, 2019 at 3:07 PM Pietro De Nicolao <powerpdn(a)gmail.com> wrote:

...

Would it be possible to have a view like `revision_userindex` (which has index on `rev_user_text`) with an index on the `rev_actor` column instead? In this way, this very common join would be way faster.

Pietro, that is a useful suggestion, please file a ticket on phabricator or comment on Brooke's ticket: that is something that looks very reasonable to me (although it may had to be done in a different way for now, as far as I can see there is not yet a *real* actor column on the underlying revision table, neither on production nor in labs, and the join on the view may be making it slow).

Brooke Storm

20 May 20 May

8:38 p.m.

New subject: [Cloud-announce] Dropping user text columns from replica views 2019-05-27 now 2019-06-03

I’d like to announce that after taking considerable user feedback about it, and fixing a performance issue regarding the revision_userindex table at https://phabricator.wikimedia.org/T221339 <https://phabricator.wikimedia.org/T221339> that affects migration of tools to the new schema, that we are extending the date to Monday, June 3rd instead to begin dropping the fields in the views. This should give additional time to fix things up as well as find issues in existing views if any more arise. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

...

On May 17, 2019, at 4:27 PM, Brooke Storm <bstorm(a)wikimedia.org> wrote: Similar to the earlier removal of text fields from the wiki replicas for comment storage refactors in Mediawiki, we are going to remove “user text” columns from the views that are deprecated in the Mediawiki schema to prepare for when they will actually be removed upstream. The column drops are tracked and explained here https://phabricator.wikimedia.org/T223406 <https://phabricator.wikimedia.org/T223406>. The tables with names such as <tablename>_compat will not see a difference in structure. The change is scheduled for Monday, May 27th. The fields that are dropping from the views are: revision: rev_user and rev_user_text. archive: ar_user and ar_user_text. ipblocks: ipb_by and ipb_by_text. image: img_user and img_user_text. oldimage: oi_user and oi_user_text. filearchive: fa_user and fa_user_text. recentchanges: rc_user and rc_user_text. logging: log_user and log_user_text. Ideally, tools that connect to the replicas should gather the information from the appropriate entries in the actor table instead, again, this is similar to the change for the comment table. The data is already there for you to start using. The alternative is to try using the related <tablename>_compat table, which won’t be changing in a user-visible way at this time. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Brooke Storm

3 Jun 3 Jun

4:27 p.m.

New subject: [Cloud-announce] Dropping user text columns from replica views 2019-05-27 now 2019-06-03

Work on this is beginning. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

...

On May 20, 2019, at 12:38 PM, Brooke Storm <bstorm(a)wikimedia.org> wrote: I’d like to announce that after taking considerable user feedback about it, and fixing a performance issue regarding the revision_userindex table at https://phabricator.wikimedia.org/T221339 <https://phabricator.wikimedia.org/T221339> that affects migration of tools to the new schema, that we are extending the date to Monday, June 3rd instead to begin dropping the fields in the views. This should give additional time to fix things up as well as find issues in existing views if any more arise. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

On May 17, 2019, at 4:27 PM, Brooke Storm <bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>> wrote: Similar to the earlier removal of text fields from the wiki replicas for comment storage refactors in Mediawiki, we are going to remove “user text” columns from the views that are deprecated in the Mediawiki schema to prepare for when they will actually be removed upstream. The column drops are tracked and explained here https://phabricator.wikimedia.org/T223406 <https://phabricator.wikimedia.org/T223406>. The tables with names such as <tablename>_compat will not see a difference in structure. The change is scheduled for Monday, May 27th. The fields that are dropping from the views are: revision: rev_user and rev_user_text. archive: ar_user and ar_user_text. ipblocks: ipb_by and ipb_by_text. image: img_user and img_user_text. oldimage: oi_user and oi_user_text. filearchive: fa_user and fa_user_text. recentchanges: rc_user and rc_user_text. logging: log_user and log_user_text. Ideally, tools that connect to the replicas should gather the information from the appropriate entries in the actor table instead, again, this is similar to the change for the comment table. The data is already there for you to start using. The alternative is to try using the related <tablename>_compat table, which won’t be changing in a user-visible way at this time. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

Brooke Storm

4 Jun 4 Jun

7:18 p.m.

New subject: [Cloud-announce] Dropping user text columns from replica views 2019-05-27 now 2019-06-03

The work on this is completed across the replicas. We’ve created a wikitech page to describe the change further and help share solutions to problems caused by the changes https://wikitech.wikimedia.org/wiki/News/Actor_storage_changes_on_the_Wiki_… <https://wikitech.wikimedia.org/wiki/News/Actor_storage_changes_on_the_Wiki_Replicas> I’m hoping to collect more useful information and tips there as we go. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

...

On Jun 3, 2019, at 8:27 AM, Brooke Storm <bstorm(a)wikimedia.org> wrote: Work on this is beginning. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

On May 20, 2019, at 12:38 PM, Brooke Storm <bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>> wrote: I’d like to announce that after taking considerable user feedback about it, and fixing a performance issue regarding the revision_userindex table at https://phabricator.wikimedia.org/T221339 <https://phabricator.wikimedia.org/T221339> that affects migration of tools to the new schema, that we are extending the date to Monday, June 3rd instead to begin dropping the fields in the views. This should give additional time to fix things up as well as find issues in existing views if any more arise. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

1787

days inactive

1805

days old

cloud@lists.wikimedia.org

Manage subscription

5 comments

3 participants

tags (0)

participants (3)

Brooke Storm
Jaime Crespo
Pietro De Nicolao