Following several requests from users over the past eight years [0], we are finally enabling access to ToolsDB's "public" databases (the ones with a name ending with "_p") [1] from both Quarry [2] and Superset [3].
The data stored in those databases have always been accessible to every Toolforge user, but after this change they will become more broadly accessible, as Quarry can be accessed by anyone with a Wikimedia account, and saved queries in Quarry can be shared with public links that require no login at all.
== This change is planned to go live on Monday, July 1st. ==
If you have any concerns or questions related to this change, please leave a comment in the Phabricator task or one of its subtasks. [0]
Thanks to everyone for your patience and for keeping the task alive over the years!
[0] https://phabricator.wikimedia.org/T151158 [1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_th... [2] https://meta.wikimedia.org/wiki/Research:Quarry [3] https://superset.wmcloud.org/
On Thu, Jun 13, 2024 at 4:59 PM Francesco Negri fnegri@wikimedia.org wrote:
== This change is planned to go live on Monday, July 1st. ==
This change is now live, and ToolsDB's "public" databases (the ones with a name ending with "_p") can be queried from Quarry.
In Superset, databases must unfortunately be added individually, so if you are interested in querying one of those databases from Superset please leave a comment in https://phabricator.wikimedia.org/T367393
The last request in the parent task (T151158) was to be able to query Quarry's own database. This is not implemented yet and is tracked in https://phabricator.wikimedia.org/T367415
I will also document all the above in Wikitech.
On 3/7/24 00:48, Francesco Negri wrote:
On Thu, Jun 13, 2024 at 4:59 PM Francesco Negri fnegri@wikimedia.org wrote:
== This change is planned to go live on Monday, July 1st. ==
This change is now live, and ToolsDB's "public" databases (the ones with a name ending with "_p") can be queried from Quarry.
In Superset, databases must unfortunately be added individually, so if you are interested in querying one of those databases from Superset please leave a comment in https://phabricator.wikimedia.org/T367393
The last request in the parent task (T151158) was to be able to query Quarry's own database. This is not implemented yet and is tracked in https://phabricator.wikimedia.org/T367415
I will also document all the above in Wikitech.
Does anything need to be done to have a tool database be visible to Quarry? Should it be possible to query s55926__wishlist_p for example (which was only created yesterday; it's for the Community Wishlist wishes). Example query: https://quarry.wmcloud.org/query/11263
Hi Sam, was that database created today?
Quarry is connecting to the replica host, which currently has a replication lag of 13 hours. It should catch up in a day or two.
The lag issue is tracked at https://phabricator.wikimedia.org/T370760 and the parent task https://phabricator.wikimedia.org/T357624
Hope this helps, Francesco
On Tue, Jul 23, 2024 at 3:23 PM Sam Wilson sam@samwilson.id.au wrote:
On 3/7/24 00:48, Francesco Negri wrote:
On Thu, Jun 13, 2024 at 4:59 PM Francesco Negri fnegri@wikimedia.org wrote:
== This change is planned to go live on Monday, July 1st. ==
This change is now live, and ToolsDB's "public" databases (the ones with a name ending with "_p") can be queried from Quarry.
In Superset, databases must unfortunately be added individually, so if you are interested in querying one of those databases from Superset please leave a comment in https://phabricator.wikimedia.org/T367393
The last request in the parent task (T151158) was to be able to query Quarry's own database. This is not implemented yet and is tracked in https://phabricator.wikimedia.org/T367415
I will also document all the above in Wikitech.
Does anything need to be done to have a tool database be visible to Quarry? Should it be possible to query s55926__wishlist_p for example (which was only created yesterday; it's for the Community Wishlist wishes). Example query: https://quarry.wmcloud.org/query/11263
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Yep, it was created yesterday.
Thanks, good to know it's just a matter of waiting.
On 23/7/24 22:14, Francesco Negri wrote:
Hi Sam, was that database created today?
Quarry is connecting to the replica host, which currently has a replication lag of 13 hours. It should catch up in a day or two.
The lag issue is tracked at https://phabricator.wikimedia.org/T370760 and the parent task https://phabricator.wikimedia.org/T357624
Hope this helps, Francesco
On Tue, Jul 23, 2024 at 3:23 PM Sam Wilson sam@samwilson.id.au wrote:
On 3/7/24 00:48, Francesco Negri wrote:
On Thu, Jun 13, 2024 at 4:59 PM Francesco Negri fnegri@wikimedia.org wrote:
== This change is planned to go live on Monday, July 1st. ==
This change is now live, and ToolsDB's "public" databases (the ones with a name ending with "_p") can be queried from Quarry.
In Superset, databases must unfortunately be added individually, so if you are interested in querying one of those databases from Superset please leave a comment in https://phabricator.wikimedia.org/T367393
The last request in the parent task (T151158) was to be able to query Quarry's own database. This is not implemented yet and is tracked in https://phabricator.wikimedia.org/T367415
I will also document all the above in Wikitech.
Does anything need to be done to have a tool database be visible to Quarry? Should it be possible to query s55926__wishlist_p for example (which was only created yesterday; it's for the Community Wishlist wishes). Example query: https://quarry.wmcloud.org/query/11263
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Hi Sam, an update on this: the replica is still delayed, and based on my estimation it will take two more days to catch up. [1] This is a recurring problem that we're trying to improve, but it usually happens once a month or less, and most of the time you should be able to see up-to-date information in Quarry.
Apologies for the inconvenience, Francesco
[1] https://phabricator.wikimedia.org/T370760 [2] https://phabricator.wikimedia.org/T357624
On Wed, Jul 24, 2024 at 1:59 AM Sam Wilson sam@samwilson.id.au wrote:
Yep, it was created yesterday.
Thanks, good to know it's just a matter of waiting.
On 23/7/24 22:14, Francesco Negri wrote:
Hi Sam, was that database created today?
Quarry is connecting to the replica host, which currently has a replication lag of 13 hours. It should catch up in a day or two.
The lag issue is tracked at https://phabricator.wikimedia.org/T370760 and the parent task https://phabricator.wikimedia.org/T357624
Hope this helps, Francesco
On Tue, Jul 23, 2024 at 3:23 PM Sam Wilson sam@samwilson.id.au wrote:
On 3/7/24 00:48, Francesco Negri wrote:
On Thu, Jun 13, 2024 at 4:59 PM Francesco Negri fnegri@wikimedia.org wrote:
== This change is planned to go live on Monday, July 1st. ==
This change is now live, and ToolsDB's "public" databases (the ones with a name ending with "_p") can be queried from Quarry.
In Superset, databases must unfortunately be added individually, so if you are interested in querying one of those databases from Superset please leave a comment in https://phabricator.wikimedia.org/T367393
The last request in the parent task (T151158) was to be able to query Quarry's own database. This is not implemented yet and is tracked in https://phabricator.wikimedia.org/T367415
I will also document all the above in Wikitech.
Does anything need to be done to have a tool database be visible to Quarry? Should it be possible to query s55926__wishlist_p for example (which was only created yesterday; it's for the Community Wishlist wishes). Example query: https://quarry.wmcloud.org/query/11263
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
an update on this: the replica is still delayed, and based on my estimation it will take two more days to catch up
The replica is finally back in sync with the primary database, and the example query for database s55926__wishlist_p is now returning results: https://quarry.wmcloud.org/query/11263
On Tue, Jul 2, 2024 at 6:48 PM Francesco Negri fnegri@wikimedia.org wrote:
In Superset, databases must unfortunately be added individually
This has been solved [1], and all ToolsDB public databases (the ones ending with "_p") can be queried from both Quarry and Superset.
The last request in the parent task (T151158) was to be able to query Quarry's own database. This is not implemented yet
This feature has now been implemented [2] and you can type "quarry" or "quarry_p" as the database name in Quarry to query 4 tables on the internal database (query, query_revision, query_run, star). These tables contain information about all queries that are being run and starred by users in Quarry. This information was already public through the web interface, but you can now query it using SQL.
I have marked the original task [0] as Resolved, almost 8 years after it was created by intracer in 2016. Thanks to all the people who participated with patches, comments and requests over this time, and thanks all for your patience!
Francesco
[0] https://phabricator.wikimedia.org/T151158 [1] https://phabricator.wikimedia.org/T367393 [2] https://phabricator.wikimedia.org/T367415