Hello, I'm writing to notify any interested party that I am stepping from maintaining Quarry to focus on superset. I'm not sure if anyone here would be interested in stepping up as a maintainer to keep Quarry running.
If no maintainer steps forward Quarry is likely to be removed when the Buster images are removed by SRE, I believe that happens in June.
Should anyone be interested in being a maintainer for Quarry please let me know and I will happily add you as one.
Some discussion can be found at https://phabricator.wikimedia.org/T169452
Thank you
That is sad, Vivian. I love Quarry.
But then again, at this point, is there any feature that Quarry offers which Superset doesn't? Maybe this is a natural time for Quarry to be sunset?
On Tue, Oct 3, 2023 at 4:59 PM Vivian Rook vrook@wikimedia.org wrote:
Hello, I'm writing to notify any interested party that I am stepping from maintaining Quarry to focus on superset. I'm not sure if anyone here would be interested in stepping up as a maintainer to keep Quarry running.
If no maintainer steps forward Quarry is likely to be removed when the Buster images are removed by SRE, I believe that happens in June.
Should anyone be interested in being a maintainer for Quarry please let me know and I will happily add you as one.
Some discussion can be found at https://phabricator.wikimedia.org/T169452
Thank you
*Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation https://wikimediafoundation.org/ _______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o... _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Hi Vivian,
First, thank you for maintaining the Quarry.
A couple of questions from a technical perspective.
First, based on GitHub requirements.txt the library versions used in the app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
Secondly, is there something that would prevent moving it to Toolforge? I am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Also, it would be worth creating a Phabricator ticket for moving maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Br, -- Kimmo Virtanen, Zache
First, based on GitHub requirements.txt the library versions used in the
app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
That's very much so the case. The primary lacking detail is time to get things updated. Should anyone want to do so, they are welcome to. There are no known abandoned or otherwise overtly problematic things in quarry. And usually the effort in upgrading quarry is in fussing with calls that changed syntax or the like between versions. There are some weird things in quarry, namely how the db is structured. I find it unintuitive that there isn't a unique ID/query, rather they're called through several different tables by different ids, this doesn't stop quarry from working, just one of the things that in my mind needs fixed. In terms of debt, it's covered in the quarry board (https://phabricator.wikimedia.org/project/view/800/), there are, reasonable, desires that the UI be updated to be more intuitive, or offer a query builder, that the database selector shouldn't be offering databases that don't exist. Many features and bug fixes are requested.
Secondly, is there something that would prevent moving it to Toolforge? I
am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Nothing really, though I think that would be more complicated than leaving it in place. In concept it could be left largely as is, so long as someone wants to do OS and resulting python/library upgrades, quarry could continue on. If you really wanted to, it would need re-written to fit into the toolforge framework, I'm not quite sure what that would entail.
Also, it would be worth creating a Phabricator ticket for moving
maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Opening a ticket for a handoff is welcome. It's mostly as you mentioned, it's just time to manage upgrades. Which I've been told to redirect to other efforts. You're welcome to open one and I can update it if desired, though I think I'll wait for a maintainer to step forward before I open such a ticket.
Caching of results is one of the features that superset and other services that I was investigating do not offer. It follows the idea that old data is less valuable than fresh data. You can share queries through things like charts, and those can be re-run as needed to refresh the data for them. Though the results themselves don't stay around for very long in superset. In terms of OAUTH https://phabricator.wikimedia.org/T336522 is to offer the ability to view without login. When I have some time for that I will see how I can get it to allow read/refresh access to anonymous users. (I say above that I'm working on superset, when in reality I'm being redirected to openstack)
Vivian Rook
On Wed, Oct 4, 2023 at 4:26 AM Kimmo Virtanen kimmo.virtanen@gmail.com wrote:
Hi Vivian,
First, thank you for maintaining the Quarry.
A couple of questions from a technical perspective.
First, based on GitHub requirements.txt the library versions used in the app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
Secondly, is there something that would prevent moving it to Toolforge? I am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Also, it would be worth creating a Phabricator ticket for moving maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Br, -- Kimmo Virtanen, Zache
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
I would be happy to be added as a maintainer unless someone more capable steps in. I also mostly use Superset nowadays, but I think there are a handful of features in Quarry (viewing cached results, wikitable export) which make it worth maintaining.
That being said, I don't have much time on my hands to work on new features or solve non-trivial existing bugs, but I can ensure the service remains up and running until its loyal user base is happy to move on to Superset.
On Wed, 4 Oct 2023 at 18:44, Vivian Rook vrook@wikimedia.org wrote:
First, based on GitHub requirements.txt the library versions used in the
app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
That's very much so the case. The primary lacking detail is time to get things updated. Should anyone want to do so, they are welcome to. There are no known abandoned or otherwise overtly problematic things in quarry. And usually the effort in upgrading quarry is in fussing with calls that changed syntax or the like between versions. There are some weird things in quarry, namely how the db is structured. I find it unintuitive that there isn't a unique ID/query, rather they're called through several different tables by different ids, this doesn't stop quarry from working, just one of the things that in my mind needs fixed. In terms of debt, it's covered in the quarry board (https://phabricator.wikimedia.org/project/view/800/), there are, reasonable, desires that the UI be updated to be more intuitive, or offer a query builder, that the database selector shouldn't be offering databases that don't exist. Many features and bug fixes are requested.
Secondly, is there something that would prevent moving it to Toolforge?
I am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Nothing really, though I think that would be more complicated than leaving it in place. In concept it could be left largely as is, so long as someone wants to do OS and resulting python/library upgrades, quarry could continue on. If you really wanted to, it would need re-written to fit into the toolforge framework, I'm not quite sure what that would entail.
Also, it would be worth creating a Phabricator ticket for moving
maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Opening a ticket for a handoff is welcome. It's mostly as you mentioned, it's just time to manage upgrades. Which I've been told to redirect to other efforts. You're welcome to open one and I can update it if desired, though I think I'll wait for a maintainer to step forward before I open such a ticket.
Caching of results is one of the features that superset and other services that I was investigating do not offer. It follows the idea that old data is less valuable than fresh data. You can share queries through things like charts, and those can be re-run as needed to refresh the data for them. Though the results themselves don't stay around for very long in superset. In terms of OAUTH https://phabricator.wikimedia.org/T336522 is to offer the ability to view without login. When I have some time for that I will see how I can get it to allow read/refresh access to anonymous users. (I say above that I'm working on superset, when in reality I'm being redirected to openstack)
Vivian Rook
On Wed, Oct 4, 2023 at 4:26 AM Kimmo Virtanen kimmo.virtanen@gmail.com wrote:
Hi Vivian,
First, thank you for maintaining the Quarry.
A couple of questions from a technical perspective.
First, based on GitHub requirements.txt the library versions used in the app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
Secondly, is there something that would prevent moving it to Toolforge? I am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Also, it would be worth creating a Phabricator ticket for moving maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Br, -- Kimmo Virtanen, Zache
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
--
*Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation https://wikimediafoundation.org/ _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
I commented on the bug to get more clarity, but I'm considering helping maintain too.
Maybe we can spread it out amongst Siddharth and myself to make it even less of a burden? That would be ideal.
Should we keep discussion on Phabricator to finalize any arrangements/handoffs?
Thanks, -Travis
On Wed, Oct 4, 2023 at 10:13 AM Siddharth VP siddharthvp@gmail.com wrote:
I would be happy to be added as a maintainer unless someone more capable steps in. I also mostly use Superset nowadays, but I think there are a handful of features in Quarry (viewing cached results, wikitable export) which make it worth maintaining.
That being said, I don't have much time on my hands to work on new features or solve non-trivial existing bugs, but I can ensure the service remains up and running until its loyal user base is happy to move on to Superset.
On Wed, 4 Oct 2023 at 18:44, Vivian Rook vrook@wikimedia.org wrote:
First, based on GitHub requirements.txt the library versions used in
the app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
That's very much so the case. The primary lacking detail is time to get things updated. Should anyone want to do so, they are welcome to. There are no known abandoned or otherwise overtly problematic things in quarry. And usually the effort in upgrading quarry is in fussing with calls that changed syntax or the like between versions. There are some weird things in quarry, namely how the db is structured. I find it unintuitive that there isn't a unique ID/query, rather they're called through several different tables by different ids, this doesn't stop quarry from working, just one of the things that in my mind needs fixed. In terms of debt, it's covered in the quarry board (https://phabricator.wikimedia.org/project/view/800/), there are, reasonable, desires that the UI be updated to be more intuitive, or offer a query builder, that the database selector shouldn't be offering databases that don't exist. Many features and bug fixes are requested.
Secondly, is there something that would prevent moving it to Toolforge?
I am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Nothing really, though I think that would be more complicated than leaving it in place. In concept it could be left largely as is, so long as someone wants to do OS and resulting python/library upgrades, quarry could continue on. If you really wanted to, it would need re-written to fit into the toolforge framework, I'm not quite sure what that would entail.
Also, it would be worth creating a Phabricator ticket for moving
maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Opening a ticket for a handoff is welcome. It's mostly as you mentioned, it's just time to manage upgrades. Which I've been told to redirect to other efforts. You're welcome to open one and I can update it if desired, though I think I'll wait for a maintainer to step forward before I open such a ticket.
Caching of results is one of the features that superset and other services that I was investigating do not offer. It follows the idea that old data is less valuable than fresh data. You can share queries through things like charts, and those can be re-run as needed to refresh the data for them. Though the results themselves don't stay around for very long in superset. In terms of OAUTH https://phabricator.wikimedia.org/T336522 is to offer the ability to view without login. When I have some time for that I will see how I can get it to allow read/refresh access to anonymous users. (I say above that I'm working on superset, when in reality I'm being redirected to openstack)
Vivian Rook
On Wed, Oct 4, 2023 at 4:26 AM Kimmo Virtanen kimmo.virtanen@gmail.com wrote:
Hi Vivian,
First, thank you for maintaining the Quarry.
A couple of questions from a technical perspective.
First, based on GitHub requirements.txt the library versions used in the app are older than the latest ones, and T169452 also hints at growing technical debt in terms of updating code. However, are there known blockers for updating them? Ie. Does Quarry use something abandoned and blocks updating others, or is there something else that would require a rewrite? Or is it that updating is expected to work, but it just takes someone's time to do it?
Secondly, is there something that would prevent moving it to Toolforge? I am unsure if it would be a viable solution, but it would reduce the maintenance burden if volunteers would maintain it, so I am asking what would be blocking it.
Also, it would be worth creating a Phabricator ticket for moving maintenance to the next person, describing what it would require and how it could be done. I do not know if there would be anyone, but for example, my use case for Quarry is sharing SQL queries and query results with wikieditors, and as long results aren't cached and reading the results requires OAUTH-login Superset doesn't work.
Br, -- Kimmo Virtanen, Zache
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
--
*Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation https://wikimediafoundation.org/ _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/