So where is the best current place to discuss scaling Commons, and all
that entails?
My impression is that we don't have one. All we hear is "it needs to be planned", but there is no transparency on what that planning involves or when it actually happens.
I'd be surprised if the bottleneck were people or budget
The main problem I see is that we end up in this kind of situation. Scaling and bug fixing critical features should be part of the annual budget. Each line of code deployed to production wikis should have an owner and associated maintenance budget each year. Without this, the team will not even commit reviews - see the thread on wikitech a few months back where a volunteer programmer willing to work on Upload Wizard was basically told "We will not review your code. Go fork."
Some examples from recent discussions
Also improvements to the Upload Wizard. There are quite a few open items in Phab on this.
I really hope you will have better luck than others with bringing this issue up in the priority list for next year - multimedia support is growing more outdated by the minute.
Strainu
Pe joi, 30 decembrie 2021, Samuel Klein meta.sj@gmail.com a scris:
Separate thread. I'm not sure which list is appropriate. ... but not all the way to sentience.
The annual community wishlist survey (implemented by a small team,
possibly in isolation?) may not be the mechanism for prioritizing large changes, but the latter also deserves a community-curated priority queue. To complement the staff-maintained priorities in phab ~
For core challenges (like Commons stability and capacity), I'd be
surprised if the bottleneck were people or budget. We do need a shared understanding of what issues are most important and most urgent, and how to solve them. For instance, a way to turn Amir's recent email about the problem (and related phab tickets) into a family of persistent, implementable specs and proposals and their articulated obstacles.
An issue tracker like phab is good for tracking the progress and
dependencies of agreed-upon tasks, but weak for discussing what is important, what we know about it, how to address it. And weak for discussing ecosystem-design issues that are important and need persistent updating but don't have a simple checklist of steps.
So where is the best current place to discuss scaling Commons, and all
that entails? Some examples from recent discussions (most from the wm-l thread below):
- Uploads: Support for large file uploads / Keeping bulk upload tools
online
- Video: Debugging + rolling out the videojs player
- Formats: Adding support for CML and dozens of other common high-demand
file formats
- Thumbs: Updating thumbor and librsvg
- Search: WCQS still down, noauth option wanted for tools
- General: Finish implementing redesign of the image table
SJ On Wed, Dec 29, 2021 at 6:26 AM Amir Sarabadani ladsgroup@gmail.com
wrote:
I'm not debating your note. It is very valid that we lack proper support
for multimedia stack. I myself wrote a detailed rant on how broken it is [1] but three notes:
- Fixing something like this takes time, you need to assign the budget
for it (which means it has to be done during the annual planning) and if gets approved, you need to start it with the fiscal year (meaning July 2022) and then hire (meaning, write JD, do recruitment, interview lots of people, get them hired) which can take from several months to years. Once they are hired, you need to onboard them and let them learn about our technical infrastructure which takes at least two good months. Software engineering is not magic, it takes time, blood and sweat. [2]
- Making another team focus on multimedia requires changes in planning,
budget, OKR, etc. etc. Are we sure moving the focus of teams is a good idea? Most teams are already focusing on vital parts of wikimedia and changing the focus will turn this into a whack-a-mole game where we fix multimedia but now we have critical issues in security or performance.
- Voting Wishlist survey is a good band-aid in the meantime. To at
least address the worst parts for now.
I don't understand your point tbh, either you think it's a good idea to
make requests for improvements in multimedia in the wishlist survey or you think it's not. If you think it's not, then it's offtopic to this thread.
[1]
https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/...
[2] There is a classic book in this topic called "The Mythical Man-month"
On Wed, Dec 29, 2021 at 11:41 AM Gnangarra gnangarra@gmail.com wrote:
we have to vote for regular maintenance and support for
essential functions like uploading files which is the core mission of Wikimedia Commons
On Fri, Dec 31, 2021 at 12:42 AM Strainu strainu10@gmail.com wrote:
So where is the best current place to discuss scaling Commons, and all
that entails?
My impression is that we don't have one. All we hear is "it needs to be planned", but there is no transparency on what that planning involves or when it actually happens.
I'd be surprised if the bottleneck were people or budget
The main problem I see is that we end up in this kind of situation. Scaling and bug fixing critical features should be part of the annual budget. Each line of code deployed to production wikis should have an owner and associated maintenance budget each year. Without this, the team will not even commit reviews - see the thread on wikitech a few months back where a volunteer programmer willing to work on Upload Wizard was basically told "We will not review your code. Go fork."
There is "code stewardship program" and its goal is to find owners for components that don't have an owner (or undeploy them). Sometimes it's successful, sometimes it's not. I have been asking for a maintainer for FlaggedRevs for four years now, beta cluster is suffering from a similar situation, etc. etc. It takes time to find an owner for everything, to fill the gaps in places we don't have a team to handle those (e.g. Multimedia, you can't just hand over that to team responsible for security for example). More info at https://www.mediawiki.org/wiki/Code_stewardship_reviews
Some examples from recent discussions
Also improvements to the Upload Wizard. There are quite a few open items in Phab on this.
+1
I really hope you will have better luck than others with bringing this issue up in the priority list for next year - multimedia support is growing more outdated by the minute.
Honestly, the situation is more dire than you think. For example, until a couple months ago, we didn't have backups for the media files. There was a live copy in the secondary datacenter but for example if due to a software issue, we lost some files, they were gone. I would like to thank Jaime Crespo for pushing for it and implementing the backups.
But I beat my drum again, it's not something you can fix overnight. I'm sure people are monitoring this mailing list and are aware of the problem.
Best
Strainu
Pe joi, 30 decembrie 2021, Samuel Klein meta.sj@gmail.com a scris:
Separate thread. I'm not sure which list is appropriate. ... but not all the way to sentience.
The annual community wishlist survey (implemented by a small team,
possibly in isolation?) may not be the mechanism for prioritizing large changes, but the latter also deserves a community-curated priority queue. To complement the staff-maintained priorities in phab ~
For core challenges (like Commons stability and capacity), I'd be
surprised if the bottleneck were people or budget. We do need a shared understanding of what issues are most important and most urgent, and how to solve them. For instance, a way to turn Amir's recent email about the problem (and related phab tickets) into a family of persistent, implementable specs and proposals and their articulated obstacles.
An issue tracker like phab is good for tracking the progress and
dependencies of agreed-upon tasks, but weak for discussing what is important, what we know about it, how to address it. And weak for discussing ecosystem-design issues that are important and need persistent updating but don't have a simple checklist of steps.
So where is the best current place to discuss scaling Commons, and all
that entails? Some examples from recent discussions (most from the wm-l thread below):
- Uploads: Support for large file uploads / Keeping bulk upload tools
online
- Video: Debugging + rolling out the videojs player
- Formats: Adding support for CML and dozens of other common high-demand
file formats
- Thumbs: Updating thumbor and librsvg
- Search: WCQS still down, noauth option wanted for tools
- General: Finish implementing redesign of the image table
SJ On Wed, Dec 29, 2021 at 6:26 AM Amir Sarabadani ladsgroup@gmail.com
wrote:
I'm not debating your note. It is very valid that we lack proper
support for multimedia stack. I myself wrote a detailed rant on how broken it is [1] but three notes:
- Fixing something like this takes time, you need to assign the budget
for it (which means it has to be done during the annual planning) and if gets approved, you need to start it with the fiscal year (meaning July 2022) and then hire (meaning, write JD, do recruitment, interview lots of people, get them hired) which can take from several months to years. Once they are hired, you need to onboard them and let them learn about our technical infrastructure which takes at least two good months. Software engineering is not magic, it takes time, blood and sweat. [2]
- Making another team focus on multimedia requires changes in
planning, budget, OKR, etc. etc. Are we sure moving the focus of teams is a good idea? Most teams are already focusing on vital parts of wikimedia and changing the focus will turn this into a whack-a-mole game where we fix multimedia but now we have critical issues in security or performance.
- Voting Wishlist survey is a good band-aid in the meantime. To at
least address the worst parts for now.
I don't understand your point tbh, either you think it's a good idea to
make requests for improvements in multimedia in the wishlist survey or you think it's not. If you think it's not, then it's offtopic to this thread.
[1]
https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/...
[2] There is a classic book in this topic called "The Mythical
Man-month"
On Wed, Dec 29, 2021 at 11:41 AM Gnangarra gnangarra@gmail.com wrote:
we have to vote for regular maintenance and support for
essential functions like uploading files which is the core mission of Wikimedia Commons _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
On Sat, 2022-01-01 at 07:41 +0100, Amir Sarabadani wrote:
There is "code stewardship program" and its goal is to find owners for components that don't have an owner (or undeploy them). Sometimes it's successful, sometimes it's not. I have been asking for a maintainer for FlaggedRevs for four years now, beta cluster is suffering from a similar situation, etc. etc. It takes time to find an owner for everything, to fill the gaps in places we don't have a team to handle those (e.g. Multimedia, you can't just hand over that to team responsible for security for example). More info at https://www.mediawiki.org/wiki/Code_stewardship_reviews
My very personal impression is that the Code Stewardship Review process currently lacks decision authority and senior leadership support within the WMF. It's not an engineering problem but a business problem. Decisions themselves can sometimes be easy. Making them happen not so.
andre
wikitech-l@lists.wikimedia.org