Continuing from my post on cloud@...
On Thu, Oct 6, 2022 at 6:21 PM Bryan Davis bd808@wikimedia.org wrote:
On Thu, Oct 6, 2022 at 5:39 AM Taavi Väänänen hi@taavi.wtf wrote:
In general, I feel that over the last few months, quite a lot of planning and progress reporting has moved from our various public channels (most notably Phabricator and -cloud-admin on IRC) to private ones. I don't particularly like this trend.
I did a thing in my late afternoon yesterday that may have aggravated Tavvi's feelings of being left out of decision loops.
I made a decision without consulting any other Toolforge admins to add about 300MiB of fonts to the php7.4 Docker image available for use on Toolforge [0]. This decision reversed my prior blocking of this exact same request in 2019 [1]. It also goes against at least as many years of the Toolforge admins telling the Toolforge member community that we do not "bloat" the Kubernetes containers with specialty features for a small number of use cases. This reversal will complicate future decisions on such issues by introducing this easily seen counter example. I acted with good intent in the moment, but I did not act with good judgement nor consideration of my partners in maintaining the Toolforge infrastructure. For that I am truly sorry.
I would also like to apologize for treating what I was doing as "urgent" when it could have easily waited for a discussion with others either in code review or in other forums. This false urgency was counter to what I know to be the best way to treat technical decisions and it was disrespectful of my co-admins in the Toolforge environment.
I would also like to have a conversation among the Toolforge admins about how to best deal with this decision going forward. That conversation is probably better had on Phabricator or the cloud-admin mailing list than here, but it should happen and it should result in either reverting the change that I made or jointly creating updated guidelines for what is and is not acceptable in the shared Kubernetes containers while we await better methods of managing per-tool feature differences.
For the fonts themselves, should we: * Revert the change and tell svgtranslate to move back to the grid? * Propagate the change outward by making the same/similar change to all php images? * Propagate the change outward by making the same/similar change to all base images? * Let it be.
For the bigger picture of breaking our long held stance on "bloat", I would like to hear suggestions from y'all. If the font decision is to revert then maybe there is nothing to talk about here. If the fonts stay then I think there is a need to either document this as a rogue action that has been allowed to stand which should not set a precedent for the future or to come up with a rubric for what is allowed and why.
I am also open to hearing from anyone on or off list who feels that I need to make additional amends to the Toolforge admins, the Toolforge user community, or any particular individuals. I really didn't mean to make a mess, but I did and I would like to work towards correcting that as much as possible.
Bryan
PS I will be out of office until 2022-10-11, but I will try to check in on this thread in the intervening days.
On 10/7/22 03:43, Bryan Davis wrote:
For the fonts themselves, should we:
- Revert the change and tell svgtranslate to move back to the grid?
- Propagate the change outward by making the same/similar change to
all php images?
- Propagate the change outward by making the same/similar change to
all base images?
- Let it be.
First of all: I don't see anything specific on the PHP images, so if we're going to keep the fonts, I think we should move them to the base images.
For the bigger picture of breaking our long held stance on "bloat", I would like to hear suggestions from y'all. If the font decision is to revert then maybe there is nothing to talk about here. If the fonts stay then I think there is a need to either document this as a rogue action that has been allowed to stand which should not set a precedent for the future or to come up with a rubric for what is allowed and why.
My concern with the current form of Docker images has been for a while that once something has been added to it, it's pretty much impossible to remove that without breaking previously-working volunteer maintained projects (for example: Python2/3 used by webservice-runner). So I'd propose to revert the change while we work out what to do with it on the medium/long term.
I do think that fonts needed for rendering images falls into the category of "specialty features for a small number of use cases" that you mention in your cloud@ post. Long-term this feels like a great use case of a build pack based image, but as I mentioned already I don't have up to date information on the current vision and roadmap for that project.
I am also open to hearing from anyone on or off list who feels that I need to make additional amends to the Toolforge admins, the Toolforge user community, or any particular individuals. I really didn't mean to make a mess, but I did and I would like to work towards correcting that as much as possible.
Thank you. I don't have anything specific in mind, but in general I don't like things happening based on private Slack threads.
Taavi
On Fri, Oct 7, 2022 at 1:40 AM Taavi Väänänen hi@taavi.wtf wrote:
On 10/7/22 03:43, Bryan Davis wrote:
For the fonts themselves, should we:
- Revert the change and tell svgtranslate to move back to the grid?
- Propagate the change outward by making the same/similar change to
all php images?
- Propagate the change outward by making the same/similar change to
all base images?
- Let it be.
First of all: I don't see anything specific on the PHP images, so if we're going to keep the fonts, I think we should move them to the base images.
This would be the "Propagate the change outward by making the same/similar change to all base images" option from my list I think? I'm ok with moving the fonts to the base images. As legoktm pointed out in https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/... the PHP images have librsvg2-bin added which is a semi-obvious use case for more fonts, but there are certainly other ways that fonts might be useful.
For the bigger picture of breaking our long held stance on "bloat", I would like to hear suggestions from y'all.
My concern with the current form of Docker images has been for a while that once something has been added to it, it's pretty much impossible to remove that without breaking previously-working volunteer maintained projects (for example: Python2/3 used by webservice-runner).
This is inherent not only in the Docker images, but also in the Puppet manifests that we use and to a lesser but real extent the operating systems and versions that we use. Anything and everything that ends up on a bastion or exec node will be treated as a feature by someone. I don't say this to imply that you are wrong, just that this is a much larger issue than Docker images.
So I'd propose to revert the change while we work out what to do with it on the medium/long term.
This would be the "Revert the change and tell svgtranslate to move back to the grid" option from my initial list of next steps. I'm ok with this too. The hopefully medium term solution really should be support for custom containers for tools which is currently hoped to be provided by a buildpack system.
I do think that fonts needed for rendering images falls into the category of "specialty features for a small number of use cases" that you mention in your cloud@ post. Long-term this feels like a great use case of a build pack based image, but as I mentioned already I don't have up to date information on the current vision and roadmap for that project.
I agree that TTF font support is a niche concern among tools primarily affecting tools that somehow generate images. It is also a thing that buildpacks should help provide a more general solution for. I don't know a whole lot more than Tavvi does about how soon buildpacks will be ready for actual users to work with. I do know that David, Slavina, and Raymond are all working on that project, but I haven't seen any concrete timeline projections.
Offered not as an excuse, but as data, the uncertainty about when and how buildpacks will be available to Toolforge users was part of my internal debate that led to reversing my past bloat blocks on this same feature. I know that there is a strong push to get tools off of the job grid. I assumed that this push was at least partially responsible for the CommTech folks having moved svgtranslate from the grid to kubernetes again after trying previously in 2019. It felt shitty to tell them that they had to go back again because fonts would not be allowed until they could add them for themselves with a buildpack... someday. In hindsight however I should have given this as the answer.
Bryan
On 10/11/22 7:22 PM, Bryan Davis wrote:
On Fri, Oct 7, 2022 at 1:40 AM Taavi Väänänen hi@taavi.wtf wrote:
On 10/7/22 03:43, Bryan Davis wrote:
For the fonts themselves, should we:
- Revert the change and tell svgtranslate to move back to the grid?
- Propagate the change outward by making the same/similar change to
all php images?
- Propagate the change outward by making the same/similar change to
all base images?
- Let it be.
First of all: I don't see anything specific on the PHP images, so if we're going to keep the fonts, I think we should move them to the base images.
At this point I don't want to tell the svgtranslate people to move backwards, so this change needs to stay in place. If, as I suspect, it's easier to maintain things if the change is made in other images as well, then let's do that.
This choice is not meant to guide future policy, though. This particular horse is already out of the barn, but going forward the answer for new feature requests in the shared images should be 1) work around it 2) wait for build packs, or 3) get consensus among people on this mailing list. I would expect #3 to happen rarely if at all. I'm entirely supportive of Bryan's desire to unblock users (and expanding language support is especially close to my heart) but we're going to have to grit our teeth and tell people 'no' for a few more months.
This thread also makes it clear that we need to provide at least a shadow of a timeline for when users can expect to actually get buildpacks. Right now the consensus answer seems to be 'by July' but I can only hope that the delivery date will approach a bit sooner than that.
I hope that all makes sense! I'm happy for this conversation to continue but I wanted to make sure that Bryan and the svgtranslate folks don't remain in suspense forever :)
This would be the "Propagate the change outward by making the same/similar change to all base images" option from my list I think? I'm ok with moving the fonts to the base images. As legoktm pointed out in https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/... the PHP images have librsvg2-bin added which is a semi-obvious use case for more fonts, but there are certainly other ways that fonts might be useful.
For the bigger picture of breaking our long held stance on "bloat", I would like to hear suggestions from y'all.
My concern with the current form of Docker images has been for a while that once something has been added to it, it's pretty much impossible to remove that without breaking previously-working volunteer maintained projects (for example: Python2/3 used by webservice-runner).
This is inherent not only in the Docker images, but also in the Puppet manifests that we use and to a lesser but real extent the operating systems and versions that we use. Anything and everything that ends up on a bastion or exec node will be treated as a feature by someone. I don't say this to imply that you are wrong, just that this is a much larger issue than Docker images.
So I'd propose to revert the change while we work out what to do with it on the medium/long term.
This would be the "Revert the change and tell svgtranslate to move back to the grid" option from my initial list of next steps. I'm ok with this too. The hopefully medium term solution really should be support for custom containers for tools which is currently hoped to be provided by a buildpack system.
I do think that fonts needed for rendering images falls into the category of "specialty features for a small number of use cases" that you mention in your cloud@ post. Long-term this feels like a great use case of a build pack based image, but as I mentioned already I don't have up to date information on the current vision and roadmap for that project.
I agree that TTF font support is a niche concern among tools primarily affecting tools that somehow generate images. It is also a thing that buildpacks should help provide a more general solution for. I don't know a whole lot more than Tavvi does about how soon buildpacks will be ready for actual users to work with. I do know that David, Slavina, and Raymond are all working on that project, but I haven't seen any concrete timeline projections.
Offered not as an excuse, but as data, the uncertainty about when and how buildpacks will be available to Toolforge users was part of my internal debate that led to reversing my past bloat blocks on this same feature. I know that there is a strong push to get tools off of the job grid. I assumed that this push was at least partially responsible for the CommTech folks having moved svgtranslate from the grid to kubernetes again after trying previously in 2019. It felt shitty to tell them that they had to go back again because fonts would not be allowed until they could add them for themselves with a buildpack... someday. In hindsight however I should have given this as the answer.
Bryan
On Wed, Oct 26, 2022 at 8:41 PM Andrew Bogott abogott@wikimedia.org wrote:
At this point I don't want to tell the svgtranslate people to move backwards, so this change needs to stay in place. If, as I suspect, it's easier to maintain things if the change is made in other images as well, then let's do that.
I agree we should not tell the svgtranslate people to move back to the grid. I don't have a strong opinion on whether these fonts should be in the PHP images or in the base images.
This choice is not meant to guide future policy, though. This particular horse is already out of the barn, but going forward the answer for new feature requests in the shared images should be 1) work around it 2) wait for build packs, or 3) get consensus among people on this mailing list. I would expect #3 to happen rarely if at all.
+1 to this approach.
Francesco
cloud-admin@lists.wikimedia.org