On 4/8/25 20:18, Taavi Väänänen wrote:

Apologies in advance for a possibly ranty email.

I'm again seeing some major Toolforge developments happening with major decisions being made in private. A few examples:

• Infrastructure IaC/OpenTofu migration (with an intern assigned to it) popping

out of nowhere to implementation tasks https://phabricator.wikimedia.org/T390056 with no public/formal discussion whether we want to replace the current tooling with that.

I can speak at least about this point, mentioning that:

* We discussed "Potential for tracking tools/toolsbeta Cloud VPS resources via opentofu" [0], right at the time when the mentioned internship was originally being planned * We discussed the options to integrate tools/toolsbeta into the tofu-infra repository, and shared the outcome of that discussion [1], and additional related decisions [2] * Finally, there have been publicly available tickets [3][4] about the ongoing work as it is getting materialized

The timeline of the different linked resources is: * Oct 2024, Nov 2024, [..], Feb 2025, March 2025 * (note I was on sick leave between Dec 2024 and Jan 2025)

regards.

[0] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Monthly_meeting/2... [1] https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/... [2] https://phabricator.wikimedia.org/T385604 [3] https://phabricator.wikimedia.org/T389081 [4] https://phabricator.wikimedia.org/T390056

Hey Taavi,

Sorry for the late reply, I'm still on sick leave, but I thought that your concerns are serious enough for me to try to clarify some things and try to avoid the misunderstandings from growing.

• UI project starting an "investigation" task

(https://phabricator.wikimedia.org/T383146) with a clear bias towards a specific solution, based on unspecific "previous discussions", which were then hidden from the description when I asked for documents relating those to be made public.

That task came out of a sync meeting between Sarai, Slavina, and me (if I remember correctly). The goal of the meeting was for Sarai to ask as many questions as she had about Toolforge and the misnamed "Toolforge UI" project (which is actually the same as Striker - though that wasn’t clear to her at the time, and I’m sure many of us can relate).

None of us were familiar with Striker. The only experience we had was trying (and failing) to set it up locally a couple of times, and hearing from you and Bryan that it needs upgrading, is deployed in production, and depends on several external services. All of those are red flags for complexity.

That’s why we created the "investigation" task - to revisit it later with the whole team. Naturally, given our lack of experience with the codebase, some of our initial assumptions turned out to be wrong - but that’s totally fine.

Then we rephrased the task to include and address some of the comments you and Bryan made, make it less biased, and more accurate ( https://phabricator.wikimedia.org/T383146#10440408).

During that change, the document that was attached was summarized and moved to a public wiki page, as was always the intention, though it seems that we forgot to make it open anyhow (I think that has been resolved now, probably missed the ping in the task, in case you are curious https://docs.google.com/document/d/1zpCks7sF7N5ttROFTAJGB3FoXzg_Bp4IgYMiG0A1... , it's very annoying that the default for google docs is not public).

In any case, that task is still there, as a reminder that we should give it a look as preparation for any UI work, that unfortunately might still take some time to start as we probably will not get resources for the next FY for it so there's no need to address it yet.

Regarding the public work on Toolforge UI, it's been quite widely shared across various channels (Phabricator, wiki, wikitech, mailing list, etc.) and has been discussed in almost every check-in and Toolforge monthly. But please do let us know if it didn’t make it to you.

The Toolforge CLI consolidation project apparently has switched

implementation languages compared to the last decision request, apparently based on "a team decision" that was apparently made "a few months" ago but never communicated anywhere as far as I can tell: https://phabricator.wikimedia.org/T356262#10722410

The CLI consolidation project has not switched languages, the direction is still going with python first, but there has been some developments that might make us want to switch to golang in the long run as we discussed in the decision task (see comments in the task https://phabricator.wikimedia.org/T348749).

While working on the consolidation, there were two big questions raised during the decision meeting to be explored when implementing whatever we decided: * Packaging for distribution, so users can run on their laptops/etc. - long term goal * How good is the code generation for each language, model validation, functionality out of the box, etc.

For those we created two tasks, one for each: * https://phabricator.wikimedia.org/T356261 - code generation * https://phabricator.wikimedia.org/T369693 - packaging for distribution

We started playing with the packaging for distribution one, and it proved quite difficult with python, the initial issues were not solvable, to the point of becoming a blocker, discarding python as the language for distributing the binaries. Golang being the next candidate (iirc this was done in July/Aug 2024). Then we started focusing on the push to deploy instead and left the cli work for later.

At the beginning of the year, there was the possibility of getting Adam to do a bit of work with us for some time too, so we came up with trying to get a POC of the code generation and distribution in golang to figure out if it's feasible (and avoid finding out later that it's too complex). This was briefly discussed in the monthly meetings and in the check-ins iirc

In the meantime, Adam has been working for some time on the generated cli POC, and I think it was last week when he officially started, so that's why the code is being shared now (note that I've been on sick leave from before, so I'm a bit out of sync).

I have not yet reviewed the code, but it seems pretty good so far, so next steps here is to review what we have learnt from both python and golang and adapt if needed our current course of action (that is, create a decision task with all the new info, and discuss next steps if any). As a treat, Adam also made the POC work from your laptop, not using the APIs directly though but a proxy I think. I think it's way more progress than we anticipated :)

First, could we please again stop doing major decisions in private?

Answering this, there's no decisions made in private anywhere, so yes, we can, and we have stopped doing major decisions in private.

Second, is there something in our processes or workflows that we could

improve so that things wouldn't always immediately start going more hidden and private every time I stop paying close attention to this?

Definitely! There's always room for improvement. One huge point imo would be to make sure that google docs are open by default, as that was the only thing that was not open this time.

There is a tradeoff happening here too about communication, it's unreasonable to try to communicate exclusively using a single medium (ex. only asynchronous written), there's always going to be other forms like sync meetings, chats, emails, ... and preserving/spreading information from one to the other is going to be lossy and sometimes inaccurate (given also that there's more people with less context too). So for that, the only solution I know of is to ask, don't assume, just please ask for clarification when it's not clear :)