This is a heads-up that we are planning to replace the host keys
for the Gerrit SSH server at gerrit.wikimedia.org:29418.
The change is planned for Tuesday, July 14th in the PDT
morning right after the MediaWiki train, that's around 11:00 UTC.
(https://wikitech.wikimedia.org/wiki/Deployments#Tuesday,_July_14)
The RSA key will be replaced with a longer version and additionally we
will start to offer ecdsa_256, ecdsa_384, ecdsa_521 and ed25519.
The service will not be RSA-only anymore which some users had already
reported as an issue.
After the change on Gerrit, your git / git-review / direct ssh
commands are expected to fail with errors about mismatched or changed
host keys or host identification.
This is expected.
You will need to remove the old, no longer used host key, and verify
the new one.
To remove the old host key, follow the instructions on screen or
consult the manual of your SSH software. Once that is done, retry the
command, and you'll be prompted to verify the new host key.
You can find the new keys for verification in this email below and on
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.…
If they match, confirm, and your command should continue. Once you
have successfully updated the host key you should no longer see any
errors.
If you are running any bots talking to gerrit-ssh please also update
their configuration accordingly and restart where needed.
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.…
ssh_host_rsa_key
2048 SHA256:j9/pXXc9WzjQwYP0t7nlzqH9EBOTw6q7DgcfnamJtsY
gerrit-code-review(a)gerrit1001.wikimedia.org (RSA)
ssh_host_ecdsa_256_key
256 SHA256:58swSiByT+4LVqs30/FqJpEPj+Mwjtn3WJY5hitlEgM
gerrit-code-review(a)gerrit1001.wikimedia.org (ECDSA)
ssh_host_ecdsa_384_key
384 SHA256:vFEVzNGuagPmYiw9EIwBStzd0X+gtprZzOi8vbLxAfc
gerrit-code-review(a)gerrit1001.wikimedia.org (ECDSA)
ssh_host_ecdsa_521_key
521 SHA256:OWb1uenhapK7AFPfEB+NRxgfxhktZ1Q6C5eCy+VbgsY
gerrit-code-review(a)gerrit1001.wikimedia.org (ECDSA)
ssh_host_ed25519_key
256 SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8
gerrit-code-review(a)gerrit1001.wikimedia.org (ED25519)
--
Daniel Zahn <dzahn(a)wikimedia.org>
Operations Engineer
Apologies for the cross-post, but doing so because the thread was
forwarded, also apologies for the length.
On Wed, Jul 8, 2020 at 5:01 PM Maarten Dammers <maarten(a)mdammers.nl> wrote:
> Of interest to the wider community. I really hope this is not part of a
> larger pattern of the WMF ignoring community.
>
>
"Never attribute to malice that which is adequately explained by
stupidity." [1]
In this case, my own stupidity (I'm the new CTO here at the WMF, for
context), or perhaps to be a little kinder to myself, a combination of bias
and naivety: my engineering bias towards wanting to solve a problem I felt
was important to take action on (context provided shortly) which got in the
way of taking a user-centric approach first in trying to understand what
the needs and wants are of the people using the system. As I said on the
talk page, I mistakenly thought that the main feedback loops would be about
porting workflows and not about the tool itself.
Even though many have publicly said that moving from Gerrit might still be
the right decision, how we go about deciding that is just as important as
what we do and I messed that up. Given that perspective, I've asked the
team to pause with moving forward on changes to our Code Review (CR) tools
and to begin a consultation that includes the option of sticking with what
we have for CR. I've also asked my team to update some of our decision
making processes relative to topics like this to make sure we properly hear
from stakeholders (e.g. in this case, both staff developers and our broader
community of developers) along the way.
For some more context, if it is helpful:
I'm ~11 months in here and still learning every day. While I've worked in
open source for a long time, this community is new to me and different
enough that I have and continue to need to update and adjust the way I
think and the way I direct my teams to do their work.
Coming in and talking with our tech teams and folks in the community, I see
a few themes that have emerged that contributed to me wanting to move
forward faster on this decision:
1. We have a lot of tech debt[2]. In many cases, I think software,
especially software that is successful, can collapse under its own weight
if people are not careful in servicing that tech debt. The work required
to both maintain existing infrastructure, products and services while at
the same time improving what we offer is a delicate balancing act. At our
scale, there is a significant and justified bias towards production, but it
has come at a cost that has compounded over the years and has a very real
human toll. Much of this debt was created because we had to invent things
that didn't exist. Now some of those things do exist and we should check
to see whether we can replace those older, albeit well-understood-by-us
systems, with newer ones that have become standards or best in class and
are still in line with our open source values.
2. The tech debt and the sheer number of services we support (many of which
aren't fully maintained[3]) is compounded by the scale at which we support
them. The result is that a number of people, especially those on the front
line of caring for that software, are either burnt out, or approaching that
point. A global pandemic hasn't helped. I view much of my role here early
on as one of trying to help somehow reduce that burnout. Modernizing and
upgrading our processes and toolchains can, I think, help fight this, even
if there is some short term pain in the shift.
All that being said, in this particular case, we have a team of people who
work on maintaining our CI (Continuous Integration) and CR systems who have
long been looking at replacing our CI system. This system runs on an
end-of-lifed version of Python and on an end-of-lifed version of Zuul, and
it’s critical we correct this since end-of-lifed software doesn’t receive
security updates. This is primarily behind the scenes work that most people
don't have to think about. There is also a growing sense of desire by some
of our developers to adopt more mainstream, well understood toolchains like
Gitlab/Github for development, combined with my own view that CI/CR is
*not* somewhere we should be deviating from broad industry norms on
ourselves and that we should adopt workflows that are (de facto) standards
(e.g. Gitlab/Github, with Gitlab being the open one of the two) amongst
developers irrespective of their backgrounds. Those two things led to my
biased thinking that it was obvious it needed to be changed and that the
primary feedback needed would therefore be on the workflows, not the tool
itself.
While I still think it needs to be changed, I completely missed, as I said
above, the stakeholder angle here and basic community laws of not
surprising people. For that I apologize. We are now working to correct
this, even if it means it's going to take longer or we end up sticking with
the status quo on CR.
Thanks,
Grant
[1] https://en.wikipedia.org/wiki/Hanlon%27s_razor
[2] https://en.wikipedia.org/wiki/Technical_debt
[3] https://www.mediawiki.org/wiki/Developers/Maintainers
> Maarten
>
> -------- Forwarded Message --------
> Subject: Re: [Wikitech-l] CI and Code Review
> Date: Wed, 8 Jul 2020 22:40:38 +0200
> From: Maarten Dammers <maarten(a)mdammers.nl>
> Reply-To: For developers discussing technical aspects and
> organization
> of Wikimedia projects <wikitech-l(a)lists.wikimedia.org>
> To: wikitech-l(a)lists.wikimedia.org
>
>
>
> Hi Greg,
>
> On 06-07-2020 19:39, Greg Grossmeier wrote:
> > First, apologies for not announcing this last week. A short work week
> > coupled with a new fiscal year delayed this until today.
> >
> > tl;dr: Wikimedia will be moving to a self-hosted (in our datacenter(s))
> > GitLab Community Edition (CE) installation for both code review and
> > continuous integration (CI).
>
> tl;dr: WMF decides to do a major change without any community
> consultation. Community members are upset.
> More at https://www.mediawiki.org/wiki/Topic:Vpbt50rwxgb2r6qn
>
> Maarten
>
>
Hey all,
TLDR: your extension CI might break due to hard deprecation of Revision
class.
Today a Gerrit change[1] has been merged, hard deprecating MediaWiki core
Revision class.
Loads of work has been done to prepare for this moment, mostly by a
volunteer developer DannyS712 with code review
support from almost every corner of the MediaWiki developer universe. You
can judge the amount of work by the number
of subtasks in the tracking ticket[2]
A lot of effort has been done to update all the WMF deployed extensions and
some non-deployed ones to the new code,
In accordance with stable interface policy deprecation section[3]. However
due to the gravity of a change, we might have
missed some corner cases. Missed usages will not cause problems in
production since the only consequence of using hard
deprecated code is a log message, but some CI tests might start failing.
If you find your tests failing, please replace all the usages of the
Revision class according to 1.35 release notes,
and your tests should start passing again. In case you’re having troubles
doing it, please reach out to core platform team
and we will try to help.
This is a huge milestone in decoupling MediaWiki core! We understand that
this might cause some inconveniences, apologize
for them and are hoping we can help with resolving any issues.
Cheers.
Petr Pchelko
1. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/608845
2. https://phabricator.wikimedia.org/T246284
3. https://www.mediawiki.org/wiki/Stable_interface_policy#Deprecation
*Cross-posting to wikitech-l as it's a topic related to development.*
On Wed, 8 Jul 2020 at 23:01, Maarten Dammers <maarten(a)mdammers.nl> wrote:
> Of interest to the wider community. I really hope this is not part of a
> larger pattern of the WMF ignoring community.
>
> Maarten
I've had a great experience in the "Discussion Tools" (WMF's talk pages
reply tool) project. Feedback is properly documented, considered and worked
on. I recall having quite some input on the UI design and I'm happy that
resulted in a clean, focused UI.
That is an example to follow, but there's a lot to improve in other areas.
Although I favor GitLab, I was surprised that this introduction didn't have
a community round other than the developer feedback survey. I assume
the non-public results of the surcey justify this move and I'm happy
with it, so no complaints, but it came as a surprise to me that this is
happening as such discussions were promptly shut down on phabricator: I've
given this suggestion once in a related topic and rude comments told me
this is off-topic and basically to keep it to myself. The discussion ended
abruptly and the ticket was closed within a day.
There was another discussion <https://phabricator.wikimedia.org/T167547> closed
and declined after 4 comments from 3 participants in 12 minutes. This
discussion was later referred to as the official decision to avoid GitLab
:-)
And prior to the current announcement, an attempt was made to rename this
discussion to "RFC: ..."
I have a hard time to understand these actions and correlate it to any
consensus process and transparent communication.
The recent update to the history diff font
<https://phabricator.wikimedia.org/T250393> also could have been
communicated better. Informing editors about a simple solution (to change
your editor font setting or add some CSS to common.css/global.css) would
have avoided disruption and community backlash. Instead the latter solution
was only shared reactively after complaints and the simple solution was not
suggested.
I'd hope the developer team gives more attention to communication and
transparency. I think this has improved in the last year, primarily in the
preparation of Desktop Improvements and Talk pages project, but it's still
a long path to improve engagement generally with the community and in
particular development matters.
Aron (Demian)
Hello,
>From now on, new CI messages send to Gerrit can be filtered out in the
web interface by selecting the "Only comments" filter. That is done by
sending a review with '--tag autogenerated:ci' which get caught by the
filter.
That might help reducing the clutter on the change view, and in most
case the tests results you are interesting in are the latest which are
now displayed as a table just below the commit message (was T256575 done
by Paladox and QChris).
An example of the feature is at:
https://phabricator.wikimedia.org/T48148#6294913
Do note that old CI messages written for the last 9 years or so are not
tagged and can not be filtered out.
If you see any bot for which messages should be filtered, please report
them on https://phabricator.wikimedia.org/T48148 or reach out to the
bot author(s) to use a tag!
--
Antoine "hashar" Musso
On March 30, we announced that we were temporarily pushing back the release
of MediaWiki 1.35 due to uncertainty resulting from the COVID-19 pandemic.
We are now ready to begin the process of moving forward with the release.
The first step is cutting the release branch, REL1_35, which will occur on
Monday, July 13.
We will then be in "pencils down" mode: developers will stop targeting
MediaWiki 1.35 for new features. Instead, any new features would continue
to be applied to master and would target MediaWiki 1.36 or later. The only
work that would continue towards MediaWIki 1.35 would be blockers -
critical bug fixes or features close to completion that need to make it
into the release. This would happen by merging those patches into master
and then backporting them to the REL1_35 branch.
We anticipate that MediaWiki 1.35 will be released at the beginning of
August.
We appreciate your patience in these difficult times. Wishing you safety
and health,
Cindy
First, apologies for not announcing this last week. A short work week
coupled with a new fiscal year delayed this until today.
tl;dr: Wikimedia will be moving to a self-hosted (in our datacenter(s))
GitLab Community Edition (CE) installation for both code review and
continuous integration (CI).
Longer:
We are making a change to our code review and continuous integration (CI)
systems in response to a complex set of inputs (Developer Satisfaction
Survey[0], passing comments/critiques, an evaluation of replacement
continuous integration infrastructure[1], feedback from leaders in the
Foundation, etc) and evaluation conversations with Wikimedia Technology
department leadership (eg CTO, VPs) and representatives from Wikimedia SRE,
Architecture, Core Platform, Product, Security, and Technical Engagement
teams. In those conversations with Technology department leadership,
coordinated by our CTO Grant Ingersoll, we determined that an RFC was not
needed for this decision[2].
We plan to replace Gerrit and Zuul+Jenkins (the software that powers our CI
system) with GitLab. We hope that making a move to GitLab now will address
most of the concerns and desires that are able to be addressed via software
alone.
We join a growing list of other free/open knowledge organizations using a
self-hosted GitLab installation and look forward to the added benefits of
working together with them.
The project portal page lives at:
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/GitLab
It is a living documentation portal and we will continue to update it as we
go. The Talk: page is also available for your feedback and questions (and
is already being used). We hope everyone will join in to make this
transition as successful as possible.
Notably, I would like to point people to the conversation[3] about
evaluating pull-request style workflows and finding a recommended workflow
for our projects. While pull-request workflows are much more common in the
wider software development world{{cn}} we would like to provide as much
guidance as reasonable so that we are using our tools to the best of their
ability.
Here is the list of stakeholders as we have them now. In a RACI[4] model
these would be Consulted. These are the people the Wikimedia Release
Engineering team will be consulting with most closely as they get this
going.
* SRE/Service Ops - Mark and delegate(s)
* Security - Chase P and Scott B
* Core Platform Team - TBD
* Technical Engagement - TBD
* Product - Daniel C
“TBD” means we have asked for a representative and we’re waiting to hear
back on confirmation. We also have a few other non-WMF groups that we have
already reached out to or will be shortly to include in that list; feel
free to ping me with other suggestions.
What does this mean for you as you do your work today? Right now, nothing.
But as we set up the new GitLab installation we will be looking for early
adopters to give us feedback as we work on improvements. As we start to
migrate more users and repositories we will strive to help everyone as much
as possible and reasonable. It should go without saying that this includes
completely volunteer projects using the shared infrastructure.
The full timeline and plan will be posted to the above project portal page.
For the avoidance of doubt, this does not impact issue/task management;
that will remain in Phabricator.
Thank you,
Greg
PS: Please let me know where else this announcement should be sent.
[0] https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2020
[1]
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Future…
[2] see also:
https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Charter#Areas_…
[3] https://www.mediawiki.org/wiki/Topic:Vpbwawb4lkdy89ym
[4] https://en.wikipedia.org/wiki/Responsibility_assignment_matrix
Hi,
for HTML version see https://www.mediawiki.org/wiki/Scrum_of_scrums/2020-07-08
Željko
--
= 2020-07-08 =
== Callouts ==
* Release Engineering
** [All] Review guidance at [[wikitech:Deployments/Covid-19]] and Code
Deployment Office Hour at 17:00UTC in #wikimedia-office
** "scap sync" will be renamed to "scap sync-world" in the next
release. If you use "scap sync" non-interactively, please add a note
to: [[phab:T250302]] (and also, explain why you're using it)
** scap sync now has option --canary-wait-time; [[phab:T217924]]
== Product ==
=== iOS native app ===
* Updates:
** Bug fixes and feature task planning for 6.7 - [[phab:project/view/4661]]
** WWDC research
=== Android native app ===
* Updates:
** Continuing work on onboarding design refresh - [[phab:project/view/4819/]]
=== Web ===
* Updates:
** '''Summary''': content max-width and sidebar persistence for
Desktop Improvements Project (DIP), building button and input for
Vue.js search.
** [[Reading/Web/Desktop_Improvements|Desktop Improvements Project
(Vector / DIP)]]:
*** [[phab:T256992|<nowiki>[site request] enable 100% sampling for
sidebar instrumentation on test.wikipedia.org</nowiki>]]
*** [[phab:T254195|<nowiki>Implement a core 'clearfix' mixin in
mediawiki.mixin and evaluate deprecation/removal of 'visualClear'
class</nowiki>]]
*** [[phab:T246420|<nowiki>Limit content width, and refine alignment &
styling of relevant elements</nowiki>]]
*** [[phab:T256092|<nowiki>[Modern Vector] Fix broken rendering of
`main` and `dialog` elements in IE9-11</nowiki>]]
*** [[phab:T251212|<nowiki>[Dev] Drop VectorTemplate usage in Vector</nowiki>]]
*** [[phab:T255319|<nowiki>Eventually deprecate
SkinTemplateNavigation::SpecialPage and SkinTemplateNavigation hooks
in favor of SkinTemplateNavigation::Universal </nowiki>]]
*** [[phab:T244392|Vue.js search case study]]:
**** See [[Reading/Web/Desktop Improvements/Vue.js case study/Status
log|weekly status updates]].
** Mobile website (MinervaNeue / MobileFrontend):
*** [[phab:T235961|<nowiki>Remove 'mediawiki.ui.anchor' module</nowiki>]]
*** [[phab:T235712|<nowiki>Fix the most common "Module not loadable on
target mobile" warnings (Oct 2019)</nowiki>]]
*** [[phab:T247033|<nowiki>Add 'i18n-directionality.less' file to core
and extract overarching theme styles from legacy.less</nowiki>]]
** Standardization
*** [[phab:T256713|<nowiki>Amend VisualEditor's WikimediaUI themed
ProgressBar to new look</nowiki>]]
*** [[phab:T256520|<nowiki>Consider 'normalize' stylesheet RL module</nowiki>]]
** Portals
*** [[phab:T128546|<nowiki>[Recurring Task] Update Wikipedia and
sister projects portals statistics</nowiki>]]
** Miscellaneous
*** [[phab:T237036|<nowiki>ext.uls.interface should set targets and
explicitly not target the Minerva skin</nowiki>]]
== Technology ==
=== Fundraising Tech ===
* Updates:
** Figuring out what old data we can delete, and how
** Working out the kinks in the matching gifts sync / donation form
employer autocomplete
** More optimizing data sync between CiviCRM and bulk mail house
** CentralNotice workaround for browsers not letting pixels set banner
hide cookies:
=== Engineering Productivity ===
==== Release Engineering ====
* Updates:
** [All] Deployments/Covid-19 [[wikitech:Deployments/Covid-19]]
** Train Health
*** Last week: 1.35.0-wmf.39 - [[phab:T254176]]
*** This week: 1.35.0-wmf.40 - [[phab:T256668]]
*** Next week: 1.35.0-wmf.41 - [[phab:T256669]]