Hello, after the Gerrit outage, my list of watched changes is seemingly empty, while it has not been a day before. Is it possible to fix this issue? MGChecker -----Ursprüngliche Nachricht----- Von: Wikitech-l [mailto:wikitech-l-bounces@lists.wikimedia.org] Im Auftrag von Pine W Gesendet: Samstag, 16. März 2019 21:14 An: Wikimedia developers Betreff: Re: [Wikitech-l] Gerrit outage Thanks for the updates and for everyone who was or is working on a weekend day. Sometime in the next few weeks if you can publish an incident report that has any sensitive information redacted, I would like to read it. Pine ( https://meta.wikimedia.org/wiki/User:Pine ) On Sat, Mar 16, 2019, 12:25 PM John Bennett <jbennett(a)wikimedia.org> wrote: _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Hello, Today we have seen Phabricator vandalism from an attacker who was also responsible for the Gerrit outage yesterday. I’d like to clarify a comment I made yesterday and provide as many additional details as I can while still maintaining operational security. While no user accounts were compromised the attacker leveraged a vulnerability in Gerrit to comprise a single staff account. This discovery is what lead to taking Gerrit offline so an investigation could occur, the vulnerability could be remediated and the service restored. However, no further evidence of compromise was discovered and additional security controls prevented malicious activities from being executed using the compromised staff account. We will continue to monitor the situation and will provide updates on this list and on the Phabricator task https://phabricator.wikimedia.org/T218472. Thanks John On Sat, Mar 16, 2019 at 2:25 PM John Bennett <jbennett(a)wikimedia.org> wrote:

Not everyone is aware that the process of cleaning up the vandalism/fixing Gerrit includes Gerrit being down temporarily. Do I need to include a reminder link to WP:AGF / WP:DICK? -- Lewis Cawte On Tue, 19 Mar 2019, 10:27 Andre Klapper, <aklapper(a)wikimedia.org> wrote:

On Tue, 2019-03-19 at 10:49 +0000, Lewis Cawte via Wikitech-l wrote: Right. Should have spent more time to rephrase and explicitly say so. Thanks for pointing that out. planetenxin: Sorry for my previous message, was not meant to be rude. andre -- Andre Klapper | Bugwrangler / Developer Advocate https://blogs.gnome.org/aklapper/

Gerrit is back up. Almost all of the vandalism has been cleaned up, some minor stuff remains, we will clean that up as well. On Tue, Mar 19, 2019 at 1:42 PM planetenxin <planetenxin(a)web.de> wrote: -- Alexandros Kosiaris Senior Site Reliability Engineer Wikimedia Foundation

Hello Fæ, While I understand and agree with your point, I must point out that this 4 days have been hectic on many people from multiple teams. The amount of work to cleanup one person's destructive half hour spree is staggering. We need better tooling for sure to combat this, something that while MediaWiki is already equipped with, some of the infrastructure tools are not (yet hopefully). It saddens me greatly to say that, but we might have to take some steps in the opposite direction, for a while at least, until we are in shape to combat this more effectively. Regards, On Tue, Mar 19, 2019 at 3:40 PM Fæ <faewik(a)gmail.com> wrote: -- Alexandros Kosiaris Senior Site Reliability Engineer Wikimedia Foundation

On Wed, Mar 20, 2019 at 06:56:19PM +0000, Pine W wrote: Me too. It's inspirational and awe-inspiring to see people work on this.