Hello,
On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit offline pending investigation.
The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating this incident as well as potential improvements to prevent future incidents. More information will be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it becomes available and is confirmed. If you have any questions, please contact the Security (security@wikimedia.org trustandsafety@wikimedia.org ).
Thanks
Hello,
Gerrit is available again but we are continuing to investigate the suspicious activity. Our preliminary findings point to no users or production systems being compromised and no loss of any confidential information. As we continue to investigate over the next few days we will add any appropriate updates to the phabricator task ( https://phabricator.wikimedia.org/T218472 ) .
Thanks
On Sat, Mar 16, 2019 at 10:26 AM John Bennett jbennett@wikimedia.org wrote:
Hello,
On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit offline pending investigation.
The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating this incident as well as potential improvements to prevent future incidents. More information will be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it becomes available and is confirmed. If you have any questions, please contact the Security (security@wikimedia.org trustandsafety@wikimedia.org).
Thanks
Thanks for the updates and for everyone who was or is working on a weekend day. Sometime in the next few weeks if you can publish an incident report that has any sensitive information redacted, I would like to read it.
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
On Sat, Mar 16, 2019, 12:25 PM John Bennett jbennett@wikimedia.org wrote:
Hello,
Gerrit is available again but we are continuing to investigate the suspicious activity. Our preliminary findings point to no users or production systems being compromised and no loss of any confidential information. As we continue to investigate over the next few days we will add any appropriate updates to the phabricator task ( https://phabricator.wikimedia.org/T218472 ) .
Thanks
On Sat, Mar 16, 2019 at 10:26 AM John Bennett jbennett@wikimedia.org wrote:
Hello,
On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit
offline
pending investigation.
The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating this incident as well as potential improvements to prevent future incidents. More information will be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it becomes available and is confirmed. If you have any questions, please contact the Security (security@wikimedia.org trustandsafety@wikimedia.org).
Thanks
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hello,
after the Gerrit outage, my list of watched changes is seemingly empty, while it has not been a day before. Is it possible to fix this issue?
MGChecker
-----Ursprüngliche Nachricht----- Von: Wikitech-l [mailto:wikitech-l-bounces@lists.wikimedia.org] Im Auftrag von Pine W Gesendet: Samstag, 16. März 2019 21:14 An: Wikimedia developers Betreff: Re: [Wikitech-l] Gerrit outage
Thanks for the updates and for everyone who was or is working on a weekend day. Sometime in the next few weeks if you can publish an incident report that has any sensitive information redacted, I would like to read it.
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
On Sat, Mar 16, 2019, 12:25 PM John Bennett jbennett@wikimedia.org wrote:
Hello,
Gerrit is available again but we are continuing to investigate the suspicious activity. Our preliminary findings point to no users or production systems being compromised and no loss of any confidential information. As we continue to investigate over the next few days we will add any appropriate updates to the phabricator task ( https://phabricator.wikimedia.org/T218472 ) .
Thanks
On Sat, Mar 16, 2019 at 10:26 AM John Bennett jbennett@wikimedia.org wrote:
Hello,
On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit
offline
pending investigation.
The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating this incident as well as potential improvements to prevent future incidents. More information will be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it becomes available and is confirmed. If you have any questions, please contact the Security (security@wikimedia.org trustandsafety@wikimedia.org).
Thanks
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
The watchlist should be in All-Users. If someone removed your watchlist you can clone that repo. Then in .git/config change refs/heads to just refs/*. Then it should show some refs (I forget which one you check out so you should try them all) you can use git log to see if someone git committed the removal.
Hello,
Today we have seen Phabricator vandalism from an attacker who was also responsible for the Gerrit outage yesterday. I’d like to clarify a comment I made yesterday and provide as many additional details as I can while still maintaining operational security.
While no user accounts were compromised the attacker leveraged a vulnerability in Gerrit to comprise a single staff account. This discovery is what lead to taking Gerrit offline so an investigation could occur, the vulnerability could be remediated and the service restored. However, no further evidence of compromise was discovered and additional security controls prevented malicious activities from being executed using the compromised staff account. We will continue to monitor the situation and will provide updates on this list and on the Phabricator task https://phabricator.wikimedia.org/T218472.
Thanks
John
On Sat, Mar 16, 2019 at 2:25 PM John Bennett jbennett@wikimedia.org wrote:
Hello,
Gerrit is available again but we are continuing to investigate the suspicious activity. Our preliminary findings point to no users or production systems being compromised and no loss of any confidential information. As we continue to investigate over the next few days we will add any appropriate updates to the phabricator task ( https://phabricator.wikimedia.org/T218472 ) .
Thanks
On Sat, Mar 16, 2019 at 10:26 AM John Bennett jbennett@wikimedia.org wrote:
Hello,
On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit offline pending investigation.
The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating this incident as well as potential improvements to prevent future incidents. More information will be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it becomes available and is confirmed. If you have any questions, please contact the Security (security@wikimedia.org trustandsafety@wikimedia.org).
Thanks
Hello,
As part of cleanup and response Gerrit's use of http tokens has been disabled. You should still be able to use the http REST api using your LDAP password.
Gerrit's command-line tools [0] that operate via SSH are also still available.
-- Tyler
[0]. https://gerrit.wikimedia.org/r/Documentation/cmd-index.html
On 19-03-16 10:26:52, John Bennett wrote:
Hello,
On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit offline pending investigation.
The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating this incident as well as potential improvements to prevent future incidents. More information will be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it becomes available and is confirmed. If you have any questions, please contact the Security (security@wikimedia.org trustandsafety@wikimedia.org ).
Thanks _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, 2019-03-19 at 10:59 +0100, planetenxin wrote:
Gerrit seems to be offline again.
Please read the other latest thread on this very mailing list.
andre
Not everyone is aware that the process of cleaning up the vandalism/fixing Gerrit includes Gerrit being down temporarily.
Do I need to include a reminder link to WP:AGF / WP:DICK?
-- Lewis Cawte
On Tue, 19 Mar 2019, 10:27 Andre Klapper, aklapper@wikimedia.org wrote:
On Tue, 2019-03-19 at 10:59 +0100, planetenxin wrote:
Gerrit seems to be offline again.
Please read the other latest thread on this very mailing list.
andre
Andre Klapper | Bugwrangler / Developer Advocate https://blogs.gnome.org/aklapper/
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, 19 Mar 2019 at 10:50, Lewis Cawte via Wikitech-l < wikitech-l@lists.wikimedia.org> wrote:
Not everyone is aware that the process of cleaning up the vandalism/fixing Gerrit includes Gerrit being down temporarily.
Do I need to include a reminder link to WP:AGF / WP:DICK?
That would not be helpful. By assuming that the other person simply missed the other thread on this mailing list, and by pointing the person to said thread, Andre *is* assuming good faith.
Dan
On Tue, 2019-03-19 at 10:49 +0000, Lewis Cawte via Wikitech-l wrote:
Not everyone is aware that the process of cleaning up the vandalism/fixing Gerrit includes Gerrit being down temporarily.
Right. Should have spent more time to rephrase and explicitly say so. Thanks for pointing that out.
planetenxin: Sorry for my previous message, was not meant to be rude.
andre
Gerrit is back up. Almost all of the vandalism has been cleaned up, some minor stuff remains, we will clean that up as well.
On Tue, Mar 19, 2019 at 1:42 PM planetenxin planetenxin@web.de wrote:
Am 19.03.2019 um 12:21 schrieb Andre Klapper:
planetenxin: Sorry for my previous message, was not meant to be rude.
no worries. Hope, that Gerrit is back alive soon. :-)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks to everyone who helped sort this out.
In some ways, the vandalism neatly demonstrates how Wikimedia projects rely on trust. When these things happen, it is a nice reminder that our open values mean that we should take a light approach to security whenever the potential exposure is always going to be recoverable. Resilience rather than impenetrable, for our community at least, is a healthy way to prioritize. The occasional predictable idiot is no reason to change that approach.
Cheers, Fae
On Tue, 19 Mar 2019 at 13:28, Alexandros Kosiaris akosiaris@wikimedia.org wrote:
Gerrit is back up. Almost all of the vandalism has been cleaned up, some minor stuff remains, we will clean that up as well.
On Tue, Mar 19, 2019 at 1:42 PM planetenxin planetenxin@web.de wrote:
Am 19.03.2019 um 12:21 schrieb Andre Klapper:
planetenxin: Sorry for my previous message, was not meant to be rude.
no worries. Hope, that Gerrit is back alive soon. :-)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hello Fæ,
While I understand and agree with your point, I must point out that this 4 days have been hectic on many people from multiple teams. The amount of work to cleanup one person's destructive half hour spree is staggering. We need better tooling for sure to combat this, something that while MediaWiki is already equipped with, some of the infrastructure tools are not (yet hopefully). It saddens me greatly to say that, but we might have to take some steps in the opposite direction, for a while at least, until we are in shape to combat this more effectively.
Regards,
On Tue, Mar 19, 2019 at 3:40 PM Fæ faewik@gmail.com wrote:
Thanks to everyone who helped sort this out.
In some ways, the vandalism neatly demonstrates how Wikimedia projects rely on trust. When these things happen, it is a nice reminder that our open values mean that we should take a light approach to security whenever the potential exposure is always going to be recoverable. Resilience rather than impenetrable, for our community at least, is a healthy way to prioritize. The occasional predictable idiot is no reason to change that approach.
Cheers, Fae
On Tue, 19 Mar 2019 at 13:28, Alexandros Kosiaris akosiaris@wikimedia.org wrote:
Gerrit is back up. Almost all of the vandalism has been cleaned up, some minor stuff remains, we will clean that up as well.
On Tue, Mar 19, 2019 at 1:42 PM planetenxin planetenxin@web.de wrote:
Am 19.03.2019 um 12:21 schrieb Andre Klapper:
planetenxin: Sorry for my previous message, was not meant to be rude.
no worries. Hope, that Gerrit is back alive soon. :-)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- faewik@gmail.com https://commons.wikimedia.org/wiki/User:Fae
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I'd like to give a quick thanks to folks who have been dealing with turbulence.
I think that short term mitigation measures sound reasonable, while longer term improvements are planned and developed.
wikitech-l@lists.wikimedia.org