Hi there!
If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email.
We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0].
You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS
Web Proxies section.
Examples of web proxy names: * accounts.wmflabs.org * glampipe.wmflabs.org * incubator.wmflabs.org
Full list can be seen in the Openstack Browser tool [1].
We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header.
This is the timeline this change will follow:
* 2020-04-01: this email, start collecting list of things that require XFF * 2020-04-07: start evaluating list of things that require XFF * 2020-04-15: introduce the change, with proper case whitelisting
When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header.
Please reach out for any questions or comments.
regards.
[0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098
Hi there,
I was wondering if you were planning on exposing some kind of rate-limiting option for the web proxies in horizon? I'm thinking this will effectively mean no more rate-limiting per remote address at the instance level. Every once in a while, our project gets hammered by script kiddies and our application service gets brought down. I've gone ahead and implemented rate limiting in nginx that has a very high limit set across all ip addresses that should basically work, but typically I would set the limits to be per-client-ip to the extent allowed by the practicalities of NAT. This is not a blocker in any way for us, and I'd rather make do with less user info wherever possible.
Thanks for making the development of tools and services for Wikimedia as painless as possible!
On Wed, Apr 1, 2020 at 8:31 AM Arturo Borrero Gonzalez < aborrero@wikimedia.org> wrote:
Hi there!
If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email.
We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0].
You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS
Web Proxies section.
Examples of web proxy names:
- accounts.wmflabs.org
- glampipe.wmflabs.org
- incubator.wmflabs.org
Full list can be seen in the Openstack Browser tool [1].
We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header.
This is the timeline this change will follow:
- 2020-04-01: this email, start collecting list of things that require XFF
- 2020-04-07: start evaluating list of things that require XFF
- 2020-04-15: introduce the change, with proper case whitelisting
When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header.
Please reach out for any questions or comments.
regards.
[0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098
-- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On 4/14/20 6:25 PM, Jason Sherman wrote:
Hi there,
I was wondering if you were planning on exposing some kind of rate-limiting option for the web proxies in horizon? I'm thinking this will effectively mean no more rate-limiting per remote address at the instance level. Every once in a while, our project gets hammered by script kiddies and our application service gets brought down. I've gone ahead and implemented rate limiting in nginx that has a very high limit set across all ip addresses that should basically work, but typically I would set the limits to be per-client-ip to the extent allowed by the practicalities of NAT. This is not a blocker in any way for us, and I'd rather make do with less user info wherever possible.
Hi there!
What you did seems correct to me, that is, implementing the controls on your own servers.
That being said, I understand your concern. We have mechanisms in place for banning concrete abusers. If we detected a more wide-spread problems we could introduce other mechanisms and controls to ensure service availability.
Should you detect someone is hammering your servers in CloudVPS, please contact us.
regards.
On 4/1/20 2:16 PM, Arturo Borrero Gonzalez wrote:
Hi there!
If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email.
We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0].
You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS
Web Proxies section.
Examples of web proxy names:
- accounts.wmflabs.org
- glampipe.wmflabs.org
- incubator.wmflabs.org
Full list can be seen in the Openstack Browser tool [1].
We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header.
This is the timeline this change will follow:
- 2020-04-01: this email, start collecting list of things that require XFF
- 2020-04-07: start evaluating list of things that require XFF
- 2020-04-15: introduce the change, with proper case whitelisting
When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header.
Please reach out for any questions or comments.
regards.
[0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098
Hi there!
This change is being applied now!
regards.
Hello, thanks for doing great job.
Do you have plan to enable HTTPS per default for web proxy names?
Best regards, Zoran Dori volunteer, Wikimedia Serbia s: zoranzoki21.github.io e: zorandori4444@gmail.com
On Wed, Apr 15, 2020 at 3:55 AM Zoran Dori zorandori4444@gmail.com wrote:
Hello, thanks for doing great job.
Do you have plan to enable HTTPS per default for web proxy names?
That is something that has been on my "remember to do this" list for a long time now. We should do it. :)
We added http->https enforcement for tools.wmflabs.org in January of 2019 [0] with no serious problems that I am aware of. Doing the same for the Cloud VPS shared proxy is technically just a matter of making a small configuration change. The more time consuming issue is making the proper announcements and helping folks deal with things that break as a result.
[0]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmflab...
Bryan