In a few hours a patch will be going out which limits the number of concurrent searches a single user will be able to make to 5. This applies to logged in and anonymous users. The failure message is `You have too many concurrent searches running. If you are sharing an IP address with other users you can log in to get your own limits.` for anonymous users, or just `You have too many concurrent searches running.` for logged in users. These are the `cirrussearch-too-busy-for-you-anonymous-error` and `cirrussearch-too-busy-for-you-logged-in-error` i18n messages.
I will be monitoring the logs when this goes out, and intermittently throughout the week as well. If necessary we will whitelist certain ip ranges that seem to be shared among large numbers of users.
Erik B.
While I understand the intent of conserving search resource use, will this change have adverse effects in situations like professors instructing their 200-student classes to search for a particular topic and its related articles on Wikipedia?
Pine On May 18, 2015 12:35 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
In a few hours a patch will be going out which limits the number of concurrent searches a single user will be able to make to 5. This applies to logged in and anonymous users. The failure message is `You have too many concurrent searches running. If you are sharing an IP address with other users you can log in to get your own limits.` for anonymous users, or just `You have too many concurrent searches running.` for logged in users. These are the `cirrussearch-too-busy-for-you-anonymous-error` and `cirrussearch-too-busy-for-you-logged-in-error` i18n messages.
I will be monitoring the logs when this goes out, and intermittently throughout the week as well. If necessary we will whitelist certain ip ranges that seem to be shared among large numbers of users.
Erik B. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This will bring such a idea to a screeching halt. Any kind of shared IPs will be unable to search
On Mon, May 18, 2015 at 7:54 PM, Pine W wiki.pine@gmail.com wrote:
While I understand the intent of conserving search resource use, will this change have adverse effects in situations like professors instructing their 200-student classes to search for a particular topic and its related articles on Wikipedia?
Pine On May 18, 2015 12:35 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
In a few hours a patch will be going out which limits the number of concurrent searches a single user will be able to make to 5. This
applies
to logged in and anonymous users. The failure message is `You have too many concurrent searches running. If you are sharing an IP address with other users you can log in to get your own limits.` for anonymous users,
or
just `You have too many concurrent searches running.` for logged in
users.
These are the `cirrussearch-too-busy-for-you-anonymous-error` and `cirrussearch-too-busy-for-you-logged-in-error` i18n messages.
I will be monitoring the logs when this goes out, and intermittently throughout the week as well. If necessary we will whitelist certain ip ranges that seem to be shared among large numbers of users.
Erik B. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I doubt all 200 students will be making concurrent searches.
On May 18, 2015, at 7:57 PM, John phoenixoverride@gmail.com wrote:
This will bring such a idea to a screeching halt. Any kind of shared IPs will be unable to search
On Mon, May 18, 2015 at 7:54 PM, Pine W wiki.pine@gmail.com wrote:
While I understand the intent of conserving search resource use, will this change have adverse effects in situations like professors instructing their 200-student classes to search for a particular topic and its related articles on Wikipedia?
Pine On May 18, 2015 12:35 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
In a few hours a patch will be going out which limits the number of concurrent searches a single user will be able to make to 5. This
applies
to logged in and anonymous users. The failure message is `You have too many concurrent searches running. If you are sharing an IP address with other users you can log in to get your own limits.` for anonymous users,
or
just `You have too many concurrent searches running.` for logged in
users.
These are the `cirrussearch-too-busy-for-you-anonymous-error` and `cirrussearch-too-busy-for-you-logged-in-error` i18n messages.
I will be monitoring the logs when this goes out, and intermittently throughout the week as well. If necessary we will whitelist certain ip ranges that seem to be shared among large numbers of users.
Erik B. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I think the use of tools to prevent high volume *automated* internal searches might be more appropriate, if good tools for that purpose are available.
Pine
On Mon, May 18, 2015 at 5:08 PM, Bahodir Mansurov bmansurov@wikimedia.org wrote:
I doubt all 200 students will be making concurrent searches.
I can easily imagine a scenario where 200 students in a large lecture classroom might be instructed to open their laptops, go to Wikipedia, and search for a particular topic at the same time. Similar to how teachers [used to] say "now everyone in the class turn to Chapter 8...".
If that is indeed what we're talking about here, it will be disruptive.
- Jonathan
On May 18, 2015, at 7:57 PM, John phoenixoverride@gmail.com wrote:
This will bring such a idea to a screeching halt. Any kind of shared IPs will be unable to search
On Mon, May 18, 2015 at 7:54 PM, Pine W wiki.pine@gmail.com wrote:
While I understand the intent of conserving search resource use, will
this
change have adverse effects in situations like professors instructing
their
200-student classes to search for a particular topic and its related articles on Wikipedia?
Pine On May 18, 2015 12:35 PM, "Erik Bernhardson" <
ebernhardson@wikimedia.org>
wrote:
In a few hours a patch will be going out which limits the number of concurrent searches a single user will be able to make to 5. This
applies
to logged in and anonymous users. The failure message is `You have too many concurrent searches running. If you are sharing an IP address
with
other users you can log in to get your own limits.` for anonymous
users,
or
just `You have too many concurrent searches running.` for logged in
users.
These are the `cirrussearch-too-busy-for-you-anonymous-error` and `cirrussearch-too-busy-for-you-logged-in-error` i18n messages.
I will be monitoring the logs when this goes out, and intermittently throughout the week as well. If necessary we will whitelist certain ip ranges that seem to be shared among large numbers of users.
Erik B. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Jonathan Morgan wrote:
On Mon, May 18, 2015 at 5:08 PM, Bahodir Mansurov bmansurov@wikimedia.org wrote:
I doubt all 200 students will be making concurrent searches.
I can easily imagine a scenario where 200 students in a large lecture classroom might be instructed to open their laptops, go to Wikipedia, and search for a particular topic at the same time. Similar to how teachers [used to] say "now everyone in the class turn to Chapter 8...".
If that is indeed what we're talking about here, it will be disruptive.
I imagine the more common cases involve either distributing a URL or instructing students to search for a particular topic, which typically routes through Google or Yahoo! or some external search engine. Both of these cases wouldn't be disrupted, as I understand it.
That said, I'm not sure what this thread is about. What problem are we trying to solve? Are we having issues with concurrent searches? Does anyone have links to Phabricator Maniphest tasks or Gerrit commits?
MZMcBride
On Mon, May 18, 2015 at 8:50 PM, MZMcBride z@mzmcbride.com wrote:
Jonathan Morgan wrote:
On Mon, May 18, 2015 at 5:08 PM, Bahodir Mansurov bmansurov@wikimedia.org wrote:
I doubt all 200 students will be making concurrent searches.
I can easily imagine a scenario where 200 students in a large lecture classroom might be instructed to open their laptops, go to Wikipedia, and search for a particular topic at the same time. Similar to how teachers [used to] say "now everyone in the class turn to Chapter 8...".
If that is indeed what we're talking about here, it will be disruptive.
I imagine the more common cases involve either distributing a URL or instructing students to search for a particular topic, which typically routes through Google or Yahoo! or some external search engine. Both of these cases wouldn't be disrupted, as I understand it.
We'll still keep an eye on it. More worrying is the assertion that some countries come through a surprisingly small number of IP for some reason. I've got a pretty itchy rollback finger and deploy rights.
That said, I'm not sure what this thread is about. What problem are we trying to solve? Are we having issues with concurrent searches? Does anyone have links to Phabricator Maniphest tasks or Gerrit commits?
This is the last of some security recommendations brownout a few months ago caused by someone finding an inefficient query and _hammering_ the reload button a couple hundred times. I'd link to the bug but it contains reproduction steps so its under some level of lock and key. The fix is us-specific so it's possible the issue is repeatable against other Lucene/Elasticsearch/SOLR users. As I said we've since prevented it from being exploitable on our side.
If we have to increase the limits or add whitelists we will. It'll be nice to have some protection but I'm sensitive to it causing trouble.
I expect Erik will be monitoring the logs tonight PDT time and I'll have a look early tomorrow morning EDT. The relevant commit in gerrit is https://gerrit.wikimedia.org/r/#/c/210622/ .
Nik
If the stressor point is a few hundred hits, lets pick a value low enough not to risk reaching the max, but high enough to not risk excessive collateral damage, Something along the lines of 40-50 would avoid most accidental triggers and low enough to limit server stress.
Its far better to incrementally step the limit down, to reach optimal values than to cut back radically and piss everyone off until you can raise the threshold
On Mon, May 18, 2015 at 9:25 PM, Nikolas Everett neverett@wikimedia.org wrote:
On Mon, May 18, 2015 at 8:50 PM, MZMcBride z@mzmcbride.com wrote:
Jonathan Morgan wrote:
On Mon, May 18, 2015 at 5:08 PM, Bahodir Mansurov bmansurov@wikimedia.org wrote:
I doubt all 200 students will be making concurrent searches.
I can easily imagine a scenario where 200 students in a large lecture classroom might be instructed to open their laptops, go to Wikipedia,
and
search for a particular topic at the same time. Similar to how teachers [used to] say "now everyone in the class turn to Chapter 8...".
If that is indeed what we're talking about here, it will be disruptive.
I imagine the more common cases involve either distributing a URL or instructing students to search for a particular topic, which typically routes through Google or Yahoo! or some external search engine. Both of these cases wouldn't be disrupted, as I understand it.
We'll still keep an eye on it. More worrying is the assertion that some countries come through a surprisingly small number of IP for some reason. I've got a pretty itchy rollback finger and deploy rights.
That said, I'm not sure what this thread is about. What problem are we trying to solve? Are we having issues with concurrent searches? Does anyone have links to Phabricator Maniphest tasks or Gerrit commits?
This is the last of some security recommendations brownout a few months ago caused by someone finding an inefficient query and _hammering_ the reload button a couple hundred times. I'd link to the bug but it contains reproduction steps so its under some level of lock and key. The fix is us-specific so it's possible the issue is repeatable against other Lucene/Elasticsearch/SOLR users. As I said we've since prevented it from being exploitable on our side.
If we have to increase the limits or add whitelists we will. It'll be nice to have some protection but I'm sensitive to it causing trouble.
I expect Erik will be monitoring the logs tonight PDT time and I'll have a look early tomorrow morning EDT. The relevant commit in gerrit is https://gerrit.wikimedia.org/r/#/c/210622/ .
Nik _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Mon, May 18, 2015 at 9:30 PM, John phoenixoverride@gmail.com wrote:
If the stressor point is a few hundred hits, lets pick a value low enough not to risk reaching the max, but high enough to not risk excessive collateral damage, Something along the lines of 40-50 would avoid most accidental triggers and low enough to limit server stress.
Its far better to incrementally step the limit down, to reach optimal values than to cut back radically and piss everyone off until you can raise the threshold
I bumped the limit from 5 to 15.
Due to unrelated issues in the evening SWAT deploy this didn't end up going out, it is scheduled for tomorrow's evening swat now (with the new limit at 15 concurrent searches).
On Mon, May 18, 2015 at 6:51 PM, Nikolas Everett neverett@wikimedia.org wrote:
On Mon, May 18, 2015 at 9:30 PM, John phoenixoverride@gmail.com wrote:
If the stressor point is a few hundred hits, lets pick a value low enough not to risk reaching the max, but high enough to not risk excessive collateral damage, Something along the lines of 40-50 would avoid most accidental triggers and low enough to limit server stress.
Its far better to incrementally step the limit down, to reach optimal values than to cut back radically and piss everyone off until you can
raise
the threshold
I bumped the limit from 5 to 15. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On May 19, 2015 2:25 AM, "Nikolas Everett" neverett@wikimedia.org wrote:
We'll still keep an eye on it. More worrying is the assertion that some countries come through a surprisingly small number of IP for some reason. I've got a pretty itchy rollback finger and deploy rights.
Yes, assuming https://en.wikipedia.org/wiki/Wikipedia:Blocking_IP_addresses#Sensitive_due_... is up to date then 30 searches for an entire country might not be enough.
wikitech-l@lists.wikimedia.org