That bug is the biggest source of test failures right now. The overnight Firefox build had two failures, both of them caused by failure to set the on/off toggles without incurring a cookie error.
For those on Mobile devices here's the link outside the subject for clicking purposes: https://bugzilla.wikimedia.org/show_bug.cgi?id=62614
Currently only you Chris seem to be able to replicate this on beta labs which makes me think it is some kind of a platform issue (both Arthur and I have run these tests locally against beta labs with only passes)
I've taken a look at the code and nothing seems abnormal there. Maybe we should try a few stabs in the dark such as introducing a wait step before clicking?
I'm very keen to get this fixed asap I just have no idea what is going on here. On 21 Mar 2014 09:11, "Chris McMahon" cmcmahon@wikimedia.org wrote:
That bug is the biggest source of test failures right now. The overnight Firefox build had two failures, both of them caused by failure to set the on/off toggles without incurring a cookie error.
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Fri, Mar 21, 2014 at 9:23 AM, Jon Robson jdlrobson@gmail.com wrote:
Currently only you Chris seem to be able to replicate this on beta labs which makes me think it is some kind of a platform issue (both Arthur and I have run these tests locally against beta labs with only passes)
FWIW I have indeed done this locally, but I haven't actually run tests against betalabs (although I've gone through a number of the test cases manually on betalabs).
I've taken a look at the code and nothing seems abnormal there. Maybe we should try a few stabs in the dark such as introducing a wait step before clicking?
I'm very keen to get this fixed asap I just have no idea what is going on here.
My theory is that there is just something wonky with session/cookie handling. Theoretically for every test run, a new browser profile is used so there should not be issues with cookies (are we sure this is working correctly?). The other possibility is that there is something going sideways with whatever is being used for PHP session storage on betalabs. I have *no idea* how this is set up (in prod, we at least used to use memcached to store session data, but dunno if this is still the case or if this is the case for betalabs). Anyways, there could be something wonky there causing token mismatch problems. This seems somewhat unlikely to me because if this were a problem, we would probably experience more issues on betalabs than we have been - but it certainly still seems possible.
Chris mentioned to me earlier that he pinged Coren about this on IRC - any followup yet, Chris?
On Fri, Mar 21, 2014 at 2:27 PM, Arthur Richards arichards@wikimedia.orgwrote:
(in prod, we at least used to use memcached to store session data, but dunno if this is still the case or if this is the case for betalabs). Anyways, there could be something wonky there causing token mismatch problems. This seems somewhat unlikely to me because if this were a problem, we would probably experience more issues on betalabs than we have been - but it certainly still seems possible.
Chris mentioned to me earlier that he pinged Coren about this on IRC - any followup yet, Chris?
Yes, I talked with Coren and hashar both briefly. We're using memcache in beta labs for session storage also. Neither Coren nor hashar had any further illumination.
So here's another question:
This code is in MF:
private function submitSettingsForm() { $context = MobileContext::singleton(); $request = $this->getRequest();
if ( $request->getVal( 'token' ) != $context->getMobileToken() )
if the token in $request is always expected to be equal to the token in $context, then why does this if() statement even exist? Why do this check in the first place? Whatever this is guarding against seems to be actually happening.
-C
On Fri, Mar 21, 2014 at 2:40 PM, Chris McMahon cmcmahon@wikimedia.orgwrote:
On Fri, Mar 21, 2014 at 2:27 PM, Arthur Richards arichards@wikimedia.orgwrote:
(in prod, we at least used to use memcached to store session data, but dunno if this is still the case or if this is the case for betalabs). Anyways, there could be something wonky there causing token mismatch problems. This seems somewhat unlikely to me because if this were a problem, we would probably experience more issues on betalabs than we have been - but it certainly still seems possible.
Chris mentioned to me earlier that he pinged Coren about this on IRC - any followup yet, Chris?
Yes, I talked with Coren and hashar both briefly. We're using memcache in beta labs for session storage also. Neither Coren nor hashar had any further illumination.
:-/ I dont know much about the under-the-hood aspect of this but is there some way we can see what's going into and coming out of memcached while replicating the problem? It would be nice to rule this out.
So here's another question:
This code is in MF:
private function submitSettingsForm() { $context = MobileContext::singleton(); $request = $this->getRequest(); if ( $request->getVal( 'token' ) !=$context->getMobileToken() )
if the token in $request is always expected to be equal to the token in $context, then why does this if() statement even exist? Why do this check in the first place? Whatever this is guarding against seems to be actually happening.
While the tokens are expected to be equal, they aren't always. Usually when they aren't equal, it is for some benign reason, but it can be the result of someone attempting to use the form as a cross site request forgery vector (https://www.mediawiki.org/wiki/Cross-site_request_forgery). Long story short, we do this for security reasons - the approach here is very common throughout mediawiki and a standard way to prevent CSRF attacks.
So.. what can we do about this? This has now been a problem for some time. We should at least find a short term fix - maybe introducing delays might work here?
Sadly I cannot replicate this so I feel pretty useless and frustrated that I am unable to fix this myself..
On Fri, Mar 21, 2014 at 2:49 PM, Arthur Richards arichards@wikimedia.org wrote:
On Fri, Mar 21, 2014 at 2:40 PM, Chris McMahon cmcmahon@wikimedia.org wrote:
On Fri, Mar 21, 2014 at 2:27 PM, Arthur Richards arichards@wikimedia.org wrote:
(in prod, we at least used to use memcached to store session data, but dunno if this is still the case or if this is the case for betalabs). Anyways, there could be something wonky there causing token mismatch problems. This seems somewhat unlikely to me because if this were a problem, we would probably experience more issues on betalabs than we have been - but it certainly still seems possible.
Chris mentioned to me earlier that he pinged Coren about this on IRC - any followup yet, Chris?
Yes, I talked with Coren and hashar both briefly. We're using memcache in beta labs for session storage also. Neither Coren nor hashar had any further illumination.
:-/ I dont know much about the under-the-hood aspect of this but is there some way we can see what's going into and coming out of memcached while replicating the problem? It would be nice to rule this out.
So here's another question:
This code is in MF:
private function submitSettingsForm() { $context = MobileContext::singleton(); $request = $this->getRequest(); if ( $request->getVal( 'token' ) !=$context->getMobileToken() )
if the token in $request is always expected to be equal to the token in $context, then why does this if() statement even exist? Why do this check in the first place? Whatever this is guarding against seems to be actually happening.
While the tokens are expected to be equal, they aren't always. Usually when they aren't equal, it is for some benign reason, but it can be the result of someone attempting to use the form as a cross site request forgery vector (https://www.mediawiki.org/wiki/Cross-site_request_forgery). Long story short, we do this for security reasons - the approach here is very common throughout mediawiki and a standard way to prevent CSRF attacks.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
On Fri, Mar 21, 2014 at 10:27 PM, Arthur Richards arichards@wikimedia.orgwrote:
Theoretically for every test run, a new browser profile is used so there should not be issues with cookies (are we sure this is working correctly?).
For every test on Sauce Labs they start a clean virtual machine with clean browser. I would be highly surprised if there was a problem with that.
Željko
Chris M, Chris S, and Jeremy are digging into it by investigating packet captures via Wireshark (see latest comments on the bug). Hopefully they'll come up with something. AFAICT, this problem is reproducible on Windows and Linux boxes, but not on OS X.
On Thu, Mar 27, 2014 at 8:23 AM, Željko Filipin zfilipin@wikimedia.orgwrote:
On Fri, Mar 21, 2014 at 10:27 PM, Arthur Richards <arichards@wikimedia.org
wrote:
Theoretically for every test run, a new browser profile is used so there should not be issues with cookies (are we sure this is working correctly?).
For every test on Sauce Labs they start a clean virtual machine with clean browser. I would be highly surprised if there was a problem with that.
Željko