I re-ran the queries over some longer time spans to see if it changed the results any. The only difference is that the small differences in Jon's results became even smaller (to the point of being nearly non-existent):
Pre A-B test, successful save events (15 days) B: 23702 A: 24967 (48669 saves in both groups, 1265 more in group A - 2.3%)
Post A/B test successful save events (15 days) B: 19352 A: 19537 (38889 saves in both groups, 185 more in group A - 0.5%)
New users pre A-B test successful save events (15 days) B: 10708 A: 10741 (21449 saves in both groups, 33 more in group A - 0.2%)
New users post A-B test successful save events (15 days) B: 8530 A: 8529 (17059 saves in both groups, 1 more in group B - 0%)
Ryan Kaldari
On Mon, Sep 22, 2014 at 4:58 PM, Jon Robson jrobson@wikimedia.org wrote:
Argggg badly named events.... 'save' is actually hitting the preview button (this is why the results vary so much between save and success). What I actually wanted was submit.
The same conclusion as above holds, but let's do those sql queries/maths again for completion:
Pre A-B test, successful save events [1] 0 11872 1 13299 (25171 saves in both groups, 1427 more in group A - 5.7%)
Post A/B test successful save events [2] 0 10075 1 10316 (20391 saves in both groups, 241 more in group A - 1.2%)
New users pre A-B test successful save events [3] 0 5643 1 5829 (11472 saves in both groups, 186 more in group A - 1.6%)
New users post A-B test successful save events [4] 0 4163 1 4359 (8522 saves in both groups, 196 more in group A - 2.3%)
[A] https://gerrit.wikimedia.org/r/162164
[1] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA
[2] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA
[3] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA
[4] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140912000000 and timestamp< 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA
On Mon, Sep 22, 2014 at 4:42 PM, Jon Robson jrobson@wikimedia.org wrote:
Thanks for this link Nemo!
I've discovered a caveat over the weekend which seems to make these results more unreliable. In test A, the success event is logged and then the page reloaded. Concerned that the event might be lost in the page reload and skew the results I decided to look at the 'save' event which is logged whenever the user hits the save button. In comparison the 'success' event is only logged when the save is successful.
Pre A-B test, successful save events [1] 0 14350 1 16518 (30868 saves in both groups, 2168 more in group A, 7%)
Post A/B test successful save events [2] 0 13462 1 13273 (26735 saves in both groups, 189 more in group B - 0.07%)
New users pre A-B test successful save events [3] 0 6216 1 6362 (12578 saves in both groups, 146 more in group A - 1%)
New users post A-B test successful save events [4] 0 4870 1 4772 (9642 saves in both groups, 98 more in group B - 1%)
These show little variance amongst the groups and lead me to believe that the save data I posted earlier cannot be relied on.
I've written a patch to ensue the events get logged appropriately [A] but I think from this my own conclusion is that ajax loading a page via JavaScript after an edit only makes a significant difference (if no difference at all) to the number of edits a user goes on to make. We can probably safely kill it without concern.
[A] https://gerrit.wikimedia.org/r/162164
[1] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'save' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA
[2] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'save' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA
[3] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'save' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA
[4] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'save' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA
On Fri, Sep 19, 2014 at 11:27 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Thanks. For reference, the tracking bug for this area is: Bug 14123 –
Make
MediaWiki more AJAXy
https://bugzilla.wikimedia.org/showdependencytree.cgi?id=14123&hide_reso...
Hoo *tripled* the number of patrollers in en.wiki by making patrol AJAXy (bug 7851/15936), so it's definitely an area worth working on.
Nemo
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l