I re-ran the queries over some longer time spans to see if it changed the results any. The only difference is that the small differences in Jon's results became even smaller (to the point of being nearly non-existent):

Pre A-B test, successful save events (15 days)
B: 23702
A: 24967
(48669 saves in both groups, 1265 more in group A - 2.3%)

Post A/B test successful save events (15 days)
B: 19352
A: 19537
(38889 saves in both groups, 185 more in group A - 0.5%)

New users pre A-B test successful save events (15 days)
B: 10708
A: 10741
(21449 saves in both groups, 33 more in group A - 0.2%)

New users post A-B test successful save events (15 days)
B: 8530
A: 8529
(17059 saves in both groups, 1 more in group B - 0%)

Ryan Kaldari

On Mon, Sep 22, 2014 at 4:58 PM, Jon Robson <jrobson@wikimedia.org> wrote:
Argggg badly named events....
'save' is actually hitting the preview button (this is why the results
vary so much between save and success). What I actually wanted was
submit.

The same conclusion as above holds, but let's do those sql
queries/maths again for completion:

Pre A-B test, successful save events [1]
0 11872
1 13299
(25171 saves in both groups, 1427 more in group A - 5.7%)

Post A/B test successful save events [2]
0 10075
1 10316
(20391 saves in both groups, 241 more in group A - 1.2%)

New users pre A-B test successful save events [3]
0 5643
1 5829
(11472 saves in both groups, 186 more in group A - 1.6%)

New users post A-B test successful save events [4]
0 4163
1 4359
(8522 saves in both groups, 196 more in group A - 2.3%)

[A] https://gerrit.wikimedia.org/r/162164

[1] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'submit' and timestamp > 20140827000000 and timestamp
< 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_mobileMode = 'stable' group by event_isTestA

[2] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'submit' and timestamp > 20140912000000 and timestamp
< 20140919000000  and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_mobileMode = 'stable' group by event_isTestA

[3] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'submit' and timestamp > 20140827000000 and timestamp
< 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_userEditCount < 5 and event_mobileMode = 'stable' group by
event_isTestA

[4] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'submit' and timestamp > 20140912000000 and timestamp<
20140919000000
and event_editor = 'SourceEditor' and wiki ='enwiki' and
event_userEditCount < 5 and event_mobileMode = 'stable' group by
event_isTestA

On Mon, Sep 22, 2014 at 4:42 PM, Jon Robson <jrobson@wikimedia.org> wrote:
> Thanks for this link Nemo!
>
> I've discovered a caveat over the weekend which seems to make these
> results more unreliable.
> In test A, the success event is logged and then the page reloaded.
> Concerned that the event might be lost in the page reload and skew the
> results I decided to look at the 'save' event which is logged whenever
> the user hits the save button. In comparison the 'success' event is
> only logged when the save is successful.
>
> Pre A-B test, successful save events [1]
> 0 14350
> 1 16518
> (30868 saves in both groups, 2168 more in group A, 7%)
>
> Post A/B test successful save events [2]
> 0 13462
> 1 13273
> (26735 saves in both groups, 189 more in group B - 0.07%)
>
> New users pre A-B test successful save events [3]
> 0 6216
> 1 6362
> (12578 saves in both groups, 146 more in group A - 1%)
>
> New users post A-B test successful save events [4]
> 0 4870
> 1 4772
> (9642 saves in both groups, 98 more in group B - 1%)
>
> These show little variance amongst the groups and lead me to believe
> that the save data I posted earlier cannot be relied on.
>
> I've written a patch to ensue the events get logged appropriately [A]
> but I think from this my own conclusion is that ajax loading a page
> via JavaScript after an edit only makes a significant difference (if
> no difference at all) to the number of edits a user goes on to make.
> We can probably safely kill it without concern.
>
> [A] https://gerrit.wikimedia.org/r/162164
>
> [1] select event_isTestA, count(*) from MobileWebEditing_8599025 where
> event_action = 'save' and timestamp > 20140827000000 and timestamp
> < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
> and event_mobileMode = 'stable' group by event_isTestA
>
> [2] select event_isTestA, count(*) from MobileWebEditing_8599025 where
> event_action = 'save' and timestamp > 20140912000000 and timestamp
> < 20140919000000  and event_editor = 'SourceEditor' and wiki ='enwiki'
> and event_mobileMode = 'stable' group by event_isTestA
>
> [3] select event_isTestA, count(*) from MobileWebEditing_8599025 where
> event_action = 'save' and timestamp > 20140827000000 and timestamp
> < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
> and event_userEditCount < 5 and event_mobileMode = 'stable' group by
> event_isTestA
>
> [4] select event_isTestA, count(*) from MobileWebEditing_8599025 where
> event_action = 'save' and timestamp > 20140912000000 and timestamp
> < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
> and event_userEditCount < 5 and event_mobileMode = 'stable' group by
> event_isTestA
>
> On Fri, Sep 19, 2014 at 11:27 PM, Federico Leva (Nemo)
> <nemowiki@gmail.com> wrote:
>> Thanks. For reference, the tracking bug for this area is: Bug 14123 – Make
>> MediaWiki more AJAXy
>> https://bugzilla.wikimedia.org/showdependencytree.cgi?id=14123&hide_resolved=1
>> Hoo *tripled* the number of patrollers in en.wiki by making patrol AJAXy
>> (bug 7851/15936), so it's definitely an area worth working on.
>>
>> Nemo

_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l