Background:
* A patch got merged Fri Aug 29 [A] that bucketed 50% of edits so they
would reload the page after an edit (test A), and 50% of edits would
re-render the page in JavaScript using an ajax loader (test B)
* The patch went live on English Wikipedia on the 11th September.
Method:
* I looked at an arbitary 7 day period before the 11th September (27th
August till 4th September) and I looked at an arbitary 7 day period
after the A/B test went live (12th till 19th September)
* I counted successful edits via the mobile interface on English
Wikipedia, restricted to the stable mode of the site and to the source
editor only (not VisualEditor interface). This would minimise variance
between the interfaces that could happen across different
projects/languages/experimental features.
* I looked at the exact same 7 day periods but this time restricted
them only to editors with less than 5 editors to restrict the data to
new editors.
* I counted edits across all projects and how they varied post A/B test
Results:
Table 1:
For the 7 day period before the A/B test [1].
isTestA Successful edits
0 9883
1 11097
Table 2:
For a 7 day period after the result went live [2].
isTestA Successful edits
0 8835
1 7011
Table 3:
For the 7 day period before the A/B test where the editors at the time
of completing an edit had less than 5 edits [3]
isTestA Successful edits
0 4276
1 4449
Table 4:
For the 7 day period after the A/B test where the editors at the time
of completing an edit had more than 5 edits [4]
isTestA Successful edits
0 3359
1 2707
A breakdown per project can be found here generated using this SQL query [5]:
https://docs.google.com/spreadsheets/d/11RFrOd4CBVk2hXIHjbSz8SPyMH7GlARUkNb…
Analysis:
Pre-A/B test for all editors, 1214 more edits occurred in bucket A
(5.7%). See Table 1.
You'd expect these to be the same. This raises concerns that there
might be a problem with the A/B bucketing.
However if you look at table 3:
Pre A/B test for editors with less than 5 edits, 173 more edits
occurred in bucket A (1.9%).
Post A/B test for all editors, 1824 more edits happened in bucket B
according to table 2 (11.5%)
Post A/B test for editors with an edit count of less than 5, 652 more
edits happened in bucket B according to table 4 (10.7%)
* Considering the shift of successful edits from occurring in bucket A
to bucket B, it is possible that ajax loading a page is generally
getting us more edits. It seems that looking at new editors (edit
count less than 5) is a more accurate way to compare the two sets of
data.
* This could possible be because the page reloads quicker, and a user
is more likely to see a mistake and do a follow up edit or to get a
better experience from editing (finds it quicker and more
pleasurable.)
I'd suggest more investigation into this but this data is pretty
fascinating. Please do tear apart my method/investigate further.
Links____
[A]
https://gerrit.wikimedia.org/r/#/c/155675/
[1] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'success' and timestamp > 20140827000000 and timestamp
< 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_mobileMode = 'stable' group by event_isTestA
[2] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'success' and timestamp > 20140912000000 and timestamp
< 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_mobileMode = 'stable' group by event_isTestA
[3] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'success' and timestamp > 20140827000000 and timestamp
< 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_userEditCount < 5 and event_mobileMode = 'stable' group by
event_isTestA
[4] select event_isTestA, count(*) from MobileWebEditing_8599025 where
event_action = 'success' and timestamp > 20140912000000 and timestamp
< 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki'
and event_userEditCount < 5 and event_mobileMode = 'stable' group by
event_isTestA
[5] select event_isTestA, wiki, count(*) from MobileWebEditing_8599025
where event_action = 'success' and timestamp > 20140912000000 and
timestamp < 20140919000000 and event_editor = 'SourceEditor' and
event_mobileMode = 'stable' group by wiki, event_isTestA order by
wiki,event_isTestA