Background: * A patch got merged Fri Aug 29 [A] that bucketed 50% of edits so they would reload the page after an edit (test A), and 50% of edits would re-render the page in JavaScript using an ajax loader (test B) * The patch went live on English Wikipedia on the 11th September. Method: * I looked at an arbitary 7 day period before the 11th September (27th August till 4th September) and I looked at an arbitary 7 day period after the A/B test went live (12th till 19th September) * I counted successful edits via the mobile interface on English Wikipedia, restricted to the stable mode of the site and to the source editor only (not VisualEditor interface). This would minimise variance between the interfaces that could happen across different projects/languages/experimental features. * I looked at the exact same 7 day periods but this time restricted them only to editors with less than 5 editors to restrict the data to new editors. * I counted edits across all projects and how they varied post A/B test Results: Table 1: For the 7 day period before the A/B test [1]. isTestA Successful edits 0 9883 1 11097 Table 2: For a 7 day period after the result went live [2]. isTestA Successful edits 0 8835 1 7011 Table 3: For the 7 day period before the A/B test where the editors at the time of completing an edit had less than 5 edits [3] isTestA Successful edits 0 4276 1 4449 Table 4: For the 7 day period after the A/B test where the editors at the time of completing an edit had more than 5 edits [4] isTestA Successful edits 0 3359 1 2707 A breakdown per project can be found here generated using this SQL query [5]: https://docs.google.com/spreadsheets/d/11RFrOd4CBVk2hXIHjbSz8SPyMH7GlARUkNb… Analysis: Pre-A/B test for all editors, 1214 more edits occurred in bucket A (5.7%). See Table 1. You'd expect these to be the same. This raises concerns that there might be a problem with the A/B bucketing. However if you look at table 3: Pre A/B test for editors with less than 5 edits, 173 more edits occurred in bucket A (1.9%). Post A/B test for all editors, 1824 more edits happened in bucket B according to table 2 (11.5%) Post A/B test for editors with an edit count of less than 5, 652 more edits happened in bucket B according to table 4 (10.7%) * Considering the shift of successful edits from occurring in bucket A to bucket B, it is possible that ajax loading a page is generally getting us more edits. It seems that looking at new editors (edit count less than 5) is a more accurate way to compare the two sets of data. * This could possible be because the page reloads quicker, and a user is more likely to see a mistake and do a follow up edit or to get a better experience from editing (finds it quicker and more pleasurable.) I'd suggest more investigation into this but this data is pretty fascinating. Please do tear apart my method/investigate further. Links____ [A] https://gerrit.wikimedia.org/r/#/c/155675/ [1] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'success' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA [2] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'success' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA [3] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'success' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA [4] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'success' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA [5] select event_isTestA, wiki, count(*) from MobileWebEditing_8599025 where event_action = 'success' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and event_mobileMode = 'stable' group by wiki, event_isTestA order by wiki,event_isTestA