Hi Folks,
TLDR:
Results from analysis of new beta header
- Mobile beta still does not have sufficient volume or stability for meaningful time-comparison experiments. - Rough numbers I was able to get (highly suspect), *suggest* that - the new header did not have a meaningful impact on either search or main menu clicks - There was a rise in opt-outs, but it started almost a week before the change (http://mobile-reportcard.wmflabs.org/, use table view) - Data and tables used here https://docs.google.com/spreadsheets/d/1oyTdP04IoYztG7ZNsJQ_0R9wjG8NOGU8tcq-iawqo8s/edit#gid=864984277
I ran the data from the new beta header, which was designed to promote main menu discovery by showing the menu anytime someone clicked in the header. We knew it was a confusing experience and made it harder to search, but since it was almost built before I joined the team, I asked the team to promote it to beta anyway so that we could see what, if any, the impact was.
Context:
Current: [image: Inline image 2] Behaviors:
- click search, begin typing immediately - click hamburger menu, see main menu
Test. Clicking anywhere on the header, including search, will now surface the main menu: [image: Inline image 1]
Behaviors:
- click anywhere in the heading and see both search and main menu. Click search again to begin typing
[image: Inline image 1]
Hypothesis:
- Main menu item clicks will increase - Search clicks will decrease
I was personally curious to see how much we could drive main menu clicks--would increased exposure improve visitation? How much would an extra click hurt search? These answers would help us as we made decisions for a new navigation. For all of the below, I looked at English Wikipedia only.
Complications/caveats:
- beta traffic is low (~500 search clicks a day, ~80 settings clicks before the change,) and fluctuates, so impacts measured should be taken with a grain of salt - pageview traffic is hard to derive, so I looked at an hour each day and used that as the index against which to measure actions, for stable pvs I also sampled 1/1000 - there is a period of missing main-menu click data whose impact is fully over by 7/11, so I could only measure the 4 days before the change. PV data seems limited to a 90 day window (at least the method I am using to query) - after the change, there was no measurement of overall 'header' clicks.
Results TLDR:
- when indexed against pageviews, search results did not decreaes! - surprisingly, main menu clicks did not have consistent improvements--largely - Home: +12% - Nearby: -6% (anomalous spike just before) - Random: +101% (there is clearly 1 day here with a major spike--just an outlier) - Collections: - 20% - Settings: + 27% (to change out of beta?) - pageviews decreased significantly over this period, however (25% over the two comparison windows). So overall actions did decrease. How to interpret the results, one has to know why pageviews decreased-- - Certainly one component is looking at partial weeks and different days of the week. Weekends see mobile spikes and the first portion is a weekend and the second was not. - Did they decrease because of a natural population decay from our pushing more people into beta? Maybe. - Did they decrease because people did not like the header. Unlikely--we see an opt out of beta jump that starts a few days before the change was promoted.
[image: Inline image 13]
(the 3 digit numbers below are dates:
Here is an example of the total number of actions during this time--a comparison to all traffic (which I dub 'stable') helps identify when a spike is or is not a beta artifact, but ultimately I ended up using pageviews as that is more relevant:
[image: Inline image 7]
[image: Inline image 10]
Here are clicks on "Home" in the main menu:
[image: Inline image 11]
Here is search: [image: Inline image 12]
The jumps you see in early May are from a banner campaign we ran to increase beta users so that we could run meaningful quantitative analysis.
Conclusion:
- We need to either increase beta users, a/b test, or test in stable more (which would also mean a/b testing on a small % of the population) - Increased exposure to the main menu in it's current state does not appear to have a strong positive impact on engagement. One might argue that this has a great deal to do with an awkward transition, but it is hard to tell with the noise. - Search was seemingly not impacted by a trivial extra step--people are possibly more resilient than we think.
