Hi Folks,
TLDR:
Results from analysis of new beta header
- Mobile beta still does not have sufficient volume or stability for
meaningful time-comparison experiments.
- Rough numbers I was able to get (highly suspect), *suggest* that
- the new header did not have a meaningful impact on either search or
main menu clicks
- There was a rise in opt-outs, but it started almost a week before
the change (
http://mobile-reportcard.wmflabs.org/, use table view)
- Data and tables used here
<https://docs.google.com/spreadsheets/d/1oyTdP04IoYztG7ZNsJQ_0R9wjG8NOGU8tcq-iawqo8s/edit#gid=864984277>
I ran the data from the new beta header, which was designed to promote main
menu discovery by showing the menu anytime someone clicked in the header.
We knew it was a confusing experience and made it harder to search, but
since it was almost built before I joined the team, I asked the team to
promote it to beta anyway so that we could see what, if any, the impact was.
Context:
Current:
[image: Inline image 2]
Behaviors:
- click search, begin typing immediately
- click hamburger menu, see main menu
Test. Clicking anywhere on the header, including search, will now surface
the main menu:
[image: Inline image 1]
Behaviors:
- click anywhere in the heading and see both search and main menu.
Click search again to begin typing
[image: Inline image 1]
Hypothesis:
- Main menu item clicks will increase
- Search clicks will decrease
I was personally curious to see how much we could drive main menu
clicks--would increased exposure improve visitation? How much would an
extra click hurt search? These answers would help us as we made decisions
for a new navigation. For all of the below, I looked at English Wikipedia
only.
Complications/caveats:
- beta traffic is low (~500 search clicks a day, ~80 settings clicks
before the change,) and fluctuates, so impacts measured should be taken
with a grain of salt
- pageview traffic is hard to derive, so I looked at an hour each day
and used that as the index against which to measure actions, for stable pvs
I also sampled 1/1000
- there is a period of missing main-menu click data whose impact is
fully over by 7/11, so I could only measure the 4 days before the change.
PV data seems limited to a 90 day window (at least the method I am using to
query)
- after the change, there was no measurement of overall 'header' clicks.
Results TLDR:
- when indexed against pageviews, search results did not decreaes!
- surprisingly, main menu clicks did not have consistent
improvements--largely
- Home: +12%
- Nearby: -6% (anomalous spike just before)
- Random: +101% (there is clearly 1 day here with a major spike--just
an outlier)
- Collections: - 20%
- Settings: + 27% (to change out of beta?)
- pageviews decreased significantly over this period, however (25% over
the two comparison windows). So overall actions did decrease. How to
interpret the results, one has to know why pageviews decreased--
- Certainly one component is looking at partial weeks and different
days of the week. Weekends see mobile spikes and the first portion is a
weekend and the second was not.
- Did they decrease because of a natural population decay from our
pushing more people into beta? Maybe.
- Did they decrease because people did not like the header.
Unlikely--we see an opt out of beta jump that starts a few days
before the
change was promoted.
[image: Inline image 13]
(the 3 digit numbers below are dates:
Here is an example of the total number of actions during this time--a
comparison to all traffic (which I dub 'stable') helps identify when a
spike is or is not a beta artifact, but ultimately I ended up using
pageviews as that is more relevant:
[image: Inline image 7]
[image: Inline image 10]
Here are clicks on "Home" in the main menu:
[image: Inline image 11]
Here is search:
[image: Inline image 12]
The jumps you see in early May are from a banner campaign we ran to
increase beta users so that we could run meaningful quantitative analysis.
Conclusion:
- We need to either increase beta users, a/b test, or test in stable
more (which would also mean a/b testing on a small % of the population)
- Increased exposure to the main menu in it's current state does not
appear to have a strong positive impact on engagement. One might argue
that this has a great deal to do with an awkward transition, but it is hard
to tell with the noise.
- Search was seemingly not impacted by a trivial extra step--people are
possibly more resilient than we think.
