I ran the data from the new beta header, which was designed to promote main menu discovery by showing the menu anytime someone clicked in the header. We knew it was a confusing experience and made it harder to search, but since it was almost built before I joined the team, I asked the team to promote it to beta anyway so that we could see what, if any, the impact was.
Context:
Current:
Behaviors:
- click search, begin typing immediately
- click hamburger menu, see main menu
Test. Clicking anywhere on the header, including search, will now surface the main menu:
Behaviors:
- click anywhere in the heading and see both search and main menu. Click search again to begin typing
Hypothesis:
- Main menu item clicks will increase
- Search clicks will decrease
I was personally curious to see how much we could drive main menu clicks--would increased exposure improve visitation? How much would an extra click hurt search? These answers would help us as we made decisions for a new navigation. For all of the below, I looked at English Wikipedia only.
Complications/caveats:
- beta traffic is low (~500 search clicks a day, ~80 settings clicks before the change,) and fluctuates, so impacts measured should be taken with a grain of salt
- pageview traffic is hard to derive, so I looked at an hour each day and used that as the index against which to measure actions, for stable pvs I also sampled 1/1000
- there is a period of missing main-menu click data whose impact is fully over by 7/11, so I could only measure the 4 days before the change. PV data seems limited to a 90 day window (at least the method I am using to query)
- after the change, there was no measurement of overall 'header' clicks.
Results TLDR:
- when indexed against pageviews, search results did not decreaes!
- surprisingly, main menu clicks did not have consistent improvements--largely
- Home: +12%
- Nearby: -6% (anomalous spike just before)
- Random: +101% (there is clearly 1 day here with a major spike--just an outlier)
- Collections: - 20%
- Settings: + 27% (to change out of beta?)
- pageviews decreased significantly over this period, however (25% over the two comparison windows). So overall actions did decrease. How to interpret the results, one has to know why pageviews decreased--
- Certainly one component is looking at partial weeks and different days of the week. Weekends see mobile spikes and the first portion is a weekend and the second was not.
- Did they decrease because of a natural population decay from our pushing more people into beta? Maybe.
- Did they decrease because people did not like the header. Unlikely--we see an opt out of beta jump that starts a few days before the change was promoted.
(the 3 digit numbers below are dates:
Here is an example of the total number of actions during this time--a comparison to all traffic (which I dub 'stable') helps identify when a spike is or is not a beta artifact, but ultimately I ended up using pageviews as that is more relevant:
Here are clicks on "Home" in the main menu:
Here is search:
The jumps you see in early May are from a banner campaign we ran to increase beta users so that we could run meaningful quantitative analysis.
Conclusion:
- We need to either increase beta users, a/b test, or test in stable more (which would also mean a/b testing on a small % of the population)
- Increased exposure to the main menu in it's current state does not appear to have a strong positive impact on engagement. One might argue that this has a great deal to do with an awkward transition, but it is hard to tell with the noise.
- Search was seemingly not impacted by a trivial extra step--people are possibly more resilient than we think.