Hey Folks (and Florian in particular),We created an internal web list to discuss admin stuff, and I failed by posting the email to the internal list. I wanted to catch you (and the rest of the team up on the latest link preview data).
Link preview rationale:
the hypothesis is that if blue-links are more effective overall for users (benefit/cost is higher on average) with link preview, people will click them more often.
Android results:
Link preview increased blue-link clicks/user by almost 30% in beta, but in stable it looks flat to negative. We tried a new version and it was slightly better, but still no improvement over 'no link-preview' condition.
Web decision
One of the reasons we have independent teams on each platform is to allow us to not make the same mistake on more than one platform. Based on the above Android results, we made a late-quarter decision not to push link preview until more analysis could be done. This is what email below kicked off.
Update
It occurred to us yesterday that our current metric is probably incomplete. We were looking at clicks/user on Android, when we should have been looking at clicks/pageview. We treated them as equivalent, but what actually happens is that pageviews drop since not all users continue onto the article. So clicks/user sees no improvement, but clicks/page looks to have an increase of ~10%!![?] So this suggests that link preview is arguably an improvement. I say arguable because, the drop in overall clicks comes from the fact that before, a user would click on a link, go to an article and then click on a link from that article (3 articles total). Now, the user will sometimes decide they have enough information and not visit the second article. We are giving more targeted information, but are losing some 'serendipity'.
As Kaity put it, "
The link preview UX is encouraging people to stay on one article and check out lots of links but not click through to any of them.
Without link preview, a user is more likely to get completely off-topic, and forget the original article.
Which of these behaviors do we want to encourage? We should balance a focused reading experience with one that encourages the wiki rabbit hole.
If a user is encouraged to explore in other ways, like the feed and read more, than a UX that encourages them to stay focused on 1 article is justified.
Next steps:
We made the call not to work on link preview on the mobile web (or hovercards on desktop web [1]) in the remainder of Q2 based on our earlier analysis. I am tempted to say that this secondary analysis justifies implementing link preview on the web, but think it is at least worth a discussion for mobile web, due to the potential trade-off. Do we try again in Q3? I know Florian is already doing a great deal of the work.
-J
[?] Dmitry on the hook for further analysis
[1] The survey results for hovercards when rolled out as'on-by-default' to specific wikis were astoundingly positive...I don't have them handy, but if you're curious, please ping me.
---------- Forwarded message ----------
From:
Jon Katz <jkatz@wikimedia.org>Date: Fri, Nov 20, 2015 at 5:05 PM
Subject: Next sprint
To: Joaquin Oltra Hernandez <
jhernandez@wikimedia.org>, Kristen Lans <
klans@wikimedia.org>, Jon Robson <
jrobson@wikimedia.org>
Hi,
I want to give you a head's up on some recent developments that could impact next sprint. All of this came in either last night or this afternoon--we can discuss on Monday at kickoff, but asynch before then will obviously save meeting time.
Link Preview:
Just under the wire, the first couple days of data are in for the new link preview on Android and they do not show a significant marked increase in links clicked per user. In other words, it looks like link preview is still not generating any measurable improvement in user engagement on Android. I am not a good judge of how much work it would take to push link preview to mobile beta for further testing (an actual a/b test), but think that is probably not a great use of your time (unless it is really small). On the one hand, I believe in the feature and want to give it a chance on web, but I also believe in staggering development by platform so that we don't make the same mistakes twice. If anything, the bar is set higher on the mobile web then on Android, since we are hijacking link clicks.
If we don't launch link preview we could (and should) continue to move hovercards along as they do not hijack the link and have rave qualitative reviews. This might still represent enough work to see us through the rest of the quarter, but I am not sure if you have what you need lined up. How do you folks feel about it?
Added to sprint board since planning:
- This bug, if it is valid, will prevent us from gathering data from our surveys until it is fixed: https://phabricator.wikimedia.org/T119152
- Added another story to account for the 2 surveys that we hope to run next week before the fundraiser block: https://phabricator.wikimedia.org/T119267
- Might not make it:
- After Toby and I met with community members yesterday, he is making the executive decision that we should get rid of the special user profile to appease our contributors: https://phabricator.wikimedia.org/T85929. I think we have stalled long enough, but I think we should only remove it if a user doesn't already have a page. Otherwise a new user who clicks on their user name gets taken to a blank edit screen. If a user doesn't have a page, we should add a message to specialuserprofile saying --click here to create your own user profile. A CTA on the link click could also work. Nirzar can you jump on this?
-J