Re: [Analytics] Share a Fact Initial Analysis

Thanks. Right, good point, will need to re-run the pageview count to exclude spiders. On Friday, May 22, 2015, Dario Taraborelli &lt;dtaraborelli(a)wikimedia.org&gt; wrote: > Thanks for sharing this, Adam. Aside from engagement/funnel data, the > critical question for this feature is: does it bring back eyeballs to the > site from social media? It looks like it doesn’t yet, at least not in a > substantial way, even with the caveat that App traffic is a very small > fraction of total mobile traffic. > > Having looked into referrals for this feature before and after comparing > them to Twitter’s own engagement analytics (and finding some big > discrepancy), you should consider removing spiders/crawlers from the data > (see [1]) to avoid inflating pageviews with non-human activity. > > I’m a big fan of this feature and look forward to seeing how you guys > intend to scale it. > > Dario > > [1] > https://github.com/ewulczyn/wmf/blob/b9f726ee3468852c3fed2780af1d8ac0004eda… > > > On May 21, 2015, at 12:37 PM, Toby Negrin &lt;tnegrin(a)wikimedia.org&gt; wrote: > > Hi all - some interesting analysis on the share-a-fact feature from the > mobile team. > > -Toby > > Begin forwarded message: > > *From:* Adam Baso &lt;abaso(a)wikimedia.org&gt; > *Date:* May 21, 2015 at 12:05:29 PDT > *To:* mobile-l &lt;mobile-l(a)lists.wikimedia.org&gt; > *Subject:* *[WikimediaMobile] Share a Fact Initial Analysis* > > Hello all, > > We’ve been looking at some initial results from the Share a Fact feature > introduced on the Wikipedia apps for Android and iOS in its basic "minimal > viable product" implementation. Here’s some analysis, using data from one > day (20150512) with respect to the latest stable versions of the apps > (2.0-r-2015-04-23 on Android and 4.1.2 on iOS) for that day. > > * On iOS, when a user initiates the first step of the default sharing > workflow - tapping the up-arrow box share button (6,194 non-highlighting > instances for the day under question) - about 11.7% of the time it yielded > successful sharing. > > * On Android, it’s not possible to easily tell when the sharing workflow > was carried through to successful share, but we anticipate the Android > success rate is currently much higher, as general engagement percentage up > to the point of picking an app for sharing is higher on Android than on iOS. > > * On Android, when presented with the share card preview, 28.0% of the > time the ‘Share as image’ button was tapped and 55.5% of the time the > 'Share as text' button was tapped, whereas on iOS it was 8.4% ‘Share as > image’ and 16.8% ‘Share as text’. > > * The forthcoming 4.1.4 version of the iOS app will relax its default > sharing snippet generation rules and be more like the Android version in > that respect. We anticipate this will result in higher engagement with both > the ‘Share as image’ and ‘Share as text’ buttons on iOS, and we should be > able to verify this once the 4.1.4 iOS version is released and generally > adopted (usually takes 4-5 days after release; the 4.1.4 release isn’t > released yet). > > * On the Android app the ‘Share’ option is located on the overflow menu, > not as part of the main set of UI buttons. This potentially increases the > likelihood of Android users being primed to step through the workflow. On > the iOS app, the share button (up-arrow box) is plainly visible from the > main UI and not an overflow menu, and this probably creates a different > priming dynamic for the iOS demographic. > > * When users on iOS tapped on the ‘Share as image’ or ‘Share as text’ > buttons, there is a pretty sharp drop off at the next stage - the system > sharesheet. Once the sharesheet was presented to iOS users, 41.6% of the > time it resulted in active abandonment. We believe this probably has > something to do with the relatively small set of default apps listed on the > sharesheet and the extra work involved with exposing additional social apps > for sharing in that context. As with the Android app, the labels of ‘Share > as image’ and ’Share as text’ may also pose something of a hurdle at least > for first time users of the feature. To this end, there is an onboarding > tutorial planned at least on Android. > > * For a one hour period (2015051201) there were about 100 pageviews in > some sense attributable to Share a Fact using a provenance parameter > available on the latest stable versions of the apps at that time; this may > slightly overstate the number of pageviews attributable to the two specific > apps reviewed in this analysis, but probably not too much (n.b., previously > a different source parameter was used than the new wprov provenance > parameter). Pageviews are not the sole motivation for the feature, but > following the trendline over the long run should be interesting. Impact on > social media and the destinations of shares is a little harder to capture > directly, but > https://twitter.com/search?f=realtime&q=%40wikipedia%20-%40itzwikipedia… > gives one a sense about image shares, at least. > > * A couple potential options for increasing sharing include: > > ** Trying to add support for sharing to the Photos app on iOS. People may > be interested in using images from the Photos apps for various workflows, > as Dan Garry has noted. > > ** Offering a more concise app picklist, in particular explicitly adding > the native OS app components (namely, Twitter and Facebook, and as > mentioned, Photos if possible), with an option to expose the sharesheet for > additional options if necessary. This is probably also somewhat confined to > iOS, although conceivably a similar approach could be possible on Android. > On Android the full list of applications in its equivalent of the > sharesheet is by default readily available to the user, though. > > ** On Android, exposing the diagonal arrow share button on the main > interface akin to how the iOS version of the app shows the up-arrow share > button. This may introduce more opportunities for sharing (and thus numbers > of abandons would go up in tandem with numbers of shares), but would also > partially clutter the interface and probably increase abandon. A controlled > experiment may be useful for observing the impact of such an approach. > > * As a point of reference, for the app versions in scope for this > analysis over a single day, there appeared to be approximately 3.78 million > Wikipedia for Android pageviews and 1.19 Wikipedia Mobile for iOS app > pageviews. There were about 6.73 million app pageviews on the “modern” > versions of these apps total for this particular day, meaning there were > about 1.75 million pageviews on other modern versions of the app. > > * Examination of the categories of successful shares on iOS showed the > following distributions: > > Images: > 48.5% messaging > 25.5% sharesheet copy > 22.9% social > 1.8% productivity > 0.9% reading > > > Text: > 53.6% messaging > 31.9% sharesheet copy > 7.1% social > 5.4% reading > 2.0% productivity > > > Here were some queries used in the analysis: > > == SHARE A FACT ATTRIBUTABLE PAGEVIEWS FOR ONE HOUR == > > select wprov, uri_host, count(*) from (select x_analytics_map['wprov'] as > wprov, uri_host > from webrequest where year = 2015 and month = 5 and day = 12 and hour = 1 > and is_pageview = true and uri_host like '%.wikipedia.org' and > x_analytics_map['wprov'] is not null) t > group by wprov, uri_host; > > > == PAGE VIEWS FOR THE DAY FOR THE “MODERN” VERSIONS OF THE APPS == > > SELECT > user_agent, count(*) > FROM > wmf.webrequest > tablesample(BUCKET 1 OUT OF 100 ON rand()) > WHERE > YEAR = 2015 > AND MONTH = 5 > AND DAY = 12 > AND is_pageview = TRUE > AND lower(uri_host) like '%.wikipedia.org' > AND user_agent like 'WikipediaApp%' > GROUP BY user_agent; > > > > == HIGHLIGHTING SESSION CASE FOR SPECIFIC VERSIONS OF THE APPS == > select CASE WHEN t2.userAgent LIKE 'WikipediaApp/2.0-r-2015-04-23%' THEN > 'Android' WHEN t2.userAgent LIKE 'WikipediaApp/4.1.2%' THEN 'iOS' END AS > 'ua', t1.event_action, t1.event_sharemode, t1.event_target, count(*) from > MobileWikiAppShareAFact_11331974 t1 inner join > MobileWikiAppShareAFact_11331974 t2 on t1.event_shareSessionToken = > t2.event_shareSessionToken where t1.timestamp > '20150512' and > t1.timestamp < '20150513' and t2.timestamp > '20150512' and t2.timestamp < > '20150513' and t1.event_action != 'highlight' and t2.event_action = > 'highlight' and (t2.userAgent like 'WikipediaApp/2.0-r-2015-04-23%' or > t2.userAgent like 'WikipediaApp/4.1.2%') group by ua, t1.event_action, > t1.event_sharemode, t1.event_target; > > > == NON-HIGHLIGHTING SESSION CASE FOR SPECIFIC VERSIONS OF THE APPS == > n.b., subtract the highlighting cases from the non-highlighting cases to > arrive at the default sharing behavior. Technically, inner joins can be > used to do more comprehensive session analysis, but the queries take a long > time. > > select CASE > WHEN userAgent LIKE 'WikipediaApp/2.0-r-2015-04-23%' THEN 'Android' > WHEN userAgent LIKE 'WikipediaApp/4.1.2%' THEN 'iOS' > END AS 'ua', event_action, event_sharemode, event_target, > count(*) from MobileWikiAppShareAFact_11331974 where timestamp > > '20150512' and timestamp < '20150513' and (userAgent like > 'WikipediaApp/2.0-r-2015-04-23%' or userAgent like 'WikipediaApp/4.1.2%') > group by ua, event_action, event_sharemode, event_target; > > -Adam > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > > >