Hi Team, I just wanted to update you on the results of something we internally referred to as the '*browse' *prototype. TLDR: as implemented the mobile 'browse by category' test did not drive significant engagement. In fact, as implemented, it seemed inferior to blue links. However, we started with a very rough and low-impact prototype, so a few tweaks would give us more definitive results.
Here is the doc from which I am pasting from below: https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizy...
Questions/comments welcome! Best,
J
Browse Prototype Results
Blue links in general https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o
Category tags https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz
Conclusion and Next Steps https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr
Do people want to browse by categories? https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19
Intro
As outlined in this doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing, the concept is a tag that allows readers to navigate WP via categories that are meaningful and populated in order of 'significance' (as determined by user input). The hypothesis:
-
users will want to navigate by category if there are fewer, more meaningful categories per page and those category pages showed the most ‘notable’ members first.
Again, see the full doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing to understand the premise.
Process
The first step was to validate: do users want to navigate via category? So we built a very lightweight prototype on mobile web, en wikipedia (stable, not beta) using hardcoded config variables, in the following categories ( ~4000 pages). Here we did not look into sub-categories with one exception (see T94732 https://phabricator.wikimedia.org/T94732 for details). There was also an error and 2 of the categories did not have tags implemented (struck through, below)
Category
Pagecount
NBA All Stars
400
American Politicians
818
Object-Oriented Programming Languages
164
European States
24
American Female Pop Singers
326
American drama television series
1048
Modern Painters
983
Landmarks in San Francisco, California
270
Here is how it appeared on the Alcatraz page
When the user clicked the tag, they were taken to a gather-like collection based on manually estimated relevance
(sorry cropped shot)
The category pages were designed to show the most relevant (as deemed by me) to the broadest audience, first. Here is the ordering: https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a...
This was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
We primarily measured a few things:
-
when a tag was seen by a user -
when a tag was clicked on by a user -
when a page in the new ‘category view’ was clicked on by a user
As a side effort, I looked to see if overall referrals from pages with tags went up--this was a timed intervention rather than an a/b test and given the click-thru on the tags, the impact would have been negligible anyway. This was confirmed by some very noisy results.
Results Blue links in general
One benefit of the side study mentioned in the previous paragraph is that I was able to generate a table that looked at the pages in question before we started the test that shows a ratio of total pageviews/pageviews referred by a page (estimate of how many links were opened from that page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have the pageview hourly table, a more robust analysis can tell us how categories differ in this regard:
Category
links clicked
#pvs
clicks/pvs
Category:20th-centuryAmericanpoliticians
761
1243
61%
Category:Americandramatelevisionseries
5981
8844
68%
Category:Americanfemalepopsingers
2502
4280
58%
Category:LandmarksinSanFrancisco,
104
287
36%
Category:Modernpainters
136
369
37%
Category:NationalBasketballAssociationAll-Stars
1908
3341
57%
Category:Object-orientedprogramminglanguages
48
181
27%
Category:WesternEurope
657
1221
54%
Grand Total
12099
19766
50%
You can see here that for pages in the category ‘Landmarks in San Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are generated on average.
I do not have the original queries for this handy, but can dig them up if you’re really interested.
Category tags
Full data and queries here: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDM...
The tags themselves generated an average click-through rate of .18%. Given the overall click thru rate on the pages estimated above ~50%, this single tag is not driving anything significant. Furthermore, given Leila and Bob’s paper suggest that this is performing no better than a mid-article click--given the mobile web sections are collapsed, I would need to understand more about their method to know just how to interpret their results against our mobile-web only implementation. Furthermore, our click through rate used the number of times the tag appeared on screen as the denominator, whereas their research looked at overall pageviews.
This being noted, the tag was implemented to be as obscure as possible to establish a baseline. Furthermore, any feature like this would probably be different in the following ways:
-
each page would be in 1-4 tag groups (as opposed to just 1) -
each page would be tagged, creating the expectation on the part of the user that this was something to look for -
presumably the categories could be implemented as a menu item as opposed to being buried at the bottom of the page (and competing with features like read more. -
Using the learnings from ‘read more’ tags with images or buttons would likely fare much better.
The follow graph shows:
-
number of impressions on the right axis -
click-thru-rate on the left-axis.
When you look at click through rates on the ‘category’ pages themselves, you see that they average at 41% (Chart below) Meaning that for every 10 times a user visited a category page, there were 4.1 clicks to one of those pages as a result.
Here is the same broken up by category:
Each ‘category’ page here had at least 400 visits, and you can see that the interest seems to vary dramatically across categories. It is worth noting that the top three categories here are the ones with the fewest entities. Each list, however, was capped at ~50 articles, so it is unclear what might be causing this effect, if it is real.
As mentioned above, the average article page has an overall click rate of 50%. So this page of categories did not have the click-through rate that a page has. However, this page had summaries of each of the pages, so it could be that users were generating value beyond what a blue link would provide. A live-user test of Gather collections, from whom this format was borrowed, suggested that the format used up too much vertical space on each article and was hard to flip through. Shortening the amount of text or image space might be something to try to make the page more useful
Conclusion and Next StepsProcess
-
This was the first time I am aware of that we ran a live prototype and learn something without building a scalable solution. Win -
Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but the chronological time for pushing to stable took a quarter. Room for improvement -
The time to analysis was almost 2 quarters, due to a lack of data analysis support (I ran the initial analysis within 2 weeks of launch, during paternity leave, but was unable to go back and get it ready to distribute for 3 months). Room for improvement--possibly solved by additional Data Analyst.
This experiment was not designed to answer questions definitively in one round, but with the understanding that multiple iterations would allow us to fully answer our questions.
The long turn-around time, particularly around analysis and communication, meant that tweaking a variable to test the conclusions or the new questions that arosee below will involve a whole lot more work and effort than if we had been able to explore modifications within a few weeks of the initial launch.
Do people want to browse by categories?
Category tags at the bottom of the mobile web page in a dull gray background that lead to manually curated categories are not a killer feature :)
I would be reluctant to say that this means users are not interested in browsing by category, however. For instance, it is likely that
-
users did not notice the tag, even if it appeared on screen -
users are accustomed to our current category tags on desktop and not interested in that experience -
users who did like the tag were unlikely to find another page that had it--there was no feedback mechanism by which the improved category page would drive additional tag interactions -
the browse experience created was not ideal
If we decide to pursue what is currently termed “cascade c: update ux”, I would like to proceed with more tests in this arena, by altering the appearance and position of the tags, and by improving the flow of the ‘category’ pages. If we choose a different strategy, hopefully other teams can build off of what was learned here.
Thanks a lot for the detailed report Jon.
I've parsed it and posted it to https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so that can keep it more accessible than the mailing list archive https://lists.wikimedia.org/pipermail/mobile-l/2015-October/009827.html.
Any help with formatting or text corrections would be appreciated.
On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Team, I just wanted to update you on the results of something we internally referred to as the '*browse' *prototype. TLDR: as implemented the mobile 'browse by category' test did not drive significant engagement. In fact, as implemented, it seemed inferior to blue links. However, we started with a very rough and low-impact prototype, so a few tweaks would give us more definitive results.
Here is the doc from which I am pasting from below: https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizy...
Questions/comments welcome! Best,
J
Browse Prototype Results
Blue links in general https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o
Category tags https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz
Conclusion and Next Steps https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr
Do people want to browse by categories? https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19
Intro
As outlined in this doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing, the concept is a tag that allows readers to navigate WP via categories that are meaningful and populated in order of 'significance' (as determined by user input). The hypothesis:
users will want to navigate by category if there are fewer, more meaningful categories per page and those category pages showed the most ‘notable’ members first.
Again, see the full doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing to understand the premise.
Process
The first step was to validate: do users want to navigate via category? So we built a very lightweight prototype on mobile web, en wikipedia (stable, not beta) using hardcoded config variables, in the following categories ( ~4000 pages). Here we did not look into sub-categories with one exception (see T94732 https://phabricator.wikimedia.org/T94732 for details). There was also an error and 2 of the categories did not have tags implemented (struck through, below)
Category
Pagecount
NBA All Stars
400
American Politicians
818
Object-Oriented Programming Languages
164
European States
24
American Female Pop Singers
326
American drama television series
1048
Modern Painters
983
Landmarks in San Francisco, California
270
Here is how it appeared on the Alcatraz page
When the user clicked the tag, they were taken to a gather-like collection based on manually estimated relevance
(sorry cropped shot)
The category pages were designed to show the most relevant (as deemed by me) to the broadest audience, first. Here is the ordering: https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a...
This was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
We primarily measured a few things:
when a tag was seen by a user
when a tag was clicked on by a user
when a page in the new ‘category view’ was clicked on by a user
As a side effort, I looked to see if overall referrals from pages with tags went up--this was a timed intervention rather than an a/b test and given the click-thru on the tags, the impact would have been negligible anyway. This was confirmed by some very noisy results.
Results Blue links in general
One benefit of the side study mentioned in the previous paragraph is that I was able to generate a table that looked at the pages in question before we started the test that shows a ratio of total pageviews/pageviews referred by a page (estimate of how many links were opened from that page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have the pageview hourly table, a more robust analysis can tell us how categories differ in this regard:
Category
links clicked
#pvs
clicks/pvs
Category:20th-centuryAmericanpoliticians
761
1243
61%
Category:Americandramatelevisionseries
5981
8844
68%
Category:Americanfemalepopsingers
2502
4280
58%
Category:LandmarksinSanFrancisco,
104
287
36%
Category:Modernpainters
136
369
37%
Category:NationalBasketballAssociationAll-Stars
1908
3341
57%
Category:Object-orientedprogramminglanguages
48
181
27%
Category:WesternEurope
657
1221
54%
Grand Total
12099
19766
50%
You can see here that for pages in the category ‘Landmarks in San Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are generated on average.
I do not have the original queries for this handy, but can dig them up if you’re really interested.
Category tags
Full data and queries here: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDM...
The tags themselves generated an average click-through rate of .18%. Given the overall click thru rate on the pages estimated above ~50%, this single tag is not driving anything significant. Furthermore, given Leila and Bob’s paper suggest that this is performing no better than a mid-article click--given the mobile web sections are collapsed, I would need to understand more about their method to know just how to interpret their results against our mobile-web only implementation. Furthermore, our click through rate used the number of times the tag appeared on screen as the denominator, whereas their research looked at overall pageviews.
This being noted, the tag was implemented to be as obscure as possible to establish a baseline. Furthermore, any feature like this would probably be different in the following ways:
each page would be in 1-4 tag groups (as opposed to just 1)
each page would be tagged, creating the expectation on the part of the user that this was something to look for
presumably the categories could be implemented as a menu item as opposed to being buried at the bottom of the page (and competing with features like read more.
Using the learnings from ‘read more’ tags with images or buttons would likely fare much better.
The follow graph shows:
number of impressions on the right axis
click-thru-rate on the left-axis.
When you look at click through rates on the ‘category’ pages themselves, you see that they average at 41% (Chart below) Meaning that for every 10 times a user visited a category page, there were 4.1 clicks to one of those pages as a result.
Here is the same broken up by category:
Each ‘category’ page here had at least 400 visits, and you can see that the interest seems to vary dramatically across categories. It is worth noting that the top three categories here are the ones with the fewest entities. Each list, however, was capped at ~50 articles, so it is unclear what might be causing this effect, if it is real.
As mentioned above, the average article page has an overall click rate of 50%. So this page of categories did not have the click-through rate that a page has. However, this page had summaries of each of the pages, so it could be that users were generating value beyond what a blue link would provide. A live-user test of Gather collections, from whom this format was borrowed, suggested that the format used up too much vertical space on each article and was hard to flip through. Shortening the amount of text or image space might be something to try to make the page more useful
Conclusion and Next StepsProcess
This was the first time I am aware of that we ran a live prototype and learn something without building a scalable solution. Win
Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but the chronological time for pushing to stable took a quarter. Room for improvement
The time to analysis was almost 2 quarters, due to a lack of data analysis support (I ran the initial analysis within 2 weeks of launch, during paternity leave, but was unable to go back and get it ready to distribute for 3 months). Room for improvement--possibly solved by additional Data Analyst.
This experiment was not designed to answer questions definitively in one round, but with the understanding that multiple iterations would allow us to fully answer our questions.
The long turn-around time, particularly around analysis and communication, meant that tweaking a variable to test the conclusions or the new questions that arosee below will involve a whole lot more work and effort than if we had been able to explore modifications within a few weeks of the initial launch.
Do people want to browse by categories?
Category tags at the bottom of the mobile web page in a dull gray background that lead to manually curated categories are not a killer feature :)
I would be reluctant to say that this means users are not interested in browsing by category, however. For instance, it is likely that
users did not notice the tag, even if it appeared on screen
users are accustomed to our current category tags on desktop and not interested in that experience
users who did like the tag were unlikely to find another page that had it--there was no feedback mechanism by which the improved category page would drive additional tag interactions
the browse experience created was not ideal
If we decide to pursue what is currently termed “cascade c: update ux”, I would like to proceed with more tests in this arena, by altering the appearance and position of the tags, and by improving the flow of the ‘category’ pages. If we choose a different strategy, hopefully other teams can build off of what was learned here.
reading-wmf mailing list reading-wmf@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/reading-wmf
Thanks, Joaquin!
On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Thanks a lot for the detailed report Jon.
I've parsed it and posted it to https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so that can keep it more accessible than the mailing list archive https://lists.wikimedia.org/pipermail/mobile-l/2015-October/009827.html.
Any help with formatting or text corrections would be appreciated.
On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Team, I just wanted to update you on the results of something we internally referred to as the '*browse' *prototype. TLDR: as implemented the mobile 'browse by category' test did not drive significant engagement. In fact, as implemented, it seemed inferior to blue links. However, we started with a very rough and low-impact prototype, so a few tweaks would give us more definitive results.
Here is the doc from which I am pasting from below: https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizy...
Questions/comments welcome! Best,
J
Browse Prototype Results
Blue links in general https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o
Category tags https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz
Conclusion and Next Steps https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr
Do people want to browse by categories? https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19
Intro
As outlined in this doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing, the concept is a tag that allows readers to navigate WP via categories that are meaningful and populated in order of 'significance' (as determined by user input). The hypothesis:
users will want to navigate by category if there are fewer, more meaningful categories per page and those category pages showed the most ‘notable’ members first.
Again, see the full doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing to understand the premise.
Process
The first step was to validate: do users want to navigate via category? So we built a very lightweight prototype on mobile web, en wikipedia (stable, not beta) using hardcoded config variables, in the following categories ( ~4000 pages). Here we did not look into sub-categories with one exception (see T94732 https://phabricator.wikimedia.org/T94732 for details). There was also an error and 2 of the categories did not have tags implemented (struck through, below)
Category
Pagecount
NBA All Stars
400
American Politicians
818
Object-Oriented Programming Languages
164
European States
24
American Female Pop Singers
326
American drama television series
1048
Modern Painters
983
Landmarks in San Francisco, California
270
Here is how it appeared on the Alcatraz page
When the user clicked the tag, they were taken to a gather-like collection based on manually estimated relevance
(sorry cropped shot)
The category pages were designed to show the most relevant (as deemed by me) to the broadest audience, first. Here is the ordering: https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a...
This was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
We primarily measured a few things:
when a tag was seen by a user
when a tag was clicked on by a user
when a page in the new ‘category view’ was clicked on by a user
As a side effort, I looked to see if overall referrals from pages with tags went up--this was a timed intervention rather than an a/b test and given the click-thru on the tags, the impact would have been negligible anyway. This was confirmed by some very noisy results.
Results Blue links in general
One benefit of the side study mentioned in the previous paragraph is that I was able to generate a table that looked at the pages in question before we started the test that shows a ratio of total pageviews/pageviews referred by a page (estimate of how many links were opened from that page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have the pageview hourly table, a more robust analysis can tell us how categories differ in this regard:
Category
links clicked
#pvs
clicks/pvs
Category:20th-centuryAmericanpoliticians
761
1243
61%
Category:Americandramatelevisionseries
5981
8844
68%
Category:Americanfemalepopsingers
2502
4280
58%
Category:LandmarksinSanFrancisco,
104
287
36%
Category:Modernpainters
136
369
37%
Category:NationalBasketballAssociationAll-Stars
1908
3341
57%
Category:Object-orientedprogramminglanguages
48
181
27%
Category:WesternEurope
657
1221
54%
Grand Total
12099
19766
50%
You can see here that for pages in the category ‘Landmarks in San Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are generated on average.
I do not have the original queries for this handy, but can dig them up if you’re really interested.
Category tags
Full data and queries here: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDM...
The tags themselves generated an average click-through rate of .18%. Given the overall click thru rate on the pages estimated above ~50%, this single tag is not driving anything significant. Furthermore, given Leila and Bob’s paper suggest that this is performing no better than a mid-article click--given the mobile web sections are collapsed, I would need to understand more about their method to know just how to interpret their results against our mobile-web only implementation. Furthermore, our click through rate used the number of times the tag appeared on screen as the denominator, whereas their research looked at overall pageviews.
This being noted, the tag was implemented to be as obscure as possible to establish a baseline. Furthermore, any feature like this would probably be different in the following ways:
each page would be in 1-4 tag groups (as opposed to just 1)
each page would be tagged, creating the expectation on the part of the user that this was something to look for
presumably the categories could be implemented as a menu item as opposed to being buried at the bottom of the page (and competing with features like read more.
Using the learnings from ‘read more’ tags with images or buttons would likely fare much better.
The follow graph shows:
number of impressions on the right axis
click-thru-rate on the left-axis.
When you look at click through rates on the ‘category’ pages themselves, you see that they average at 41% (Chart below) Meaning that for every 10 times a user visited a category page, there were 4.1 clicks to one of those pages as a result.
Here is the same broken up by category:
Each ‘category’ page here had at least 400 visits, and you can see that the interest seems to vary dramatically across categories. It is worth noting that the top three categories here are the ones with the fewest entities. Each list, however, was capped at ~50 articles, so it is unclear what might be causing this effect, if it is real.
As mentioned above, the average article page has an overall click rate of 50%. So this page of categories did not have the click-through rate that a page has. However, this page had summaries of each of the pages, so it could be that users were generating value beyond what a blue link would provide. A live-user test of Gather collections, from whom this format was borrowed, suggested that the format used up too much vertical space on each article and was hard to flip through. Shortening the amount of text or image space might be something to try to make the page more useful
Conclusion and Next StepsProcess
This was the first time I am aware of that we ran a live prototype and learn something without building a scalable solution. Win
Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but the chronological time for pushing to stable took a quarter. Room for improvement
The time to analysis was almost 2 quarters, due to a lack of data analysis support (I ran the initial analysis within 2 weeks of launch, during paternity leave, but was unable to go back and get it ready to distribute for 3 months). Room for improvement--possibly solved by additional Data Analyst.
This experiment was not designed to answer questions definitively in one round, but with the understanding that multiple iterations would allow us to fully answer our questions.
The long turn-around time, particularly around analysis and communication, meant that tweaking a variable to test the conclusions or the new questions that arosee below will involve a whole lot more work and effort than if we had been able to explore modifications within a few weeks of the initial launch.
Do people want to browse by categories?
Category tags at the bottom of the mobile web page in a dull gray background that lead to manually curated categories are not a killer feature :)
I would be reluctant to say that this means users are not interested in browsing by category, however. For instance, it is likely that
users did not notice the tag, even if it appeared on screen
users are accustomed to our current category tags on desktop and not interested in that experience
users who did like the tag were unlikely to find another page that had it--there was no feedback mechanism by which the improved category page would drive additional tag interactions
the browse experience created was not ideal
If we decide to pursue what is currently termed “cascade c: update ux”, I would like to proceed with more tests in this arena, by altering the appearance and position of the tags, and by improving the flow of the ‘category’ pages. If we choose a different strategy, hopefully other teams can build off of what was learned here.
reading-wmf mailing list reading-wmf@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/reading-wmf
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Great experiment! A couple questions/comments:
1. The % clickthrough per category shows SF Landmarks at 120%. Is that correct, and if so, what does it mean? 2. As a big believer in the power of categories as a driver for engagement, I would love to see more variations of this experiment w/ different placements, in a feed, different categories, add'n of portals, as a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep dive educational experience) 3. Also loved the win/needs-improvement breakdown at the end
Again, nice work!
On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz jkatz@wikimedia.org wrote:
Thanks, Joaquin!
On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Thanks a lot for the detailed report Jon.
I've parsed it and posted it to https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so that can keep it more accessible than the mailing list archive https://lists.wikimedia.org/pipermail/mobile-l/2015-October/009827.html .
Any help with formatting or text corrections would be appreciated.
On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Team, I just wanted to update you on the results of something we internally referred to as the '*browse' *prototype. TLDR: as implemented the mobile 'browse by category' test did not drive significant engagement. In fact, as implemented, it seemed inferior to blue links. However, we started with a very rough and low-impact prototype, so a few tweaks would give us more definitive results.
Here is the doc from which I am pasting from below: https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizy...
Questions/comments welcome! Best,
J
Browse Prototype Results
Blue links in general https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o
Category tags https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz
Conclusion and Next Steps https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr
Do people want to browse by categories? https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19
Intro
As outlined in this doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing, the concept is a tag that allows readers to navigate WP via categories that are meaningful and populated in order of 'significance' (as determined by user input). The hypothesis:
users will want to navigate by category if there are fewer, more meaningful categories per page and those category pages showed the most ‘notable’ members first.
Again, see the full doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing to understand the premise.
Process
The first step was to validate: do users want to navigate via category? So we built a very lightweight prototype on mobile web, en wikipedia (stable, not beta) using hardcoded config variables, in the following categories ( ~4000 pages). Here we did not look into sub-categories with one exception (see T94732 https://phabricator.wikimedia.org/T94732 for details). There was also an error and 2 of the categories did not have tags implemented (struck through, below)
Category
Pagecount
NBA All Stars
400
American Politicians
818
Object-Oriented Programming Languages
164
European States
24
American Female Pop Singers
326
American drama television series
1048
Modern Painters
983
Landmarks in San Francisco, California
270
Here is how it appeared on the Alcatraz page
When the user clicked the tag, they were taken to a gather-like collection based on manually estimated relevance
(sorry cropped shot)
The category pages were designed to show the most relevant (as deemed by me) to the broadest audience, first. Here is the ordering: https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a...
This was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
We primarily measured a few things:
when a tag was seen by a user
when a tag was clicked on by a user
when a page in the new ‘category view’ was clicked on by a user
As a side effort, I looked to see if overall referrals from pages with tags went up--this was a timed intervention rather than an a/b test and given the click-thru on the tags, the impact would have been negligible anyway. This was confirmed by some very noisy results.
Results Blue links in general
One benefit of the side study mentioned in the previous paragraph is that I was able to generate a table that looked at the pages in question before we started the test that shows a ratio of total pageviews/pageviews referred by a page (estimate of how many links were opened from that page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have the pageview hourly table, a more robust analysis can tell us how categories differ in this regard:
Category
links clicked
#pvs
clicks/pvs
Category:20th-centuryAmericanpoliticians
761
1243
61%
Category:Americandramatelevisionseries
5981
8844
68%
Category:Americanfemalepopsingers
2502
4280
58%
Category:LandmarksinSanFrancisco,
104
287
36%
Category:Modernpainters
136
369
37%
Category:NationalBasketballAssociationAll-Stars
1908
3341
57%
Category:Object-orientedprogramminglanguages
48
181
27%
Category:WesternEurope
657
1221
54%
Grand Total
12099
19766
50%
You can see here that for pages in the category ‘Landmarks in San Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are generated on average.
I do not have the original queries for this handy, but can dig them up if you’re really interested.
Category tags
Full data and queries here: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDM...
The tags themselves generated an average click-through rate of .18%. Given the overall click thru rate on the pages estimated above ~50%, this single tag is not driving anything significant. Furthermore, given Leila and Bob’s paper suggest that this is performing no better than a mid-article click--given the mobile web sections are collapsed, I would need to understand more about their method to know just how to interpret their results against our mobile-web only implementation. Furthermore, our click through rate used the number of times the tag appeared on screen as the denominator, whereas their research looked at overall pageviews.
This being noted, the tag was implemented to be as obscure as possible to establish a baseline. Furthermore, any feature like this would probably be different in the following ways:
each page would be in 1-4 tag groups (as opposed to just 1)
each page would be tagged, creating the expectation on the part of the user that this was something to look for
presumably the categories could be implemented as a menu item as opposed to being buried at the bottom of the page (and competing with features like read more.
Using the learnings from ‘read more’ tags with images or buttons would likely fare much better.
The follow graph shows:
number of impressions on the right axis
click-thru-rate on the left-axis.
When you look at click through rates on the ‘category’ pages themselves, you see that they average at 41% (Chart below) Meaning that for every 10 times a user visited a category page, there were 4.1 clicks to one of those pages as a result.
Here is the same broken up by category:
Each ‘category’ page here had at least 400 visits, and you can see that the interest seems to vary dramatically across categories. It is worth noting that the top three categories here are the ones with the fewest entities. Each list, however, was capped at ~50 articles, so it is unclear what might be causing this effect, if it is real.
As mentioned above, the average article page has an overall click rate of 50%. So this page of categories did not have the click-through rate that a page has. However, this page had summaries of each of the pages, so it could be that users were generating value beyond what a blue link would provide. A live-user test of Gather collections, from whom this format was borrowed, suggested that the format used up too much vertical space on each article and was hard to flip through. Shortening the amount of text or image space might be something to try to make the page more useful
Conclusion and Next StepsProcess
This was the first time I am aware of that we ran a live prototype and learn something without building a scalable solution. Win
Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but the chronological time for pushing to stable took a quarter. Room for improvement
The time to analysis was almost 2 quarters, due to a lack of data analysis support (I ran the initial analysis within 2 weeks of launch, during paternity leave, but was unable to go back and get it ready to distribute for 3 months). Room for improvement--possibly solved by additional Data Analyst.
This experiment was not designed to answer questions definitively in one round, but with the understanding that multiple iterations would allow us to fully answer our questions.
The long turn-around time, particularly around analysis and communication, meant that tweaking a variable to test the conclusions or the new questions that arosee below will involve a whole lot more work and effort than if we had been able to explore modifications within a few weeks of the initial launch.
Do people want to browse by categories?
Category tags at the bottom of the mobile web page in a dull gray background that lead to manually curated categories are not a killer feature :)
I would be reluctant to say that this means users are not interested in browsing by category, however. For instance, it is likely that
users did not notice the tag, even if it appeared on screen
users are accustomed to our current category tags on desktop and not interested in that experience
users who did like the tag were unlikely to find another page that had it--there was no feedback mechanism by which the improved category page would drive additional tag interactions
the browse experience created was not ideal
If we decide to pursue what is currently termed “cascade c: update ux”, I would like to proceed with more tests in this arena, by altering the appearance and position of the tags, and by improving the flow of the ‘category’ pages. If we choose a different strategy, hopefully other teams can build off of what was learned here.
reading-wmf mailing list reading-wmf@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/reading-wmf
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Jon thanks so much for writing this up and thanks Joaquin for putting it on the wiki (you beat me to it :))
I'm a little confused to the meaning of "tag impression" - does it mean they rendered or that the user saw them?
If the latter... https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse#/media... suggests that 50% of the time a reader saw a tag (.45% of the time) (there was a tag impression) they clicked on them (0.18%). Am I misunderstanding what tag impression means?
Is this a fair summary?: "When the tags were __visible__ to the user, 30-50% of the time they clicked through to the "category page". On this page only 40% of visits resulted in a click to an article in the list.
To me this is significant traffic... especially if made more visible. I personally would expect normal link traffic to be higher and for this to be a new source of engagement.
Can we clarify this on the wiki page and in this thread as I fear I'm misunderstanding something..?
Other questions: * What was the number of clicks to tags per visit? (were being opening new tabs or clicking on one?)
On Tue, Oct 13, 2015 at 8:36 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
Great experiment! A couple questions/comments:
- The % clickthrough per category shows SF Landmarks at 120%. Is that
correct, and if so, what does it mean? 2. As a big believer in the power of categories as a driver for engagement, I would love to see more variations of this experiment w/ different placements, in a feed, different categories, add'n of portals, as a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep dive educational experience) 3. Also loved the win/needs-improvement breakdown at the end
Again, nice work!
On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz jkatz@wikimedia.org wrote:
Thanks, Joaquin!
On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Thanks a lot for the detailed report Jon.
I've parsed it and posted it to https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so that can keep it more accessible than the mailing list archive https://lists.wikimedia.org/pipermail/mobile-l/2015-October/009827.html .
Any help with formatting or text corrections would be appreciated.
On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Team, I just wanted to update you on the results of something we internally referred to as the '*browse' *prototype. TLDR: as implemented the mobile 'browse by category' test did not drive significant engagement. In fact, as implemented, it seemed inferior to blue links. However, we started with a very rough and low-impact prototype, so a few tweaks would give us more definitive results.
Here is the doc from which I am pasting from below: https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizy...
Questions/comments welcome! Best,
J
Browse Prototype Results
Blue links in general https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o
Category tags https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz
Conclusion and Next Steps https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr
Do people want to browse by categories? https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19
Intro
As outlined in this doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing, the concept is a tag that allows readers to navigate WP via categories that are meaningful and populated in order of 'significance' (as determined by user input). The hypothesis:
users will want to navigate by category if there are fewer, more meaningful categories per page and those category pages showed the most ‘notable’ members first.
Again, see the full doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing to understand the premise.
Process
The first step was to validate: do users want to navigate via category? So we built a very lightweight prototype on mobile web, en wikipedia (stable, not beta) using hardcoded config variables, in the following categories ( ~4000 pages). Here we did not look into sub-categories with one exception (see T94732 https://phabricator.wikimedia.org/T94732 for details). There was also an error and 2 of the categories did not have tags implemented (struck through, below)
Category
Pagecount
NBA All Stars
400
American Politicians
818
Object-Oriented Programming Languages
164
European States
24
American Female Pop Singers
326
American drama television series
1048
Modern Painters
983
Landmarks in San Francisco, California
270
Here is how it appeared on the Alcatraz page
When the user clicked the tag, they were taken to a gather-like collection based on manually estimated relevance
(sorry cropped shot)
The category pages were designed to show the most relevant (as deemed by me) to the broadest audience, first. Here is the ordering: https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a...
This was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
We primarily measured a few things:
when a tag was seen by a user
when a tag was clicked on by a user
when a page in the new ‘category view’ was clicked on by a user
As a side effort, I looked to see if overall referrals from pages with tags went up--this was a timed intervention rather than an a/b test and given the click-thru on the tags, the impact would have been negligible anyway. This was confirmed by some very noisy results.
Results Blue links in general
One benefit of the side study mentioned in the previous paragraph is that I was able to generate a table that looked at the pages in question before we started the test that shows a ratio of total pageviews/pageviews referred by a page (estimate of how many links were opened from that page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have the pageview hourly table, a more robust analysis can tell us how categories differ in this regard:
Category
links clicked
#pvs
clicks/pvs
Category:20th-centuryAmericanpoliticians
761
1243
61%
Category:Americandramatelevisionseries
5981
8844
68%
Category:Americanfemalepopsingers
2502
4280
58%
Category:LandmarksinSanFrancisco,
104
287
36%
Category:Modernpainters
136
369
37%
Category:NationalBasketballAssociationAll-Stars
1908
3341
57%
Category:Object-orientedprogramminglanguages
48
181
27%
Category:WesternEurope
657
1221
54%
Grand Total
12099
19766
50%
You can see here that for pages in the category ‘Landmarks in San Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are generated on average.
I do not have the original queries for this handy, but can dig them up if you’re really interested.
Category tags
Full data and queries here: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDM...
The tags themselves generated an average click-through rate of .18%. Given the overall click thru rate on the pages estimated above ~50%, this single tag is not driving anything significant. Furthermore, given Leila and Bob’s paper suggest that this is performing no better than a mid-article click--given the mobile web sections are collapsed, I would need to understand more about their method to know just how to interpret their results against our mobile-web only implementation. Furthermore, our click through rate used the number of times the tag appeared on screen as the denominator, whereas their research looked at overall pageviews.
This being noted, the tag was implemented to be as obscure as possible to establish a baseline. Furthermore, any feature like this would probably be different in the following ways:
each page would be in 1-4 tag groups (as opposed to just 1)
each page would be tagged, creating the expectation on the part of the user that this was something to look for
presumably the categories could be implemented as a menu item as opposed to being buried at the bottom of the page (and competing with features like read more.
Using the learnings from ‘read more’ tags with images or buttons would likely fare much better.
The follow graph shows:
number of impressions on the right axis
click-thru-rate on the left-axis.
When you look at click through rates on the ‘category’ pages themselves, you see that they average at 41% (Chart below) Meaning that for every 10 times a user visited a category page, there were 4.1 clicks to one of those pages as a result.
Here is the same broken up by category:
Each ‘category’ page here had at least 400 visits, and you can see that the interest seems to vary dramatically across categories. It is worth noting that the top three categories here are the ones with the fewest entities. Each list, however, was capped at ~50 articles, so it is unclear what might be causing this effect, if it is real.
As mentioned above, the average article page has an overall click rate of 50%. So this page of categories did not have the click-through rate that a page has. However, this page had summaries of each of the pages, so it could be that users were generating value beyond what a blue link would provide. A live-user test of Gather collections, from whom this format was borrowed, suggested that the format used up too much vertical space on each article and was hard to flip through. Shortening the amount of text or image space might be something to try to make the page more useful
Conclusion and Next StepsProcess
This was the first time I am aware of that we ran a live prototype and learn something without building a scalable solution. Win
Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but the chronological time for pushing to stable took a quarter. Room for improvement
The time to analysis was almost 2 quarters, due to a lack of data analysis support (I ran the initial analysis within 2 weeks of launch, during paternity leave, but was unable to go back and get it ready to distribute for 3 months). Room for improvement--possibly solved by additional Data Analyst.
This experiment was not designed to answer questions definitively in one round, but with the understanding that multiple iterations would allow us to fully answer our questions.
The long turn-around time, particularly around analysis and communication, meant that tweaking a variable to test the conclusions or the new questions that arosee below will involve a whole lot more work and effort than if we had been able to explore modifications within a few weeks of the initial launch.
Do people want to browse by categories?
Category tags at the bottom of the mobile web page in a dull gray background that lead to manually curated categories are not a killer feature :)
I would be reluctant to say that this means users are not interested in browsing by category, however. For instance, it is likely that
users did not notice the tag, even if it appeared on screen
users are accustomed to our current category tags on desktop and not interested in that experience
users who did like the tag were unlikely to find another page that had it--there was no feedback mechanism by which the improved category page would drive additional tag interactions
the browse experience created was not ideal
If we decide to pursue what is currently termed “cascade c: update ux”, I would like to proceed with more tests in this arena, by altering the appearance and position of the tags, and by improving the flow of the ‘category’ pages. If we choose a different strategy, hopefully other teams can build off of what was learned here.
reading-wmf mailing list reading-wmf@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/reading-wmf
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Thanks for thinking through this, Jon. I think I made something unclear--responses inline. In green to make reading easier:
On Tue, Oct 13, 2015 at 8:58 AM, Jon Robson jrobson@wikimedia.org wrote:
Jon thanks so much for writing this up and thanks Joaquin for putting it on the wiki (you beat me to it :))
I'm a little confused to the meaning of "tag impression" - does it mean they rendered or that the user saw them?
User saw them, or, more specifically, appeared on screen.
If the latter...
https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse#/media... suggests that 50% of the time a reader saw a tag (.45% of the time) (there was a tag impression) they clicked on them (0.18%). Am I misunderstanding what tag impression means?
Sorry, this graph is a little confusing because it has 2 axes. The red (impressions) corresponds to the right axis and the blue (% click-through) is the left- axis. So, the way to read it is that there were ~50,000 impressions a day, and the click-thru rate was .18%. This means that for every ~1000 times a tag appeared to a user, ~2 clicks occurred. The question of what % of pageviews resulted in an 'impression' is a really good one - I think it could be easily looked up using the pageview_hourly table.
Is this a fair summary?: "When the tags were __visible__ to the user, 30-50% of the time they clicked through to the "category page". On this page only 40% of visits resulted in a click to an article in the list.
No--for reasons explained above.
To me this is significant traffic... especially if made more visible. I personally would expect normal link traffic to be higher and for this to be a new source of engagement.
Yes, it would be hard to get higher click-through than our current engagement model, but anything additive represents not only massive numbers, but an opportunity to have our users increasingly think of us as a place to learn (rather than a simple question and answer)
Can we clarify this on the wiki page and in this thread as I fear I'm misunderstanding something..?
Yes--will add clarity to the graph on wiki. It might be awhile, however, before I have the brain and time to execute the query your question brought up of "what % of total pages lead to the tag being shown"? This will just as easily be answered by the read more data you will be implementing.
Other questions:
- What was the number of clicks to tags per visit? (were being opening new
tabs or clicking on one?)
I don't understand this question. I know we can't track 'visits'.
Thanks for thinking through! Responses inline On Tue, Oct 13, 2015 at 8:36 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
Great experiment! A couple questions/comments:
- The % clickthrough per category shows SF Landmarks at 120%. Is that
correct, and if so, what does it mean?
It means that for every 10 times the SF Landmarks list was visited from a
tag, there were 12 clicks on to other articles (this is possible because a user can visit 2 or more articles from the page by clicking on the 'back' button in their browser)
- As a big believer in the power of categories as a driver for
engagement, I would love to see more variations of this experiment w/ different placements, in a feed, different categories, add'n of portals, as a FTUE, etc. (likely to have a great deal of overlap w/ cascade D: deep dive educational experience)
Me too! it is very hard to learn anything from a first stab like this
- Also loved the win/needs-improvement breakdown at the end
Thanks!
On Tue, Oct 13, 2015 at 11:23 AM, Jon Katz jkatz@wikimedia.org wrote:
Thanks, Joaquin!
On Tue, Oct 13, 2015 at 4:32 AM, Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Thanks a lot for the detailed report Jon.
I've parsed it and posted it to https://www.mediawiki.org/wiki/Reading/Web/Projects/Categories_Browse so that can keep it more accessible than the mailing list archive https://lists.wikimedia.org/pipermail/mobile-l/2015-October/009827.html .
Any help with formatting or text corrections would be appreciated.
On Sun, Oct 11, 2015 at 8:32 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Team, I just wanted to update you on the results of something we internally referred to as the '*browse' *prototype. TLDR: as implemented the mobile 'browse by category' test did not drive significant engagement. In fact, as implemented, it seemed inferior to blue links. However, we started with a very rough and low-impact prototype, so a few tweaks would give us more definitive results.
Here is the doc from which I am pasting from below: https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizy...
Questions/comments welcome! Best,
J
Browse Prototype Results
Blue links in general https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.8nn07h675j3o
Category tags https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.gagragojxpiz
Conclusion and Next Steps https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.z3p82tg8enr
Do people want to browse by categories? https://docs.google.com/document/d/1Mqw-awAcp01IcLhHPsHmWsqaAyK1l2-w_LMDtizyFQ4/edit#heading=h.9ksw2zvt8q19
Intro
As outlined in this doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing, the concept is a tag that allows readers to navigate WP via categories that are meaningful and populated in order of 'significance' (as determined by user input). The hypothesis:
users will want to navigate by category if there are fewer, more meaningful categories per page and those category pages showed the most ‘notable’ members first.
Again, see the full doc https://docs.google.com/presentation/d/1ZssE8G0P5WVg8XmkBTi5G3n4OdLHPFGWZDZFW5_DSS0/edit?usp=sharing to understand the premise.
Process
The first step was to validate: do users want to navigate via category? So we built a very lightweight prototype on mobile web, en wikipedia (stable, not beta) using hardcoded config variables, in the following categories ( ~4000 pages). Here we did not look into sub-categories with one exception (see T94732 https://phabricator.wikimedia.org/T94732 for details). There was also an error and 2 of the categories did not have tags implemented (struck through, below)
Category
Pagecount
NBA All Stars
400
American Politicians
818
Object-Oriented Programming Languages
164
European States
24
American Female Pop Singers
326
American drama television series
1048
Modern Painters
983
Landmarks in San Francisco, California
270
Here is how it appeared on the Alcatraz page
When the user clicked the tag, they were taken to a gather-like collection based on manually estimated relevance
(sorry cropped shot)
The category pages were designed to show the most relevant (as deemed by me) to the broadest audience, first. Here is the ordering: https://docs.google.com/spreadsheets/d/12xLXQsH1zcg6E8lDuSonumZNdBvfaBuHOS1a...
This was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
We primarily measured a few things:
when a tag was seen by a user
when a tag was clicked on by a user
when a page in the new ‘category view’ was clicked on by a user
As a side effort, I looked to see if overall referrals from pages with tags went up--this was a timed intervention rather than an a/b test and given the click-thru on the tags, the impact would have been negligible anyway. This was confirmed by some very noisy results.
Results Blue links in general
One benefit of the side study mentioned in the previous paragraph is that I was able to generate a table that looked at the pages in question before we started the test that shows a ratio of total pageviews/pageviews referred by a page (estimate of how many links were opened from that page). Though it is literally just for 0-1 GMT, 6/29/15, now that we have the pageview hourly table, a more robust analysis can tell us how categories differ in this regard:
Category
links clicked
#pvs
clicks/pvs
Category:20th-centuryAmericanpoliticians
761
1243
61%
Category:Americandramatelevisionseries
5981
8844
68%
Category:Americanfemalepopsingers
2502
4280
58%
Category:LandmarksinSanFrancisco,
104
287
36%
Category:Modernpainters
136
369
37%
Category:NationalBasketballAssociationAll-Stars
1908
3341
57%
Category:Object-orientedprogramminglanguages
48
181
27%
Category:WesternEurope
657
1221
54%
Grand Total
12099
19766
50%
You can see here that for pages in the category ‘Landmarks in San Francisco’, if there are 10 pageviews, 5.4 clicks to other pages are generated on average.
I do not have the original queries for this handy, but can dig them up if you’re really interested.
Category tags
Full data and queries here: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1vD3DopxGyeh9FQsuTQDM...
The tags themselves generated an average click-through rate of .18%. Given the overall click thru rate on the pages estimated above ~50%, this single tag is not driving anything significant. Furthermore, given Leila and Bob’s paper suggest that this is performing no better than a mid-article click--given the mobile web sections are collapsed, I would need to understand more about their method to know just how to interpret their results against our mobile-web only implementation. Furthermore, our click through rate used the number of times the tag appeared on screen as the denominator, whereas their research looked at overall pageviews.
This being noted, the tag was implemented to be as obscure as possible to establish a baseline. Furthermore, any feature like this would probably be different in the following ways:
each page would be in 1-4 tag groups (as opposed to just 1)
each page would be tagged, creating the expectation on the part of the user that this was something to look for
presumably the categories could be implemented as a menu item as opposed to being buried at the bottom of the page (and competing with features like read more.
Using the learnings from ‘read more’ tags with images or buttons would likely fare much better.
The follow graph shows:
number of impressions on the right axis
click-thru-rate on the left-axis.
When you look at click through rates on the ‘category’ pages themselves, you see that they average at 41% (Chart below) Meaning that for every 10 times a user visited a category page, there were 4.1 clicks to one of those pages as a result.
Here is the same broken up by category:
Each ‘category’ page here had at least 400 visits, and you can see that the interest seems to vary dramatically across categories. It is worth noting that the top three categories here are the ones with the fewest entities. Each list, however, was capped at ~50 articles, so it is unclear what might be causing this effect, if it is real.
As mentioned above, the average article page has an overall click rate of 50%. So this page of categories did not have the click-through rate that a page has. However, this page had summaries of each of the pages, so it could be that users were generating value beyond what a blue link would provide. A live-user test of Gather collections, from whom this format was borrowed, suggested that the format used up too much vertical space on each article and was hard to flip through. Shortening the amount of text or image space might be something to try to make the page more useful
Conclusion and Next StepsProcess
This was the first time I am aware of that we ran a live prototype and learn something without building a scalable solution. Win
Developer time was estimated at 1 FTE for 2 weeks (by pheudx), but the chronological time for pushing to stable took a quarter. Room for improvement
The time to analysis was almost 2 quarters, due to a lack of data analysis support (I ran the initial analysis within 2 weeks of launch, during paternity leave, but was unable to go back and get it ready to distribute for 3 months). Room for improvement--possibly solved by additional Data Analyst.
This experiment was not designed to answer questions definitively in one round, but with the understanding that multiple iterations would allow us to fully answer our questions.
The long turn-around time, particularly around analysis and communication, meant that tweaking a variable to test the conclusions or the new questions that arosee below will involve a whole lot more work and effort than if we had been able to explore modifications within a few weeks of the initial launch.
Do people want to browse by categories?
Category tags at the bottom of the mobile web page in a dull gray background that lead to manually curated categories are not a killer feature :)
I would be reluctant to say that this means users are not interested in browsing by category, however. For instance, it is likely that
users did not notice the tag, even if it appeared on screen
users are accustomed to our current category tags on desktop and not interested in that experience
users who did like the tag were unlikely to find another page that had it--there was no feedback mechanism by which the improved category page would drive additional tag interactions
the browse experience created was not ideal
If we decide to pursue what is currently termed “cascade c: update ux”, I would like to proceed with more tests in this arena, by altering the appearance and position of the tags, and by improving the flow of the ‘category’ pages. If we choose a different strategy, hopefully other teams can build off of what was learned here.
reading-wmf mailing list reading-wmf@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/reading-wmf
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
On Sun, Oct 11, 2015 at 11:32 AM, Jon Katz jkatz@wikimedia.org wrote:
[Most-relevant category tag display experiment] was intended to lie in contrast with our current category pages, which are alphabetical and not really intended for human browsing: https://en.wikipedia.org/wiki/Category:American_male_film_actors
A category page is more an index to search or skim. But we could experiment with making them more engaging for human browsing. Just as the search results dropdown in mobile shows a page thumbnail and description along with each title, we could show the same thumbnail and description with every title on a category page. It could be a button [Enhance title display].
That would leave categories alphabetized instead of most relevant, but perhaps more engaging.
Cheers, -- =S Page WMF Tech writer