(cc-ing analytics list)
Jon:
To get a prompt response please cc analytics or analytics-internal on e-mails that way -given that we operate in three timezones- someone is bound to see your e-mail and respond in a timely fashion.
Right now the Gather feature is only available/discoverable on mobile web
beta for english wikipedia. I assume "mobile web beta" is deployed to enwiki in production just like an special "skin" or 'extension.
Mobile web beta does not have any special url. It is triggered by a
cookie. If the that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about) then there is no way for you to see that traffic. (this is the most-likely case I was looking around and couldn't find any code that would be persisting this cookies further than varnish). However, if the cookie value is persisted to x-analytics field then records are "findable".
Now, if there are any javascript/css files ONLY used by the "mobile web beta feature" you could do a (very, very) rough estimation of requests by looking at those. Probably you want to team up with a developer to find out whether this is possible.
On Thu, Apr 2, 2015 at 4:33 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Nuria, Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia. As such, it is really important that I know what the overall traffic is on mobile web beta. How can I discover this using web request logs? Mobile web beta does not have any special url.
It is triggered by a cookie. Here is how a user opts in. Any help would be greatly appreciated!!!
[image: Inline image 1]
Best,
Jon
Sorry, this should be:
Mobile web beta does not have any special url. It is triggered by a
cookie. If the COOKIE that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about)...
On Thu, Apr 2, 2015 at 5:05 PM, Nuria Ruiz nuria@wikimedia.org wrote:
(cc-ing analytics list)
Jon:
To get a prompt response please cc analytics or analytics-internal on e-mails that way -given that we operate in three timezones- someone is bound to see your e-mail and respond in a timely fashion.
Right now the Gather feature is only available/discoverable on mobile
web beta for english wikipedia. I assume "mobile web beta" is deployed to enwiki in production just like an special "skin" or 'extension.
Mobile web beta does not have any special url. It is triggered by a
cookie. If the that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about) then there is no way for you to see that traffic. (this is the most-likely case I was looking around and couldn't find any code that would be persisting this cookies further than varnish). However, if the cookie value is persisted to x-analytics field then records are "findable".
Now, if there are any javascript/css files ONLY used by the "mobile web beta feature" you could do a (very, very) rough estimation of requests by looking at those. Probably you want to team up with a developer to find out whether this is possible.
On Thu, Apr 2, 2015 at 4:33 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Nuria, Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia. As such, it is really important that I know what the overall traffic is on mobile web beta. How can I discover this using web request logs? Mobile web beta does not have any special url.
It is triggered by a cookie. Here is how a user opts in. Any help would be greatly appreciated!!!
[image: Inline image 1]
Best,
Jon
I recall at some point the mobile site was setting the site type (Beta vs. Normal) in one of the headers, either X-Analytics or X-CS. But I don't see any of that in the most recent varnish code [1]. I might not be looking in the right place, or it may not be happening any more. CC-ing mobile list as someone there likely knows.
[1] https://github.com/wikimedia/operations-puppet/blob/fb4205ccc384612e267d59ba...
On Thu, Apr 2, 2015 at 9:57 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Sorry, this should be:
Mobile web beta does not have any special url. It is triggered by a
cookie. If the COOKIE that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about)...
On Thu, Apr 2, 2015 at 5:05 PM, Nuria Ruiz nuria@wikimedia.org wrote:
(cc-ing analytics list)
Jon:
To get a prompt response please cc analytics or analytics-internal on e-mails that way -given that we operate in three timezones- someone is bound to see your e-mail and respond in a timely fashion.
Right now the Gather feature is only available/discoverable on mobile
web beta for english wikipedia. I assume "mobile web beta" is deployed to enwiki in production just like an special "skin" or 'extension.
Mobile web beta does not have any special url. It is triggered by a
cookie. If the that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about) then there is no way for you to see that traffic. (this is the most-likely case I was looking around and couldn't find any code that would be persisting this cookies further than varnish). However, if the cookie value is persisted to x-analytics field then records are "findable".
Now, if there are any javascript/css files ONLY used by the "mobile web beta feature" you could do a (very, very) rough estimation of requests by looking at those. Probably you want to team up with a developer to find out whether this is possible.
On Thu, Apr 2, 2015 at 4:33 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Nuria, Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia. As such, it is really important that I know what the overall traffic is on mobile web beta. How can I discover this using web request logs? Mobile web beta does not have any special url.
It is triggered by a cookie. Here is how a user opts in. Any help would be greatly appreciated!!!
[image: Inline image 1]
Best,
Jon
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
X-CS historically has been for Wikipedia Zero operator tagging. Not sure about use of X-Analytics or other headers for alpha/beta/stable channel stuff.
On Thu, Apr 2, 2015 at 8:53 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
I recall at some point the mobile site was setting the site type (Beta vs. Normal) in one of the headers, either X-Analytics or X-CS. But I don't see any of that in the most recent varnish code [1]. I might not be looking in the right place, or it may not be happening any more. CC-ing mobile list as someone there likely knows.
[1] https://github.com/wikimedia/operations-puppet/blob/fb4205ccc384612e267d59ba...
On Thu, Apr 2, 2015 at 9:57 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Sorry, this should be:
Mobile web beta does not have any special url. It is triggered by a
cookie. If the COOKIE that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about)...
On Thu, Apr 2, 2015 at 5:05 PM, Nuria Ruiz nuria@wikimedia.org wrote:
(cc-ing analytics list)
Jon:
To get a prompt response please cc analytics or analytics-internal on e-mails that way -given that we operate in three timezones- someone is bound to see your e-mail and respond in a timely fashion.
Right now the Gather feature is only available/discoverable on mobile
web beta for english wikipedia. I assume "mobile web beta" is deployed to enwiki in production just like an special "skin" or 'extension.
Mobile web beta does not have any special url. It is triggered by a
cookie. If the that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about) then there is no way for you to see that traffic. (this is the most-likely case I was looking around and couldn't find any code that would be persisting this cookies further than varnish). However, if the cookie value is persisted to x-analytics field then records are "findable".
Now, if there are any javascript/css files ONLY used by the "mobile web beta feature" you could do a (very, very) rough estimation of requests by looking at those. Probably you want to team up with a developer to find out whether this is possible.
On Thu, Apr 2, 2015 at 4:33 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Nuria, Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia. As such, it is really important that I know what the overall traffic is on mobile web beta. How can I discover this using web request logs? Mobile web beta does not have any special url.
It is triggered by a cookie. Here is how a user opts in. Any help would be greatly appreciated!!!
[image: Inline image 1]
Best,
Jon
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I'm not really sure - maybe Max knows? I know we vary on an `optin` cookie - maybe that will provide some clues?
On Fri, Apr 3, 2015 at 1:27 PM, Adam Baso abaso@wikimedia.org wrote:
X-CS historically has been for Wikipedia Zero operator tagging. Not sure about use of X-Analytics or other headers for alpha/beta/stable channel stuff.
On Thu, Apr 2, 2015 at 8:53 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
I recall at some point the mobile site was setting the site type (Beta vs. Normal) in one of the headers, either X-Analytics or X-CS. But I don't see any of that in the most recent varnish code [1]. I might not be looking in the right place, or it may not be happening any more. CC-ing mobile list as someone there likely knows.
[1] https://github.com/wikimedia/operations-puppet/blob/fb4205ccc384612e267d59ba...
On Thu, Apr 2, 2015 at 9:57 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Sorry, this should be:
Mobile web beta does not have any special url. It is triggered by a
cookie. If the COOKIE that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about)...
On Thu, Apr 2, 2015 at 5:05 PM, Nuria Ruiz nuria@wikimedia.org wrote:
(cc-ing analytics list)
Jon:
To get a prompt response please cc analytics or analytics-internal on e-mails that way -given that we operate in three timezones- someone is bound to see your e-mail and respond in a timely fashion.
Right now the Gather feature is only available/discoverable on mobile
web beta for english wikipedia. I assume "mobile web beta" is deployed to enwiki in production just like an special "skin" or 'extension.
Mobile web beta does not have any special url. It is triggered by a
cookie. If the that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about) then there is no way for you to see that traffic. (this is the most-likely case I was looking around and couldn't find any code that would be persisting this cookies further than varnish). However, if the cookie value is persisted to x-analytics field then records are "findable".
Now, if there are any javascript/css files ONLY used by the "mobile web beta feature" you could do a (very, very) rough estimation of requests by looking at those. Probably you want to team up with a developer to find out whether this is possible.
On Thu, Apr 2, 2015 at 4:33 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Nuria, Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia. As such, it is really important that I know what the overall traffic is on mobile web beta. How can I discover this using web request logs? Mobile web beta does not have any special url.
It is triggered by a cookie. Here is how a user opts in. Any help would be greatly appreciated!!!
[image: Inline image 1]
Best,
Jon
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I'm not really sure - maybe Max knows? I know we vary on an `optin` cookie - maybe that will provide some clues?
Christian's response (below) tells us the "mf-m" key in the X-Analytics header has the information we need here. So we're all set. "a" means alpha site and "b" means beta.
[From Christian]:
For X-Analytics stuff,
would be the go-to place. I guess you're after the “mf-m” key there. That key is not set in varnish, but is set in the backend (MobileFrontendExtension), and it is still getting logged. But I am not aware of any reports still using that key.
Thanks Christian, sorry I didn't read the obvious documentation before dumping vague memories for awesome volunteers to help with.
Ah sorry! (Ignore my prior reply)
On Apr 6, 2015, at 1:41 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
I'm not really sure - maybe Max knows? I know we vary on an `optin` cookie - maybe that will provide some clues?
Christian's response (below) tells us the "mf-m" key in the X-Analytics header has the information we need here. So we're all set. "a" means alpha site and "b" means beta.
[From Christian]:
For X-Analytics stuff,
would be the go-to place. I guess you're after the “mf-m” key there. That key is not set in varnish, but is set in the backend (MobileFrontendExtension), and it is still getting logged. But I am not aware of any reports still using that key.
Thanks Christian, sorry I didn't read the obvious documentation before dumping vague memories for awesome volunteers to help with. _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Thanks all--you're awesome! I will look at the X-analytics header(?) once I get permissions and hit you back if I have any trouble. Best, J
On Mon, Apr 6, 2015 at 2:05 PM, Nuria nuria@wikimedia.org wrote:
Ah sorry! (Ignore my prior reply)
On Apr 6, 2015, at 1:41 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
I'm not really sure - maybe Max knows?
I know we vary on an `optin` cookie - maybe that will provide some clues?
Christian's response (below) tells us the "mf-m" key in the X-Analytics header has the information we need here. So we're all set. "a" means alpha site and "b" means beta.
[From Christian]:
For X-Analytics stuff,
would be the go-to place. I guess you're after the “mf-m” key there. That key is not set in varnish, but is set in the backend (MobileFrontendExtension), and it is still getting logged. But I am not aware of any reports still using that key.
Thanks Christian, sorry I didn't read the obvious documentation before dumping vague memories for awesome volunteers to help with.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi, Just following up to thank you guys.
Kevin and Ottomoata guided me through getting set up on hive and I made the first queries I needed to! Here were the results (if you're interested) Best,
J
4/12 full day
*mobile_site* *Pageviews* *% of stable* a 8,578 0.004% b 46,705 0.019% stable 244,816,000 Strangely, the event-logs for specific actions showed much higher traffic for beta but this is likely due to different user groups? (multiple days) *mobile_site* * search events* *% of stable* alpha 18,768 0.01% beta 153,617 0.09% stable 178,118,332
queries:
for alpha beta SELECT x_analytics_map['mf-m'] AS mobile_site, count(*) FROM wmf.webrequest WHERE access_method = "mobile web" AND YEAR = 2015 AND MONTH = 4 AND DAY = 12 AND user_agent_map['device_family'] <> "Spider" AND is_pageview = TRUE AND x_analytics_map['mf-m'] IS NOT NULL GROUP BY x_analytics_map['mf-m'] LIMIT 50;
For stable: SELECT count(*) FROM wmf.webrequest tablesample(BUCKET 1 OUT OF 1000 ON rand()) WHERE access_method = "mobile web" AND YEAR = 2015 AND MONTH = 4 AND DAY = 12 AND is_pageview = TRUE AND user_agent_map['device_family'] <> "Spider" AND x_analytics_map['mf-m'] IS NULL LIMIT 50;
For events: SELECT count(uuid), event_name, event_mobileMode FROM MobileWebUIClickTracking_10742159 GROUP BY event_name, event_mobileMode;
Some things to have in mind:
1) Bots AND user_agent_map['device_family'] <> "Spider" Doesn't remove all bots, only very prominent ones, so stats still include traffic from say, wmf robots, for example.
2) Sampling:
Strangely, the event-logs for specific actions showed much higher traffic
for
beta but this is likely due to different user groups?
It is worth checking whether you are gathering data in equal sample rates for all 3 sites. For tyraffic you are comparing days for eventlogging data you are comparing an unknown interval of time.
3) Daily versus weekly It is also worth checking weekly figures as they might differ.
4) MobileWebUIClickTracking I thought this schema had been split in several and was no longer in use, again, there is no time interval in your query.
Thanks,
Nuria
On Wed, Apr 15, 2015 at 12:14 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi, Just following up to thank you guys.
Kevin and Ottomoata guided me through getting set up on hive and I made the first queries I needed to! Here were the results (if you're interested) Best,
J
4/12 full day
*mobile_site* *Pageviews* *% of stable* a 8,578 0.004% b 46,705 0.019% stable 244,816,000 Strangely, the event-logs for specific actions showed much higher traffic for beta but this is likely due to different user groups? (multiple days) *mobile_site* * search events* *% of stable* alpha 18,768 0.01% beta 153,617 0.09% stable 178,118,332
queries:
for alpha beta SELECT x_analytics_map['mf-m'] AS mobile_site, count(*) FROM wmf.webrequest WHERE access_method = "mobile web" AND YEAR = 2015 AND MONTH = 4 AND DAY = 12 AND user_agent_map['device_family'] <> "Spider" AND is_pageview = TRUE AND x_analytics_map['mf-m'] IS NOT NULL GROUP BY x_analytics_map['mf-m'] LIMIT 50;
For stable: SELECT count(*) FROM wmf.webrequest tablesample(BUCKET 1 OUT OF 1000 ON rand()) WHERE access_method = "mobile web" AND YEAR = 2015 AND MONTH = 4 AND DAY = 12 AND is_pageview = TRUE AND user_agent_map['device_family'] <> "Spider" AND x_analytics_map['mf-m'] IS NULL LIMIT 50;
For events: SELECT count(uuid), event_name, event_mobileMode FROM MobileWebUIClickTracking_10742159 GROUP BY event_name, event_mobileMode;
Thanks, Nuria. I really appreciate the guidance. -J
On Wed, Apr 15, 2015 at 4:17 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Some things to have in mind:
- Bots AND user_agent_map['device_family'] <> "Spider"
Doesn't remove all bots, only very prominent ones, so stats still include traffic from say, wmf robots, for example.
- Sampling:
Strangely, the event-logs for specific actions showed much higher
traffic for
beta but this is likely due to different user groups?
It is worth checking whether you are gathering data in equal sample rates for all 3 sites. For tyraffic you are comparing days for eventlogging data you are comparing an unknown interval of time.
- Daily versus weekly
It is also worth checking weekly figures as they might differ.
- MobileWebUIClickTracking
I thought this schema had been split in several and was no longer in use, again, there is no time interval in your query.
Thanks,
Nuria
On Wed, Apr 15, 2015 at 12:14 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi, Just following up to thank you guys.
Kevin and Ottomoata guided me through getting set up on hive and I made the first queries I needed to! Here were the results (if you're interested) Best,
J
4/12 full day
*mobile_site* *Pageviews* *% of stable* a 8,578 0.004% b 46,705 0.019% stable 244,816,000 Strangely, the event-logs for specific actions showed much higher traffic for beta but this is likely due to different user groups? (multiple days) *mobile_site* * search events* *% of stable* alpha 18,768 0.01% beta 153,617 0.09% stable 178,118,332
queries:
for alpha beta SELECT x_analytics_map['mf-m'] AS mobile_site, count(*) FROM wmf.webrequest WHERE access_method = "mobile web" AND YEAR = 2015 AND MONTH = 4 AND DAY = 12 AND user_agent_map['device_family'] <> "Spider" AND is_pageview = TRUE AND x_analytics_map['mf-m'] IS NOT NULL GROUP BY x_analytics_map['mf-m'] LIMIT 50;
For stable: SELECT count(*) FROM wmf.webrequest tablesample(BUCKET 1 OUT OF 1000 ON rand()) WHERE access_method = "mobile web" AND YEAR = 2015 AND MONTH = 4 AND DAY = 12 AND is_pageview = TRUE AND user_agent_map['device_family'] <> "Spider" AND x_analytics_map['mf-m'] IS NULL LIMIT 50;
For events: SELECT count(uuid), event_name, event_mobileMode FROM MobileWebUIClickTracking_10742159 GROUP BY event_name, event_mobileMode;
I think - from looking at code - we can conclude that youcannot identify mobile-webbeta traffic.
On Apr 6, 2015, at 1:19 PM, Jon Robson jrobson@wikimedia.org wrote:
I'm not really sure - maybe Max knows? I know we vary on an `optin` cookie - maybe that will provide some clues?
On Fri, Apr 3, 2015 at 1:27 PM, Adam Baso abaso@wikimedia.org wrote: X-CS historically has been for Wikipedia Zero operator tagging. Not sure about use of X-Analytics or other headers for alpha/beta/stable channel stuff.
On Thu, Apr 2, 2015 at 8:53 PM, Dan Andreescu dandreescu@wikimedia.org wrote: I recall at some point the mobile site was setting the site type (Beta vs. Normal) in one of the headers, either X-Analytics or X-CS. But I don't see any of that in the most recent varnish code [1]. I might not be looking in the right place, or it may not be happening any more. CC-ing mobile list as someone there likely knows.
[1] https://github.com/wikimedia/operations-puppet/blob/fb4205ccc384612e267d59ba...
On Thu, Apr 2, 2015 at 9:57 PM, Nuria Ruiz nuria@wikimedia.org wrote: Sorry, this should be:
Mobile web beta does not have any special url. It is triggered by a cookie.
If the COOKIE that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about)...
On Thu, Apr 2, 2015 at 5:05 PM, Nuria Ruiz nuria@wikimedia.org wrote: (cc-ing analytics list)
Jon:
To get a prompt response please cc analytics or analytics-internal on e-mails that way -given that we operate in three timezones- someone is bound to see your e-mail and respond in a timely fashion.
Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia.
I assume "mobile web beta" is deployed to enwiki in production just like an special "skin" or 'extension.
Mobile web beta does not have any special url. It is triggered by a cookie.
If the that identifies 'mobile-web-beta' is stripped off in varnish (something you can ask your devs about) then there is no way for you to see that traffic. (this is the most-likely case I was looking around and couldn't find any code that would be persisting this cookies further than varnish). However, if the cookie value is persisted to x-analytics field then records are "findable".
Now, if there are any javascript/css files ONLY used by the "mobile web beta feature" you could do a (very, very) rough estimation of requests by looking at those. Probably you want to team up with a developer to find out whether this is possible.
On Thu, Apr 2, 2015 at 4:33 PM, Jon Katz jkatz@wikimedia.org wrote: Hi Nuria, Right now the Gather feature is only available/discoverable on mobile web beta for english wikipedia. As such, it is really important that I know what the overall traffic is on mobile web beta. How can I discover this using web request logs? Mobile web beta does not have any special url.
It is triggered by a cookie. Here is how a user opts in. Any help would be greatly appreciated!!!
<image.png>
Best,
Jon
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi,
On Thu, Apr 02, 2015 at 11:53:24PM -0400, Dan Andreescu wrote:
I recall at some point the mobile site was setting the site type (Beta vs. Normal) in one of the headers, either X-Analytics or X-CS. But I don't see any of that in the most recent varnish code [1].
For X-Analytics stuff,
https://wikitech.wikimedia.org/wiki/X-Analytics
would be the go-to place. I guess you're after the “mf-m” key there. That key is not set in varnish, but is set in the backend (MobileFrontendExtension), and it is still getting logged. But I am not aware of any reports still using that key.
Have fun, Christian
