Hey, I was wondering if it is possible to export the results of Hive queries to some world-readable place?
What I'm trying to achieve: for my www portals work, I want the results of aggregation (SELECT project, sum(view_count) AS num FROM projectview_hourly WHERE year=2015 AND month=11 AND day=3 GROUP BY project) published somewhere in a machine-readable format. Ideally, this could be published externally (for example, https://stats.wikimedia.org/daily_pageviews.csv or whatever). If that is hard, making it somehow available on the cluster would suffice. What are the options for doing that?
Max, there's a pageview API that we're not fully ready to announce because we haven't finished the documentation but it works so I'll tell you offline about it. It has the data you're looking for in that query.
Anyone else who is interested in the API - we're just finishing up docs and synchronizing with a blog post, it won't be long now, the actual code and infrastructure is stable.
On Wed, Nov 4, 2015 at 7:50 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Hey, I was wondering if it is possible to export the results of Hive queries to some world-readable place?
What I'm trying to achieve: for my www portals work, I want the results of aggregation (SELECT project, sum(view_count) AS num FROM projectview_hourly WHERE year=2015 AND month=11 AND day=3 GROUP BY project) published somewhere in a machine-readable format. Ideally, this could be published externally (for example, https://stats.wikimedia.org/daily_pageviews.csv or whatever). If that is hard, making it somehow available on the cluster would suffice. What are the options for doing that?
-- Best regards, Max Semenik ([[User:MaxSem]])
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Oh, and in general, you can dump the results of queries to a location on stat1002 that rsyncs to a public place. But we need people to be very careful with that so we usually want to go through code review for any code that does it. Reportupdater is a tool that you can use to write SQL scripts or bash scripts which make the process of "publishing" data a little better.
On Thu, Nov 5, 2015 at 9:45 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
Max, there's a pageview API that we're not fully ready to announce because we haven't finished the documentation but it works so I'll tell you offline about it. It has the data you're looking for in that query.
Anyone else who is interested in the API - we're just finishing up docs and synchronizing with a blog post, it won't be long now, the actual code and infrastructure is stable.
On Wed, Nov 4, 2015 at 7:50 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Hey, I was wondering if it is possible to export the results of Hive queries to some world-readable place?
What I'm trying to achieve: for my www portals work, I want the results of aggregation (SELECT project, sum(view_count) AS num FROM projectview_hourly WHERE year=2015 AND month=11 AND day=3 GROUP BY project) published somewhere in a machine-readable format. Ideally, this could be published externally (for example, https://stats.wikimedia.org/daily_pageviews.csv or whatever). If that is hard, making it somehow available on the cluster would suffice. What are the options for doing that?
-- Best regards, Max Semenik ([[User:MaxSem]])
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Oh, I also realized these files already has the data you need:
In http://dumps.wikimedia.org/other/pageviews/2015/2015-11/, files projectviews-20151103-000000 through projectviews-20151103-230000
On Thu, Nov 5, 2015 at 9:47 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
Oh, and in general, you can dump the results of queries to a location on stat1002 that rsyncs to a public place. But we need people to be very careful with that so we usually want to go through code review for any code that does it. Reportupdater is a tool that you can use to write SQL scripts or bash scripts which make the process of "publishing" data a little better.
On Thu, Nov 5, 2015 at 9:45 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
Max, there's a pageview API that we're not fully ready to announce because we haven't finished the documentation but it works so I'll tell you offline about it. It has the data you're looking for in that query.
Anyone else who is interested in the API - we're just finishing up docs and synchronizing with a blog post, it won't be long now, the actual code and infrastructure is stable.
On Wed, Nov 4, 2015 at 7:50 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Hey, I was wondering if it is possible to export the results of Hive queries to some world-readable place?
What I'm trying to achieve: for my www portals work, I want the results of aggregation (SELECT project, sum(view_count) AS num FROM projectview_hourly WHERE year=2015 AND month=11 AND day=3 GROUP BY project) published somewhere in a machine-readable format. Ideally, this could be published externally (for example, https://stats.wikimedia.org/daily_pageviews.csv or whatever). If that is hard, making it somehow available on the cluster would suffice. What are the options for doing that?
-- Best regards, Max Semenik ([[User:MaxSem]])
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Yeah, they should have it.
Max, what's the underlying goal here? The Portal work has direct analysis support. If you need queries or data you can always file a phab ticket.
On 5 November 2015 at 09:50, Dan Andreescu dandreescu@wikimedia.org wrote:
Oh, I also realized these files already has the data you need:
In http://dumps.wikimedia.org/other/pageviews/2015/2015-11/, files projectviews-20151103-000000 through projectviews-20151103-230000
On Thu, Nov 5, 2015 at 9:47 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
Oh, and in general, you can dump the results of queries to a location on stat1002 that rsyncs to a public place. But we need people to be very careful with that so we usually want to go through code review for any code that does it. Reportupdater is a tool that you can use to write SQL scripts or bash scripts which make the process of "publishing" data a little better.
On Thu, Nov 5, 2015 at 9:45 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
Max, there's a pageview API that we're not fully ready to announce because we haven't finished the documentation but it works so I'll tell you offline about it. It has the data you're looking for in that query.
Anyone else who is interested in the API - we're just finishing up docs and synchronizing with a blog post, it won't be long now, the actual code and infrastructure is stable.
On Wed, Nov 4, 2015 at 7:50 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Hey, I was wondering if it is possible to export the results of Hive queries to some world-readable place?
What I'm trying to achieve: for my www portals work, I want the results of aggregation (SELECT project, sum(view_count) AS num FROM projectview_hourly WHERE year=2015 AND month=11 AND day=3 GROUP BY project) published somewhere in a machine-readable format. Ideally, this could be published externally (for example, https://stats.wikimedia.org/daily_pageviews.csv or whatever). If that is hard, making it somehow available on the cluster would suffice. What are the options for doing that?
-- Best regards, Max Semenik ([[User:MaxSem]])
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics