really good summary of the situation, Neil, I'm bookmarking this and will
re-use it when people ask :)
On Thu, Mar 22, 2018 at 7:07 AM, Neil Patel Quinn <nquinn(a)wikimedia.org>
wrote:
On 22 March 2018 at 13:41, Neil Patel Quinn
<nquinn(a)wikimedia.org> wrote:
Both the edit data and pageview data that you're talking about come from
the Hadoop-based Analytics Data Lake
<https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake>. However,
because of limitations in the underlying MediaWiki application databases
<https://www.mediawiki.org/wiki/Manual:Database_layout> *that Hive pulls
edit data from*, the data requires some complex reconstruction and
denormalization
<https://wikitech.wikimedia.org/wiki/Analytics/Systems/Data_Lake/Edits/Pipeline>
that takes several days to a week.
Sorry, I garbled that a little. It's more correct to say: "because of
limitations in the underlying MediaWiki application databases *that are
the source of the edit data*, the data requires..."
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics