Dan Andreescu <dandreescu(a)wikimedia.org> wrote:
Maybe something exists already in Hadoop
The page properties table is already loaded into Hadoop on a monthly basis
(wmf_raw.mediawiki_page_props). I haven't played with it much, but Hive
also has JSON-parsing goodies, so give it a shot and let me know if you get
stuck. In general, data from the databases can be sqooped into Hadoop. We
do this for large pipelines like edit history
<https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Edit_data_loading>
and
it's very easy
<https://github.com/wikimedia/analytics-refinery/blob/master/python/refinery/sqoop.py#L505
to add a table. We're looking at just
replicating the whole db on a more
frequent basis, but we have to do some groundwork first to allow
incremental updates (see Apache Iceberg if you're interested).
Yes, I like that and all of the other wmf_raw goodies! I'll follow up off
thread on accessing the parser cache DBs (they're in site.pp and
db-eqiad.php, but I don't think those are presently represented by
refinery.util as they're not in .dblist files).