Notes from Hadoop and Beyond

• Video
• Slides

webrequest logs

This is a log for every WMF HTTP request. It can max out beyond 200k requests per second, which is a lot.

udp2log

Doesn't scale, because every instance must process every message, every packet.

Because it uses UDP, it's not guaranteed not to drop data.

Wikimedia Statistics

http://stats.wikimedia.org

Most data here is generated by udp2log-collected data.

It's sampled because there's too much traffic for our storage/processing capacity.

Analytics cluster

Uses Hadoop for batch processing of logs, and (mostly) uses Hive to expose the data to analysts.

This diagram is a useful (and frequent) reference:

Analytics cluster diagram

Note the loopy-back lines in the diagram -- these are batch jobs to do various things (geocoding, anonymizing, etc.).

Hadoop

Hadoop = a distributed file system + a framework for distributed computation

Hive

Hypothetical analyst question: How to get the top referrals for an article?

Hive maps a SQL-like language onto Hadoop MapReduce jobs.

Example Hive query to answer the above question:

SELECT
  SUBSTR(referer,30) AS SOURCE, COUNT(DISTINCT ip) AS hits
FROM
  webrequest
WHERE
  uri_path = "/wiki/London" AND uri_host = "en.wikipedia.org" AND
             referer LIKE "http://en.wikipedia.org/wiki/%" AND
             http_status = 200 AND webrequest_source = ‘text’ AND
             year = 2014 AND month= 07 AND day = 14 AND hour = 14
GROUP BY
  SUBSTR(referer,30)
ORDER BY hits DESC LIMIT 50;

This is nice because it lets you run SQL on top of text data.

Kafka cluster

This serves as a replacement for udp2log.

Kafka is a reliable, horizontally scalable, distributed pub/sub buffer.

Processes up to 200k messages per second at 30 MB per second.

Data is consumed every ten minutes into Hadoop

Camus

A job that runs on Hadoop to consume from Kafka, and write the data into HDFS.

Launches a MapReduce job every hour(?).

Can inspect the data as it's coming in -- lets it handle data based on time/content.

Oozie

Oozie is a Hadoop job scheduler that allows the composition of complex workflows.

Jobs are launched based on the existence of new data sets, rather than simply based on time or periodic intervals. This lets us can trigger Oozie whenever a Camus job completes.

Hue

Web GUI for interacting with Hadoop, Hive, Oozie, etc.

Provides a Hive query interface, a Pig script interface, a way to launch jobs, browse the file system, and install add-ons.

A command-line interface is also available for all of the above.

MediaWiki Vagrant

To play with this in Vagrant:

• Edit your .settings.yaml file and add:
• vagrant_ram: 2048
• Comment out include role::mediawiki in puppet/manifests/site.pp (unless you really need this on your VM)
• Run: vagrant enable-role analytics
• Run: vagrant up

Hive and Hadoop should now be available in Vagrant!