It is worth pointing that our base database infrastracture is 35 times larger. Our (non duplicate) base data set is 10-15 times larger, compressed. And that we serve 30 times the number of database queries than they do; with peaks at 10-20x the number of queries per second, per server, despite his hardware being twice as powerful than our newest hardware.
All that with around 6-7 people working in infrastructure (vs 11 of us).
This doesn't have anything to do with the original post. I just wanted to a) agree with Dan that we need better analytics infrastructure (Re: Something to aspire to, perhaps collaborate with them on.) and b) explain why this hasn't been done already and why it is complex. But it is a known request both from analytics, research and other labs users.
Sources: http://stackexchange.com/performance https://wikimediafoundation.org/wiki/Staff_and_contractors https://www.mediawiki.org/wiki/File:MySQL_at_Wikipedia.pdf http://stackexchange.com/about/team#Engineering
On Sat, Nov 14, 2015 at 2:18 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
For anyone else interested: Nemo was able to answer this question because
StackExchange has a Quarry http://quarry.wmflabs.org/-like public query interface of their own. You should go play with it right now: http://data.stackexchange.com/
It's worth pointing out one major difference between their Quarry-like thing and our Quarry. I love both, btw. But our Quarry suffers because the only public database we have is raw and has a schema meant for OLTP. StackExchange's is clearly hitting a well organized OLAP style schema. Something to aspire to, perhaps collaborate with them on.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics