Explain what? Everything we have in labs is for experimentation and prototyping. It should not be used for prod. 16 июня 2015 г. 18:16 пользователь "Oliver Keyes" <okeyes(a)wikimedia.org> написал:

Labs hosts a special project called Tool Labs which https://wikitech.wikimedia.org/wiki/Help:FAQ describes as "designed to host tools that are not so complicated that they would require a dedicated instance". One notable tool is Geohack (https://tools.wmflabs.org/geohack/): every article about a location links to it. Example from https://en.wikipedia.org/wiki/Aberdeen: Coordinates: 57.1526°N 2.1100°W <https://tools.wmflabs.org/geohack/geohack.php?pagename=Aberdeen&params=57.1526_N_-2.1100_E_type:city_region:GB> There are also various analytics dashboards in labs which might be considered as "production" https://glam-metrics.wmflabs.org/ https://metrics.wmflabs.org/ http://reportcard.wmflabs.org/ http://searchdata.wmflabs.org/ As others have mentioned, part of the problem is the ease in spinning up a labs instance vs hardware procurement for a deployment into production. Ops (Alex) is working on virtualization infrastructure for use in prod called Ganeti (an alternative to the OpenStack virtualization software which powers labs); the first service was recently migrated from hardware to a Ganeti instance: etherpad. Ganeti could be the way forward for promoting services from labs to prod without requiring dedicated hardware. https://wikitech.wikimedia.org/wiki/Ganeti On Wed, Jun 17, 2015 at 8:01 AM, Kevin Smith &lt;ksmith(a)wikimedia.org&gt; wrote:

Yes, but it seems possible to me that the firewalls could be modified to support specific data flows.

Is it running for a really long time? Is it designed to not fall over? Is it no longer being actively looked at with the look you give a rickety Jenga tower because you're confident it's bug-free? Are users being pointed at it without caveats? Then it's production. On 16 June 2015 at 21:52, Max Semenik &lt;maxsem.wiki(a)gmail.com&gt; wrote: -- Oliver Keyes Research Analyst Wikimedia Foundation

Strictly-speaking "production" is the "primary" cluster but the point here is about the reliability users assume around a service rather than the reliability it has. On 17 June 2015 at 12:09, James Douglas &lt;jdouglas(a)wikimedia.org&gt; wrote: -- Oliver Keyes Research Analyst Wikimedia Foundation

James Douglas, 17/06/2015 18:12: Huge +1. Nemo

On 16 June 2015 at 20:50, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; wrote: See below. The problem, as we've gone back and forth about for a while on phabricator, is that labs has absolutely zero inbuilt infrastructure for analytics. If things are in production they go through the frontend varnishes, which are hooked up to HDFS, and all is fine. We have the request logs. If things are in labs...nothing. There is no access to HDFS, there is no consistent varnish setup that pipes things there, and analytics engineering has pretty much no plans to set up that sort of infrastructure. -- Oliver Keyes Research Analyst Wikimedia Foundation

Well, no, HDFS is a means to and end of storing data in a form that can be cleaned with ETL processes so that /then/ they can go to the somewhere/something - which is a lot of use cases but most prominently our dashboards and ad-hoc research tasks. Let me be clear here that this isn't a theoretical exercise existing in a vacuum; we do not want /a/ answer that can be hooked up to the dashboards. That's easy. That's a hideous shell script that scaps nginx files over. We want a answer that can be hooked up to the dashboards for many, many, many things, because we're not just wanting metrics and analytics for WDQS, we're also wanting them for the production API and for user events and for the cirrus logs and for high-level KPIs and that's just the things we've wanted this month. I can't be building out an entirely new pipeline every single time someone builds a thing. That's not an efficient use of our analysts time and it massively increases the chance that something will go wrong. I'm not asking for an alternative to HDFS, because I don't want to be doing that. I'm asking for HDFS because then we don't need to reinvent the wheel every time we build a thing. If we can't do HDFS and going to production isn't going to work, then let's talk about what the alternatives are. Until then the use case is "the data being in HDFS so that analysts can consume it" and higher-level use cases are overthinking. On 17 June 2015 at 03:22, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; wrote: -- Oliver Keyes Research Analyst Wikimedia Foundation

On 17 June 2015 at 16:33, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; wrote: Thanks! It does make sense; at the moment the blocker is somewhere wooly around analytics and ops. So, on the same infrastructure, Analytics Engineering have indicated (iirc) they're comfortable standing up a HDFS instance but not so keen on maintaining it indefinitely. This makes total sense with their priorities. On building the bridge; at the moment the HDFS cluster is very deliberately firewalled. We'd need to deal with that (perhaps, as suggested, making highly specific and authenticated holes in the firewall?) before it was possible, and that seems to be an Analytics/Opsen thing. We do! if the WDQS queries are going through Production's varnish caches (an existing cluster) they go in automatically. If they go through a new frontend cluster, the cost of switching them in is fairly small. -- Oliver Keyes Research Analyst Wikimedia Foundation