On Monday, December 15, 2014, Kevin Leduc kevin@wikimedia.org wrote:
I'd like to move this to a video conference call between analytics developers and analytics engineering to come to a mutual understanding of what the current pain points are and what's the biggest priority.
It probably makes sense to have someone from R&D with experience in QA in that meeting (Dario if you want a more experienced person, myself otherwise). Not sure if you meant the same when you said analytics engineering.
Leila
On Mon, Dec 15, 2014 at 4:37 PM, Nuria Ruiz <nuria@wikimedia.org javascript:_e(%7B%7D,'cvml','nuria@wikimedia.org');> wrote:
QA in beta labs is good but not enough. We still need to do QA when a
feature goes to production and currently This is true but at the same time, I do not see anything in the description of your FF events that could not be tested on beta-labs. If we are talking add-block that can be tested even earlier, vagrant will be a fine venue. All the issues related to the client (browser) not emitting events can be tested on the development environment with ease.
On Mon, Dec 15, 2014 at 4:18 PM, Leila Zia <leila@wikimedia.org javascript:_e(%7B%7D,'cvml','leila@wikimedia.org');> wrote:
On Mon, Dec 15, 2014 at 10:06 AM, Toby Negrin <tnegrin@wikimedia.org javascript:_e(%7B%7D,'cvml','tnegrin@wikimedia.org');> wrote:
I share Christian's concerns -
Dario/Leila - can you comment based on your recent experiences with WikiGrok?
I agree with Christian.
QA in beta labs is good but not enough. We still need to do QA when a feature goes to production and currently, it's very hard to figure out if there's a problem with logging. An example:
While testing WikiGrok in production, we learned that after some point tests from Firefox browser from my machine were not logged. We did not get any errors for this. I found out about this because I was trying to manually make a trace of activities and see if I can stitch them together and make sense of them. We eventually figured out what was going on in that case [1], but it concerns me that there may be other important events that we don't log in the DB and we never know that we're not logging.
Leila [1] https://lists.wikimedia.org/pipermail/analytics/2014-December/002864.html
Thanks
-Toby
On Dec 15, 2014, at 9:42 AM, Christian Aistleitner <
christian@quelltextlich.at javascript:_e(%7B%7D,'cvml','christian@quelltextlich.at');> wrote:
Hi,
On Mon, Dec 15, 2014 at 08:34:39AM -0800, Kevin Leduc wrote: I closed the Phabricator task with a links to this thread and the
wikitech
doc for testing on beta cluster.
I am fine with keeping the task closed.
But I am somewhat surprised to see beta mentioned in the resolution. Note that Dario's request set scope as [1]
However, there are types of data quality issues that we only discover when collecting data at scale and in the wild (on browsers/platforms that we don’t necessarily test for internally).
. That's a valid scope, but from my point of view, beta does not match that scope.
Neither is beta large scale, nor is it hammered on with crazy devices.
Beta is just a halfing the distance between EventLogging's devserver (Vagrant!) and production.
Have fun, Christian
[1]
https://lists.wikimedia.org/pipermail/analytics/2014-December/002884.html
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian@quelltextlich.at
javascript:_e(%7B%7D,'cvml','christian@quelltextlich.at');
4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org
javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org');
Analytics mailing list Analytics@lists.wikimedia.org javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org'); https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org'); https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org'); https://lists.wikimedia.org/mailman/listinfo/analytics