Analytics November 2018

analytics@lists.wikimedia.org

13 participants
10 discussions

Community health metrics kit: Input needed!
by Joe Sutherland 27 Feb '20

27 Feb '20

Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction. We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary. Right now, we'd like to know: * Which metrics make sense to collect? Which don't? What are we missing? * Where would such a tool ideally be hosted? Where would you normally look for statistics like these? * We are aware of the overlap in scope between this and Wikistats < https://stats.wikimedia.org/v2/#/all-projects> — how might these tools coexist? Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page. Looking forward to reading your thoughts. best, Joe P.S.: Please feel free to CC me in conversations that might happen on this list! [1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data. -- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks

7 8

Unscheduled reboot of stat1007
by Luca Toscano 22 Nov '18

22 Nov '18

Hi everybody, as FYI today I rebooted stat1007 due to unexpected maintenance (an error from my side) while investigating a Spark2 issue (that is now fixed). Apologies if this has impacted your work! Luca

1 0

Purging schedule for geoeditors_daily dataset
by Neil Patel Quinn 21 Nov '18

21 Nov '18

Hey there! Could someone from Analytics clarify the purging schedule for geoeditors_daily and add it on Wikitech <https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors#Ge…>? I've added some information based on my experience using the dataset, but it may not be fully accurate. I wrote: *Because these tables contain the countries of individual editors, we only keep the data corresponding to the two most recent full months (the month of the latest mediawiki_history snapshot and the previous).*

2 4

Upcoming move of users from stat1005 to stat1007
by Luca Toscano 20 Nov '18

20 Nov '18

Hi everybody, as part of https://phabricator.wikimedia.org/T205846 we are going to ask to all the stat1005's users to move to stat1007 during the next two weeks. The deadline is November 14th, by which time ssh access to stat1005 will be removed. Background: on stat1005 we have a GPU (more details in https://phabricator.wikimedia.org/T148843) that has been sitting there for almost two years, and it would be great to try to make it work during the next months. This effort will require a lot of tests/reboots/etc.. that can of course impact ongoing work of all of you, so we prefer to move everybody to another identical machine beforehand. Please reach out to me or to the analytics team in T205846 or IRC (#wikimedia-analytics on Freenode) if you have any questions/doubts/blocker/etc.., we are not going to enforce the deadline if anybody will raise concerns or blockers of course. It would be great to move everybody by Nov 14th but we surely don't want to disrupt any ongoing important work. I am going to update the Wikitech documentation about stat1005 and stat1007 as soon as possible, for the moment keep in mind that stat1007 will take over completely everything that stat1005 currently does. I have already copied over all the stat1005 directories to stat1007, and I'll periodically sync them during the following days. If you don't find anything important, please add a note in T205846. Thanks a lot and sorry for the trouble, Luca (on behalf of the Analytics team)

1 2

Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas
by Andrew Otto 19 Nov '18

19 Nov '18

> Does "fixed" mean that the missing data already been backfilled? I’m seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12. Hm. Fixed means the data has been refined to Hive. I didn’t check on Druid EL data. Marcel, can you check up on that? This sounds related, we might need to figure out how to re run the Druid ingestion if/when base data partitions change. On Thu, Nov 15, 2018 at 10:17 AM Tilman Bayer <tbayer(a)wikimedia.org> wrote: > Hi Andrew, > > PS, offtopic: reminder that data-analysts(a)wikimedia.org has been > deprecated/retired a while ago; our team can be reached at > product-analytics(a)wikimedia.org (CCed) > > On Thu, Nov 15, 2018 at 7:13 AM Tilman Bayer <tbayer(a)wikimedia.org> wrote: > >> Does "fixed" mean that the missing data already been backfilled? I'm >> seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the >> timespans between around 6am-16pm on November 13, and 7am-10am on November >> 12. >> >> On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto <otto(a)wikimedia.org> wrote: >> >>> OH I’m sorry! There is a Phab task, and this is fixed. >>> https://phabricator.wikimedia.org/T209407 >>> >>> Very sorry, I should have updated and linked the task. >>> >>> On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc <gilles(a)wikimedia.org> >>> wrote: >>> >>>> Is this issue still ongoing? Is there a corresponding Phabricator task >>>> to follow? >>>> >>>> On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto <otto(a)wikimedia.org> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Yesterday we upgraded the Hadoop cluster to a newer version. It seems >>>>> that along the way the job that imports EventLogging data into Hive has >>>>> started failing for some EventLogging datasets. I’m still investigating, >>>>> but it seems that tables (I believe ones that need to have their schema >>>>> modified during refinement) are failing refinement. The list of affected >>>>> tables at this moment is large, and will likely only grow as time goes on. >>>>> >>>>> We’re looking into this and will try to have things fixed ASAP. For >>>>> now you might expect Hive EventLogging data to be delayed. >>>>> >>>>> Sorry for the trouble, and we’ll keep you updated! >>>>> -Andrew Otto >>>>> Systems Engineer, WMF >>>>> >>>>> >>>>> >>>>> The list of currently affected tables is: >>>>> >>>>> AdvancedSearchRequest >>>>> CentralAuth >>>>> CentralNoticeBannerHistory >>>>> CentralNoticeImpression >>>>> CentralNoticeTiming >>>>> ChangesListFilterGrouping >>>>> ChangesListFilters >>>>> CitationUsage >>>>> CitationUsagePageLoad >>>>> ContentTranslation >>>>> ContentTranslationAbuseFilter >>>>> ContentTranslationCTA >>>>> ContentTranslationError >>>>> ContentTranslationSuggestion >>>>> CpuBenchmark >>>>> EchoInteraction >>>>> EchoMail >>>>> EditAttemptStep >>>>> EditConflict >>>>> EditorActivation >>>>> EUCCVisit >>>>> EventError >>>>> FlowReplies >>>>> GettingStartedRedirectImpression >>>>> GuidedTourButtonClick >>>>> GuidedTourExited >>>>> GuidedTourExternalLinkActivation >>>>> GuidedTourGuiderHidden >>>>> GuidedTourGuiderImpression >>>>> LandingPageImpression >>>>> MediaViewer >>>>> MediaWikiPingback >>>>> MobileAppCategorizationAttempts >>>>> MobileAppUploadAttempts >>>>> MobileWebMainMenuClickTracking >>>>> MobileWebSearch >>>>> MobileWebUIClickTracking >>>>> MobileWikiAppAppearanceSettings >>>>> MobileWikiAppArticleSuggestions >>>>> MobileWikiAppCreateAccount >>>>> MobileWikiAppDailyStats >>>>> MobileWikiAppEdit >>>>> MobileWikiAppFeed >>>>> MobileWikiAppFeedConfigure >>>>> MobileWikiAppFindInPage >>>>> MobileWikiAppInstallReferrer >>>>> MobileWikiAppIntents >>>>> MobileWikiAppiOSFeed >>>>> MobileWikiAppiOSLoginAction >>>>> MobileWikiAppiOSReadingLists >>>>> MobileWikiAppiOSSearch >>>>> MobileWikiAppiOSSessions >>>>> MobileWikiAppiOSSettingAction >>>>> MobileWikiAppiOSUserHistory >>>>> MobileWikiAppLangSelect >>>>> MobileWikiAppLanguageSearching >>>>> MobileWikiAppLanguageSettings >>>>> MobileWikiAppLinkPreview >>>>> MobileWikiAppLogin >>>>> MobileWikiAppMediaGallery >>>>> MobileWikiAppNavMenu >>>>> MobileWikiAppNotificationInteraction >>>>> MobileWikiAppNotificationPreferences >>>>> MobileWikiAppOfflineLibrary >>>>> MobileWikiAppOnboarding >>>>> MobileWikiAppOnThisDay >>>>> MobileWikiAppPageScroll >>>>> MobileWikiAppProtectedEditAttempt >>>>> MobileWikiAppRandomizer >>>>> MobileWikiAppReadingLists >>>>> MobileWikiAppSavedPages >>>>> MobileWikiAppSearch >>>>> MobileWikiAppSessions >>>>> MobileWikiAppShareAFact >>>>> MobileWikiAppStuffHappens >>>>> MobileWikiAppTabs >>>>> MobileWikiAppToCInteraction >>>>> MobileWikiAppWidgets >>>>> MobileWikiAppWiktionaryPopup >>>>> MultimediaViewerAttribution >>>>> MultimediaViewerDimensions >>>>> MultimediaViewerDuration >>>>> MultimediaViewerNetworkPerformance >>>>> NavigationTiming >>>>> PageIssues >>>>> Popups >>>>> PrefUpdate >>>>> Print >>>>> QuickSurveyInitiation >>>>> QuickSurveysResponses >>>>> ReadingDepth >>>>> ResourceTiming >>>>> SaveTiming >>>>> SearchSatisfaction >>>>> SearchSatisfactionErrors >>>>> ServerSideAccountCreation >>>>> TemplateWizard >>>>> TestSearchSatisfaction2 >>>>> TranslationRecommendationAPIRequests >>>>> TranslationRecommendationUIRequests >>>>> TranslationRecommendationUserAction >>>>> TwoColConflictConflict >>>>> UniversalLanguageSelector >>>>> UploadWizardErrorFlowEvent >>>>> UploadWizardExceptionFlowEvent >>>>> UploadWizardFlowEvent >>>>> UploadWizardStep >>>>> UploadWizardTutorialActions >>>>> UploadWizardUploadFlowEvent >>>>> VirtualPageView >>>>> WikidataCompletionSearchClicks >>>>> WikimediaBlogVisit >>>>> WikipediaPortal >>>>> WikipediaZeroUsage >>>>> WMDEBannerEvents >>>>> WMDEBannerSizeIssue >>>>> _______________________________________________ >>>>> Analytics mailing list >>>>> Analytics(a)lists.wikimedia.org >>>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>>>> >>>> _______________________________________________ >>>> Analytics mailing list >>>> Analytics(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>>> >>> >> >> -- >> Tilman Bayer >> Senior Analyst >> Wikimedia Foundation >> IRC (Freenode): HaeB >> > > > -- > Tilman Bayer > Senior Analyst > Wikimedia Foundation > IRC (Freenode): HaeB >

3 3

EventLogging Hive Refine currently stalled for some Schemas
by Andrew Otto 15 Nov '18

15 Nov '18

Hi all, Yesterday we upgraded the Hadoop cluster to a newer version. It seems that along the way the job that imports EventLogging data into Hive has started failing for some EventLogging datasets. I’m still investigating, but it seems that tables (I believe ones that need to have their schema modified during refinement) are failing refinement. The list of affected tables at this moment is large, and will likely only grow as time goes on. We’re looking into this and will try to have things fixed ASAP. For now you might expect Hive EventLogging data to be delayed. Sorry for the trouble, and we’ll keep you updated! -Andrew Otto Systems Engineer, WMF The list of currently affected tables is: AdvancedSearchRequest CentralAuth CentralNoticeBannerHistory CentralNoticeImpression CentralNoticeTiming ChangesListFilterGrouping ChangesListFilters CitationUsage CitationUsagePageLoad ContentTranslation ContentTranslationAbuseFilter ContentTranslationCTA ContentTranslationError ContentTranslationSuggestion CpuBenchmark EchoInteraction EchoMail EditAttemptStep EditConflict EditorActivation EUCCVisit EventError FlowReplies GettingStartedRedirectImpression GuidedTourButtonClick GuidedTourExited GuidedTourExternalLinkActivation GuidedTourGuiderHidden GuidedTourGuiderImpression LandingPageImpression MediaViewer MediaWikiPingback MobileAppCategorizationAttempts MobileAppUploadAttempts MobileWebMainMenuClickTracking MobileWebSearch MobileWebUIClickTracking MobileWikiAppAppearanceSettings MobileWikiAppArticleSuggestions MobileWikiAppCreateAccount MobileWikiAppDailyStats MobileWikiAppEdit MobileWikiAppFeed MobileWikiAppFeedConfigure MobileWikiAppFindInPage MobileWikiAppInstallReferrer MobileWikiAppIntents MobileWikiAppiOSFeed MobileWikiAppiOSLoginAction MobileWikiAppiOSReadingLists MobileWikiAppiOSSearch MobileWikiAppiOSSessions MobileWikiAppiOSSettingAction MobileWikiAppiOSUserHistory MobileWikiAppLangSelect MobileWikiAppLanguageSearching MobileWikiAppLanguageSettings MobileWikiAppLinkPreview MobileWikiAppLogin MobileWikiAppMediaGallery MobileWikiAppNavMenu MobileWikiAppNotificationInteraction MobileWikiAppNotificationPreferences MobileWikiAppOfflineLibrary MobileWikiAppOnboarding MobileWikiAppOnThisDay MobileWikiAppPageScroll MobileWikiAppProtectedEditAttempt MobileWikiAppRandomizer MobileWikiAppReadingLists MobileWikiAppSavedPages MobileWikiAppSearch MobileWikiAppSessions MobileWikiAppShareAFact MobileWikiAppStuffHappens MobileWikiAppTabs MobileWikiAppToCInteraction MobileWikiAppWidgets MobileWikiAppWiktionaryPopup MultimediaViewerAttribution MultimediaViewerDimensions MultimediaViewerDuration MultimediaViewerNetworkPerformance NavigationTiming PageIssues Popups PrefUpdate Print QuickSurveyInitiation QuickSurveysResponses ReadingDepth ResourceTiming SaveTiming SearchSatisfaction SearchSatisfactionErrors ServerSideAccountCreation TemplateWizard TestSearchSatisfaction2 TranslationRecommendationAPIRequests TranslationRecommendationUIRequests TranslationRecommendationUserAction TwoColConflictConflict UniversalLanguageSelector UploadWizardErrorFlowEvent UploadWizardExceptionFlowEvent UploadWizardFlowEvent UploadWizardStep UploadWizardTutorialActions UploadWizardUploadFlowEvent VirtualPageView WikidataCompletionSearchClicks WikimediaBlogVisit WikipediaPortal WikipediaZeroUsage WMDEBannerEvents WMDEBannerSizeIssue

3 3

Request for pageview statistics pre 2015
by William Corin David East, Mr 14 Nov '18

14 Nov '18

Hi, As part of my research on political behavior, I’m working on a replication project of using pageview statistics to predict election outcomes, as done by Yessari et al (2016). The underlying replication relies on pageview statistics for the 2009 and 2014 EU election, but the WMFL pageview API cuts data before 2015. I was informed of stats.grok.se database, but that has been inactive for about 1.5 years now. My question: Is there any archive of pageview statistics for before 2015? If so, where could I find it? Best wishes, and thanks in advance, William East McGill University, Department of Political Science. 855 Sherbrooke St. West, Montreal, QC, Canada H3A 2T7

4 3

Pageviews by agent for May 18-21 2015
by Jennifer Pan 13 Nov '18

13 Nov '18

Hi there, I'm an assistant professor in the Department of Communication at Stanford. My co-author, Molly Roberts (Political Science, UCSD), and I are working on a paper examining the effect of China's 2015 block of Chinese language wikipedia on pageviews, which builds on our previous work on censorship in China. We are using the block to conduct a interrupted time series design to measure the effect of censorship on Chinese users. Our main finding is that Chinese users were using Wikipedia to browse (starting at the home page), and the block influenced users' ability to explore and encounter unexpected information. One question we have is whether the pageviews we observe are driven by bots and spiders. We know that the wikimedia rest api provides this information going back to July 1 2015. Since the China block of Wikipedia was on May 19, 2015, we are wondering if there is pageview data by agent type for zh.wikipedia.org pages (all or some subset like most popular) going back to May 2015 (specifically May 18-21, 2015)? From https://meta.wikimedia.org/wiki/Research:Timeline_of_Wikimedia_analytics, it says that pageview data is available in bulk starting on May 1, 2015, so we thought maybe there was some chance this data exists. Any suggestions would be greatly appreciated, and if this is not possible, please let us know. Thank you! Jennifer Pan

4 3

Analytics Hadoop Cloudera upgrade scheduled for Nov 12th (Monday) at 14:00 CEST
by Luca Toscano 08 Nov '18

08 Nov '18

Hi everybody, the Analytics team will shutdown completely the Hadoop cluster for a couple of hours on Monday Nov 12th at 14:00 CEST to upgrade the Cloudera distribution to 5.15 (currently 5.10). No big updates but only a collection of small/medium fixes that (hopefully) will improve the reliability of our cluster. For more info, please check https://phabricator.wikimedia.org/T204759. This means that tools like HDFS, Hive, Oozie, etc.. will not be available during the maintenance window, so if this impacts your work please reach out to us so we can chat about it and possibly re-schedule if needed (in the task or #wikimedia-analytics on Freenode IRC). Thanks a lot for the patience, we are trying to do our best to keep all our systems as up to date as possible :) Luca (on behalf of the Analytics team)

1 0

[docs] Non-public ticket linked from Analytics/Data access
by Martin Urbanec 02 Nov '18

02 Nov '18

Hi, I'm reading through Analytics/Data access on Wikitech and I can find a non-public ticket T141634 linked in chapter "Of special interest for external researchers". As this chapter looks to be written for non-WMF staff, I think this link should be either replaced with public task or replaced with instructions how should a request look like and where should it be created (using which tags and so on). As I don't see the task and don't know what it contains, I'm notifying you here. Could somebody with enough privileges fix it? Thanks! Martin

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics November 2018