Following up on this, data is been re-ingested into druid so turnilo should not show up any holes related to this event.
Thanks,
Nuria
On Thu, Nov 15, 2018 at 8:45 AM Marcel Ruiz Forns mforns@wikimedia.org wrote:
Not all data sources are populated at the same time, the data on Druid is
ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing.
+1 The EL2Druid daily loading job will cover up the holes for the 12th and 13th in 1 or 2 days.
On Thu, Nov 15, 2018 at 5:03 PM Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
Not all data sources are populated at the same time, the data on Druid is ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing.
Thanks,
Nuria
On Thu, Nov 15, 2018 at 7:49 AM Andrew Otto otto@wikimedia.org wrote:
Does "fixed" mean that the missing data already been backfilled? I’m
seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.
Hm. Fixed means the data has been refined to Hive. I didn’t check on Druid EL data. Marcel, can you check up on that? This sounds related, we might need to figure out how to re run the Druid ingestion if/when base data partitions change.
On Thu, Nov 15, 2018 at 10:17 AM Tilman Bayer tbayer@wikimedia.org wrote:
Hi Andrew,
PS, offtopic: reminder that data-analysts@wikimedia.org has been deprecated/retired a while ago; our team can be reached at product-analytics@wikimedia.org (CCed)
On Thu, Nov 15, 2018 at 7:13 AM Tilman Bayer tbayer@wikimedia.org wrote:
Does "fixed" mean that the missing data already been backfilled? I'm seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.
On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto otto@wikimedia.org wrote:
OH I’m sorry! There is a Phab task, and this is fixed. https://phabricator.wikimedia.org/T209407
Very sorry, I should have updated and linked the task.
On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc gilles@wikimedia.org wrote:
> Is this issue still ongoing? Is there a corresponding Phabricator > task to follow? > > On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto otto@wikimedia.org > wrote: > >> Hi all, >> >> Yesterday we upgraded the Hadoop cluster to a newer version. It >> seems that along the way the job that imports EventLogging data into Hive >> has started failing for some EventLogging datasets. I’m still >> investigating, but it seems that tables (I believe ones that need to have >> their schema modified during refinement) are failing refinement. The list >> of affected tables at this moment is large, and will likely only grow as >> time goes on. >> >> We’re looking into this and will try to have things fixed ASAP. >> For now you might expect Hive EventLogging data to be delayed. >> >> Sorry for the trouble, and we’ll keep you updated! >> -Andrew Otto >> Systems Engineer, WMF >> >> >> >> The list of currently affected tables is: >> >> AdvancedSearchRequest >> CentralAuth >> CentralNoticeBannerHistory >> CentralNoticeImpression >> CentralNoticeTiming >> ChangesListFilterGrouping >> ChangesListFilters >> CitationUsage >> CitationUsagePageLoad >> ContentTranslation >> ContentTranslationAbuseFilter >> ContentTranslationCTA >> ContentTranslationError >> ContentTranslationSuggestion >> CpuBenchmark >> EchoInteraction >> EchoMail >> EditAttemptStep >> EditConflict >> EditorActivation >> EUCCVisit >> EventError >> FlowReplies >> GettingStartedRedirectImpression >> GuidedTourButtonClick >> GuidedTourExited >> GuidedTourExternalLinkActivation >> GuidedTourGuiderHidden >> GuidedTourGuiderImpression >> LandingPageImpression >> MediaViewer >> MediaWikiPingback >> MobileAppCategorizationAttempts >> MobileAppUploadAttempts >> MobileWebMainMenuClickTracking >> MobileWebSearch >> MobileWebUIClickTracking >> MobileWikiAppAppearanceSettings >> MobileWikiAppArticleSuggestions >> MobileWikiAppCreateAccount >> MobileWikiAppDailyStats >> MobileWikiAppEdit >> MobileWikiAppFeed >> MobileWikiAppFeedConfigure >> MobileWikiAppFindInPage >> MobileWikiAppInstallReferrer >> MobileWikiAppIntents >> MobileWikiAppiOSFeed >> MobileWikiAppiOSLoginAction >> MobileWikiAppiOSReadingLists >> MobileWikiAppiOSSearch >> MobileWikiAppiOSSessions >> MobileWikiAppiOSSettingAction >> MobileWikiAppiOSUserHistory >> MobileWikiAppLangSelect >> MobileWikiAppLanguageSearching >> MobileWikiAppLanguageSettings >> MobileWikiAppLinkPreview >> MobileWikiAppLogin >> MobileWikiAppMediaGallery >> MobileWikiAppNavMenu >> MobileWikiAppNotificationInteraction >> MobileWikiAppNotificationPreferences >> MobileWikiAppOfflineLibrary >> MobileWikiAppOnboarding >> MobileWikiAppOnThisDay >> MobileWikiAppPageScroll >> MobileWikiAppProtectedEditAttempt >> MobileWikiAppRandomizer >> MobileWikiAppReadingLists >> MobileWikiAppSavedPages >> MobileWikiAppSearch >> MobileWikiAppSessions >> MobileWikiAppShareAFact >> MobileWikiAppStuffHappens >> MobileWikiAppTabs >> MobileWikiAppToCInteraction >> MobileWikiAppWidgets >> MobileWikiAppWiktionaryPopup >> MultimediaViewerAttribution >> MultimediaViewerDimensions >> MultimediaViewerDuration >> MultimediaViewerNetworkPerformance >> NavigationTiming >> PageIssues >> Popups >> PrefUpdate >> Print >> QuickSurveyInitiation >> QuickSurveysResponses >> ReadingDepth >> ResourceTiming >> SaveTiming >> SearchSatisfaction >> SearchSatisfactionErrors >> ServerSideAccountCreation >> TemplateWizard >> TestSearchSatisfaction2 >> TranslationRecommendationAPIRequests >> TranslationRecommendationUIRequests >> TranslationRecommendationUserAction >> TwoColConflictConflict >> UniversalLanguageSelector >> UploadWizardErrorFlowEvent >> UploadWizardExceptionFlowEvent >> UploadWizardFlowEvent >> UploadWizardStep >> UploadWizardTutorialActions >> UploadWizardUploadFlowEvent >> VirtualPageView >> WikidataCompletionSearchClicks >> WikimediaBlogVisit >> WikipediaPortal >> WikipediaZeroUsage >> WMDEBannerEvents >> WMDEBannerSizeIssue >> _______________________________________________ >> Analytics mailing list >> Analytics@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> > _______________________________________________ > Analytics mailing list > Analytics@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics >
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- *Marcel Ruiz Forns* Analytics Developer Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics