Not all data sources are populated at the same time, the data on Druid is ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing.
+1 The EL2Druid daily loading job will cover up the holes for the 12th and 13th in 1 or 2 days.
On Thu, Nov 15, 2018 at 5:03 PM Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
Not all data sources are populated at the same time, the data on Druid is ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing.
Thanks,
Nuria
On Thu, Nov 15, 2018 at 7:49 AM Andrew Otto otto@wikimedia.org wrote:
Does "fixed" mean that the missing data already been backfilled? I’m
seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.
Hm. Fixed means the data has been refined to Hive. I didn’t check on Druid EL data. Marcel, can you check up on that? This sounds related, we might need to figure out how to re run the Druid ingestion if/when base data partitions change.
On Thu, Nov 15, 2018 at 10:17 AM Tilman Bayer tbayer@wikimedia.org wrote:
Hi Andrew,
PS, offtopic: reminder that data-analysts@wikimedia.org has been deprecated/retired a while ago; our team can be reached at product-analytics@wikimedia.org (CCed)
On Thu, Nov 15, 2018 at 7:13 AM Tilman Bayer tbayer@wikimedia.org wrote:
Does "fixed" mean that the missing data already been backfilled? I'm seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.
On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto otto@wikimedia.org wrote:
OH I’m sorry! There is a Phab task, and this is fixed. https://phabricator.wikimedia.org/T209407
Very sorry, I should have updated and linked the task.
On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc gilles@wikimedia.org wrote:
Is this issue still ongoing? Is there a corresponding Phabricator task to follow?
On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto otto@wikimedia.org wrote:
> Hi all, > > Yesterday we upgraded the Hadoop cluster to a newer version. It > seems that along the way the job that imports EventLogging data into Hive > has started failing for some EventLogging datasets. I’m still > investigating, but it seems that tables (I believe ones that need to have > their schema modified during refinement) are failing refinement. The list > of affected tables at this moment is large, and will likely only grow as > time goes on. > > We’re looking into this and will try to have things fixed ASAP. For > now you might expect Hive EventLogging data to be delayed. > > Sorry for the trouble, and we’ll keep you updated! > -Andrew Otto > Systems Engineer, WMF > > > > The list of currently affected tables is: > > AdvancedSearchRequest > CentralAuth > CentralNoticeBannerHistory > CentralNoticeImpression > CentralNoticeTiming > ChangesListFilterGrouping > ChangesListFilters > CitationUsage > CitationUsagePageLoad > ContentTranslation > ContentTranslationAbuseFilter > ContentTranslationCTA > ContentTranslationError > ContentTranslationSuggestion > CpuBenchmark > EchoInteraction > EchoMail > EditAttemptStep > EditConflict > EditorActivation > EUCCVisit > EventError > FlowReplies > GettingStartedRedirectImpression > GuidedTourButtonClick > GuidedTourExited > GuidedTourExternalLinkActivation > GuidedTourGuiderHidden > GuidedTourGuiderImpression > LandingPageImpression > MediaViewer > MediaWikiPingback > MobileAppCategorizationAttempts > MobileAppUploadAttempts > MobileWebMainMenuClickTracking > MobileWebSearch > MobileWebUIClickTracking > MobileWikiAppAppearanceSettings > MobileWikiAppArticleSuggestions > MobileWikiAppCreateAccount > MobileWikiAppDailyStats > MobileWikiAppEdit > MobileWikiAppFeed > MobileWikiAppFeedConfigure > MobileWikiAppFindInPage > MobileWikiAppInstallReferrer > MobileWikiAppIntents > MobileWikiAppiOSFeed > MobileWikiAppiOSLoginAction > MobileWikiAppiOSReadingLists > MobileWikiAppiOSSearch > MobileWikiAppiOSSessions > MobileWikiAppiOSSettingAction > MobileWikiAppiOSUserHistory > MobileWikiAppLangSelect > MobileWikiAppLanguageSearching > MobileWikiAppLanguageSettings > MobileWikiAppLinkPreview > MobileWikiAppLogin > MobileWikiAppMediaGallery > MobileWikiAppNavMenu > MobileWikiAppNotificationInteraction > MobileWikiAppNotificationPreferences > MobileWikiAppOfflineLibrary > MobileWikiAppOnboarding > MobileWikiAppOnThisDay > MobileWikiAppPageScroll > MobileWikiAppProtectedEditAttempt > MobileWikiAppRandomizer > MobileWikiAppReadingLists > MobileWikiAppSavedPages > MobileWikiAppSearch > MobileWikiAppSessions > MobileWikiAppShareAFact > MobileWikiAppStuffHappens > MobileWikiAppTabs > MobileWikiAppToCInteraction > MobileWikiAppWidgets > MobileWikiAppWiktionaryPopup > MultimediaViewerAttribution > MultimediaViewerDimensions > MultimediaViewerDuration > MultimediaViewerNetworkPerformance > NavigationTiming > PageIssues > Popups > PrefUpdate > Print > QuickSurveyInitiation > QuickSurveysResponses > ReadingDepth > ResourceTiming > SaveTiming > SearchSatisfaction > SearchSatisfactionErrors > ServerSideAccountCreation > TemplateWizard > TestSearchSatisfaction2 > TranslationRecommendationAPIRequests > TranslationRecommendationUIRequests > TranslationRecommendationUserAction > TwoColConflictConflict > UniversalLanguageSelector > UploadWizardErrorFlowEvent > UploadWizardExceptionFlowEvent > UploadWizardFlowEvent > UploadWizardStep > UploadWizardTutorialActions > UploadWizardUploadFlowEvent > VirtualPageView > WikidataCompletionSearchClicks > WikimediaBlogVisit > WikipediaPortal > WikipediaZeroUsage > WMDEBannerEvents > WMDEBannerSizeIssue > _______________________________________________ > Analytics mailing list > Analytics@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics