Hi Joshua,
today's delay is unrelated to the problem mentioned by Andrew,
it was due to a silent issue with our scheduling system. We are
investigating it.
In the meantime, the data is now available.
Sorry for the inconvenience!
On Fri, Mar 3, 2023 at 12:38 PM Joshua Haecker <josh(a)predata.com> wrote:
Thanks for this Andrew! Sadly it appears to be
down again, did we try the
patch again? Thanks in advance for any details you can provide.
On Fri, Feb 24, 2023 at 2:50 PM Andrew Otto <otto(a)wikimedia.org> wrote:
Hello!
Here is what I know.
- 2023-02-22T14:33 UTC - This patch
<https://gerrit.wikimedia.org/r/c/operations/puppet/+/890843/3#message-dcf8af5c4e17a81ad79d16fa6cd7aad504432f87>
was merged
- Over the next several hours systemd timers (cron jobs) across a lot of
the fleet stopped running.
- 2023-02-22T17:34 UTC - This patch
<https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/> reverted
the change
After this jobs began to run again. However, because the webrequest
dataset is so huge, it took hours for ingestion of it to catch up.
Downstream jobs that use webrequest as input (including pageviews
computation) began to timeout while waiting for input.
- We have been slowly restarting and recovering jobs now that webrequest
ingestion has caught up again.
I don't know exactly how long data is delayed or when it will be fully
available, but I'd guess: soon / today?
-Andrew Otto
WMF
On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker <josh(a)predata.com> wrote:
Another day another delay, is there a known
broader issue? Just trying
to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker <josh(a)predata.com>
wrote:
> Hi all,
>
> Just curious if there is a known cause for the multiple long delays
> we've had on the AQS API data being available this week? I know periodic
> delays are not uncommon but these seem beyond normal levels.
>
> Thanks!
>
> ~Josh
>
> --
Joshua Haecker
CEO, Co-Founder
Predata, Inc.
josh(a)predata.com
609-865-2181
1201 Pennsylvania Ave NW
<https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g>
Washington, D.C.
<https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g>
*This message and its contents are confidential. If you received this
message in error, do not use or rely upon it. Instead, please inform the
sender and then delete it. Thank you.*
_______________________________________________
Analytics mailing list --
analytics(a)lists.wikimedia.org
To unsubscribe send an email to analytics-leave(a)lists.wikimedia.org
_______________________________________________
Analytics mailing list -- analytics(a)lists.wikimedia.org
To unsubscribe send an email to analytics-leave(a)lists.wikimedia.org
--
Joshua Haecker
CEO, Co-Founder
Predata, Inc.
josh(a)predata.com
609-865-2181
1201 Pennsylvania Ave NW
<https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g>
Washington, D.C.
<https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g>
*This message and its contents are confidential. If you received this
message in error, do not use or rely upon it. Instead, please inform the
sender and then delete it. Thank you.*
_______________________________________________
Analytics mailing list -- analytics(a)lists.wikimedia.org
To unsubscribe send an email to analytics-leave(a)lists.wikimedia.org
--
*Marcel Ruiz Forns** (he/him)*
Senior Software Engineer
_______________________________________________
Analytics mailing list -- analytics(a)lists.wikimedia.org
To unsubscribe send an email to analytics-leave(a)lists.wikimedia.org