Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
Another day another delay, is there a known broader issue? Just trying to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker josh@predata.com wrote:
Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
--
Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW Washington, D.C.
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.*
hi,
Could you please clarify what API endpoints you are calling, what you expect to see, and what you actually see?
Thanks, Kosta
On 24. Feb 2023, at 13:09, Joshua Haecker josh@predata.com wrote:
Another day another delay, is there a known broader issue? Just trying to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker <josh@predata.com mailto:josh@predata.com> wrote:
Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
-- Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com mailto:josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW Washington, D.C.
This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you. _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Hello!
Here is what I know. - 2023-02-22T14:33 UTC - This patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/890843/3#message-dcf8af5c4e17a81ad79d16fa6cd7aad504432f87 was merged - Over the next several hours systemd timers (cron jobs) across a lot of the fleet stopped running. - 2023-02-22T17:34 UTC - This patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/ reverted the change After this jobs began to run again. However, because the webrequest dataset is so huge, it took hours for ingestion of it to catch up. Downstream jobs that use webrequest as input (including pageviews computation) began to timeout while waiting for input. - We have been slowly restarting and recovering jobs now that webrequest ingestion has caught up again.
I don't know exactly how long data is delayed or when it will be fully available, but I'd guess: soon / today?
-Andrew Otto WMF
On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker josh@predata.com wrote:
Another day another delay, is there a known broader issue? Just trying to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker josh@predata.com wrote:
Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
--
Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW Washington, D.C.
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.* _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Thanks for this Andrew! Sadly it appears to be down again, did we try the patch again? Thanks in advance for any details you can provide.
On Fri, Feb 24, 2023 at 2:50 PM Andrew Otto otto@wikimedia.org wrote:
Hello!
Here is what I know.
- 2023-02-22T14:33 UTC - This patch
- Over the next several hours systemd timers (cron jobs) across a lot of
the fleet stopped running.
- 2023-02-22T17:34 UTC - This patch
https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/ reverted the change After this jobs began to run again. However, because the webrequest dataset is so huge, it took hours for ingestion of it to catch up. Downstream jobs that use webrequest as input (including pageviews computation) began to timeout while waiting for input.
- We have been slowly restarting and recovering jobs now that webrequest
ingestion has caught up again.
I don't know exactly how long data is delayed or when it will be fully available, but I'd guess: soon / today?
-Andrew Otto WMF
On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker josh@predata.com wrote:
Another day another delay, is there a known broader issue? Just trying to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker josh@predata.com wrote:
Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
--
Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g Washington, D.C. https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.*
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Hi Joshua, today's delay is unrelated to the problem mentioned by Andrew, it was due to a silent issue with our scheduling system. We are investigating it. In the meantime, the data is now available. Sorry for the inconvenience!
On Fri, Mar 3, 2023 at 12:38 PM Joshua Haecker josh@predata.com wrote:
Thanks for this Andrew! Sadly it appears to be down again, did we try the patch again? Thanks in advance for any details you can provide.
On Fri, Feb 24, 2023 at 2:50 PM Andrew Otto otto@wikimedia.org wrote:
Hello!
Here is what I know.
- 2023-02-22T14:33 UTC - This patch
- Over the next several hours systemd timers (cron jobs) across a lot of
the fleet stopped running.
- 2023-02-22T17:34 UTC - This patch
https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/ reverted the change After this jobs began to run again. However, because the webrequest dataset is so huge, it took hours for ingestion of it to catch up. Downstream jobs that use webrequest as input (including pageviews computation) began to timeout while waiting for input.
- We have been slowly restarting and recovering jobs now that webrequest
ingestion has caught up again.
I don't know exactly how long data is delayed or when it will be fully available, but I'd guess: soon / today?
-Andrew Otto WMF
On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker josh@predata.com wrote:
Another day another delay, is there a known broader issue? Just trying to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker josh@predata.com wrote:
Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
--
Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g Washington, D.C. https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.*
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
-- Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW Washington, D.C.
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.* _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Thanks Marcel!
On Fri, Mar 3, 2023 at 4:59 PM Marcel Ruiz Forns mforns@wikimedia.org wrote:
Hi Joshua, today's delay is unrelated to the problem mentioned by Andrew, it was due to a silent issue with our scheduling system. We are investigating it. In the meantime, the data is now available. Sorry for the inconvenience!
On Fri, Mar 3, 2023 at 12:38 PM Joshua Haecker josh@predata.com wrote:
Thanks for this Andrew! Sadly it appears to be down again, did we try the patch again? Thanks in advance for any details you can provide.
On Fri, Feb 24, 2023 at 2:50 PM Andrew Otto otto@wikimedia.org wrote:
Hello!
Here is what I know.
- 2023-02-22T14:33 UTC - This patch
- Over the next several hours systemd timers (cron jobs) across a lot of
the fleet stopped running.
- 2023-02-22T17:34 UTC - This patch
https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/ reverted the change After this jobs began to run again. However, because the webrequest dataset is so huge, it took hours for ingestion of it to catch up. Downstream jobs that use webrequest as input (including pageviews computation) began to timeout while waiting for input.
- We have been slowly restarting and recovering jobs now that webrequest
ingestion has caught up again.
I don't know exactly how long data is delayed or when it will be fully available, but I'd guess: soon / today?
-Andrew Otto WMF
On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker josh@predata.com wrote:
Another day another delay, is there a known broader issue? Just trying to provide reasonable expectations of when data might be available. Thanks
On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker josh@predata.com wrote:
Hi all,
Just curious if there is a known cause for the multiple long delays we've had on the AQS API data being available this week? I know periodic delays are not uncommon but these seem beyond normal levels.
Thanks!
~Josh
--
Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g Washington, D.C. https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.*
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
-- Joshua Haecker CEO, Co-Founder Predata, Inc.
josh@predata.com 609-865-2181 1201 Pennsylvania Ave NW https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g Washington, D.C. https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g
*This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.* _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
-- *Marcel Ruiz Forns** (he/him)* Senior Software Engineer _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org