Root cause fixed.
I just cleared wait_for_weekly_tables, and the pipeline is running again.
On Mon, May 15, 2023 at 11:23 AM Cormac Parle cparle@wikimedia.org wrote:
Thanks Xabriel
On 15 May 2023, at 15:54, Xabriel Collazo Mojica xcollazo@wikimedia.org wrote:
Root cause is that an upstream job on the analytics Airflow instance needed adjustment due to the datacenter switchover.
Should be solved shortly. Details: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_req...
On Mon, May 15, 2023 at 10:41 AM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Looks like latest snapshot of wmf.wikidata_item_page_link is missing. Will investigate further.
hive (default)> show partitions structured_data.commons_entity; OK partition log4j:ERROR No output stream or file set for the appender named [console]. snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 1.031 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_item_page_link; OK partition snapshot=2023-03-20 snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 Time taken: 0.108 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_entity; OK partition snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 0.083 seconds, Fetched: 6 row(s)
On Sun, May 14, 2023 at 8:08 PM < airflow-platform_eng@an-airflow1004.eqiad.wmnet> wrote:
Try 1 out of 6 Exception: Sensor has timed out; run duration of 605282.955168 seconds exceeds the specified timeout of 604800.0. Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=wait_for_weekly_tables&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=wait_for_weekly_tables&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
Cool, muchas gracias!
On Tue, May 16, 2023 at 4:26 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Root cause fixed.
I just cleared wait_for_weekly_tables, and the pipeline is running again.
On Mon, May 15, 2023 at 11:23 AM Cormac Parle cparle@wikimedia.org wrote:
Thanks Xabriel
On 15 May 2023, at 15:54, Xabriel Collazo Mojica xcollazo@wikimedia.org wrote:
Root cause is that an upstream job on the analytics Airflow instance needed adjustment due to the datacenter switchover.
Should be solved shortly. Details: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_req...
On Mon, May 15, 2023 at 10:41 AM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Looks like latest snapshot of wmf.wikidata_item_page_link is missing. Will investigate further.
hive (default)> show partitions structured_data.commons_entity; OK partition log4j:ERROR No output stream or file set for the appender named [console]. snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 1.031 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_item_page_link; OK partition snapshot=2023-03-20 snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 Time taken: 0.108 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_entity; OK partition snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 0.083 seconds, Fetched: 6 row(s)
On Sun, May 14, 2023 at 8:08 PM < airflow-platform_eng@an-airflow1004.eqiad.wmnet> wrote:
Try 1 out of 6 Exception: Sensor has timed out; run duration of 605282.955168 seconds exceeds the specified timeout of 604800.0. Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=wait_for_weekly_tables&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=wait_for_weekly_tables&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
Well, don't thank me yet, as we broke the Spark job. :D
Working on it: https://phabricator.wikimedia.org/T336800
On Tue, May 16, 2023 at 1:48 PM Marco Fossati mfossati@wikimedia.org wrote:
Cool, muchas gracias!
On Tue, May 16, 2023 at 4:26 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Root cause fixed.
I just cleared wait_for_weekly_tables, and the pipeline is running again.
On Mon, May 15, 2023 at 11:23 AM Cormac Parle cparle@wikimedia.org wrote:
Thanks Xabriel
On 15 May 2023, at 15:54, Xabriel Collazo Mojica xcollazo@wikimedia.org wrote:
Root cause is that an upstream job on the analytics Airflow instance needed adjustment due to the datacenter switchover.
Should be solved shortly. Details: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_req...
On Mon, May 15, 2023 at 10:41 AM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Looks like latest snapshot of wmf.wikidata_item_page_link is missing. Will investigate further.
hive (default)> show partitions structured_data.commons_entity; OK partition log4j:ERROR No output stream or file set for the appender named [console]. snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 1.031 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_item_page_link; OK partition snapshot=2023-03-20 snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 Time taken: 0.108 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_entity; OK partition snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 0.083 seconds, Fetched: 6 row(s)
On Sun, May 14, 2023 at 8:08 PM < airflow-platform_eng@an-airflow1004.eqiad.wmnet> wrote:
Try 1 out of 6 Exception: Sensor has timed out; run duration of 605282.955168 seconds exceeds the specified timeout of 604800.0. Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=wait_for_weekly_tables&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=wait_for_weekly_tables&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
Just deployed a hot fix for this issue. It should fix all the failing DAGs.
Details on https://phabricator.wikimedia.org/T336800
On Tue, May 16, 2023 at 2:27 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Well, don't thank me yet, as we broke the Spark job. :D
Working on it: https://phabricator.wikimedia.org/T336800
On Tue, May 16, 2023 at 1:48 PM Marco Fossati mfossati@wikimedia.org wrote:
Cool, muchas gracias!
On Tue, May 16, 2023 at 4:26 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Root cause fixed.
I just cleared wait_for_weekly_tables, and the pipeline is running again.
On Mon, May 15, 2023 at 11:23 AM Cormac Parle cparle@wikimedia.org wrote:
Thanks Xabriel
On 15 May 2023, at 15:54, Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Root cause is that an upstream job on the analytics Airflow instance needed adjustment due to the datacenter switchover.
Should be solved shortly. Details: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_req...
On Mon, May 15, 2023 at 10:41 AM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
Looks like latest snapshot of wmf.wikidata_item_page_link is missing. Will investigate further.
hive (default)> show partitions structured_data.commons_entity; OK partition log4j:ERROR No output stream or file set for the appender named [console]. snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 1.031 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_item_page_link; OK partition snapshot=2023-03-20 snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 Time taken: 0.108 seconds, Fetched: 6 row(s) hive (default)> show partitions wmf.wikidata_entity; OK partition snapshot=2023-03-27 snapshot=2023-04-03 snapshot=2023-04-10 snapshot=2023-04-17 snapshot=2023-04-24 snapshot=2023-05-01 Time taken: 0.083 seconds, Fetched: 6 row(s)
On Sun, May 14, 2023 at 8:08 PM < airflow-platform_eng@an-airflow1004.eqiad.wmnet> wrote:
Try 1 out of 6 Exception: Sensor has timed out; run duration of 605282.955168 seconds exceeds the specified timeout of 604800.0. Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=wait_for_weekly_tables&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=wait_for_weekly_tables&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel