Try 6 out of 6
Exception:
Cannot execute: ['/usr/lib/spark2/bin/spark-submit', '--master', 'yarn', '--conf', 'spark.dynamicAllocation.maxExecutors=450', '--conf', 'spark.executor.memory=2g', '--conf', 'spark.jars.ivy=/tmp/airflow_ivy2', '--conf', 'spark.jars.ivySettings=/etc/maven/ivysettings.xml', '--conf', 'spark.pyspark.python=mjolnir_venv/bin/python', '--conf', 'spark.sql.shuffle.partitions=1000', '--conf', 'spark.yarn.maxAppAttempts=1', '--archives', '/srv/deployment/wikimedia/discovery/analytics/environments/mjolnir/venv.zip#mjolnir_venv', '--jars', 'hdfs://analytics-hadoop/wmf/refinery/current/artifacts/refinery-hive.jar', '--packages', 'org.wikimedia.search:mjolnir:0.7', '--driver-memory', '8g', '--name', 'mjolnir-norm_query_clustering-20210928-spark', '--queue', 'default', '--deploy-mode', 'cluster', '/srv/deployment/wikimedia/discovery/analytics/spark/mjolnir-utilities.py', 'norm_query_clustering', '--date', '20210928', '--output-path', 'hdfs://analytics-hadoop/wmf/data/discovery/mjolnir/query_clustering/date=20210928/algorithm=norm_query', '--brokers', 'kafka-jumbo1001.eqiad.wmnet:9092,kafka-jumbo1002.eqiad.wmnet:9092,kafka-jumbo1003.eqiad.wmnet:9092', '--clicks-table', 'mjolnir.query_clicks_ltr', '--min-sessions-per-query', '10', '--output-table', 'mjolnir.query_clustering', '--top-n', '5', '--topic-request', 'mjolnir.msearch-prod-request', '--topic-response', 'mjolnir.msearch-prod-response']. Error code is: 1.
Log: Link
Host: an-airflow1001.eqiad.wmnet
Log file: /var/log/airflow/mjolnir/norm_query_clustering/2021-09-28T08:08:14.395129+00:00.log
Mark success: Link