Hey Neil,
there were two Yarn jobs running related to your notebooks, I just killed
them, let's see if it solves the problem (you might need to restart again
your notebook). If not, let's open a task and investigate :)
Luca
Il giorno gio 6 feb 2020 alle ore 02:08 Neil Shah-Quinn <
nshahquinn(a)wikimedia.org> ha scritto:
Whoa—I just got the same stopped SparkContext error on
the query even
after restarting the notebook, without an intermediate Java heap space
error. That seems very strange to me.
On Wed, 5 Feb 2020 at 16:09, Neil Shah-Quinn <nshahquinn(a)wikimedia.org>
wrote:
Hey there!
I was running SQL queries via PySpark (using the wmfdata package
<https://github.com/neilpquinn/wmfdata/blob/master/wmfdata/hive.py>) on
SWAP when one of my queries failed with "java.lang.OutofMemoryError: Java
heap space".
After that, when I tried to call the spark.sql function again (via
wmfdata.hive.run), it failed with "java.lang.IllegalStateException: Cannot
call methods on a stopped SparkContext."
When I tried to create a new Spark context using
SparkSession.builder.getOrCreate (whether using wmfdata.spark.get_session
or directly), it returned a SparkContent object properly, but calling the
object's sql function still gave the "stopped SparkContext error".
Any idea what's going on? I assume restarting the notebook kernel would
take care of the problem, but it seems like there has to be a better way to
recover.
Thank you!
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics