Mikhail, I know Joseph was working with that job whose test is failing,
I'll cc him personally to see if he has any idea. Does this work if you do
it on the cluster? Since it's the weekend, you probably won't get a
response until Monday, let me know if there's anything urgent and we can
try to find a work-around.
On Fri, Mar 31, 2017 at 4:45 PM, Mikhail Popov <mpopov(a)wikimedia.org> wrote:
Hello! Has anyone experienced issues `mvn
package`-ing
analytics/refinery/source on a local machine?
Wikimedia Analytics Refinery Jobs fails for me as of "Add mediawiki
history spark jobs to refinery-job" (
https://gerrit.wikimedia.org/
r/#/c/325312/)
Here's my `mvn --version`:
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T08:41:47-08:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_
121.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.4", arch: "x86_64",
family: "mac"
When I set HEAD to the commit prior to that everything succeeds. Any
commit after that one makes Jobs tests fail with warnings and errors like:
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING] org.spark-project.akka:akka-actor_2.10:2.2.3-shaded-protobuf
requires scala version: 2.10.4
[WARNING] org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
requires scala version: 2.10.4
[WARNING] org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf
requires scala version: 2.10.4
[WARNING] org.apache.spark:spark-core_2.10:1.6.0-cdh5.10.0 requires
scala version: 2.10.4
[WARNING] org.json4s:json4s-jackson_2.10:3.2.10 requires scala version:
2.10.4
[WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version:
2.10.4
[WARNING] org.json4s:json4s-ast_2.10:3.2.10 requires scala version:
2.10.4
[WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version:
2.10.0
[WARNING] Multiple versions of scala libraries detected!
TestDenormalizedRevisionsBuilder:
17/03/31 13:38:41 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
populateDeleteTime
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
at org.apache.spark.io.SnappyCompressionCodec$.liftedTree1$1(
CompressionCodec.scala:169)
at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark$io$
SnappyCompressionCodec$$version$lzycompute(CompressionCodec.scala:168)
at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark$io$
SnappyCompressionCodec$$version(CompressionCodec.scala:168)
at org.apache.spark.io.SnappyCompressionCodec.<init>(
CompressionCodec.scala:152)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.io.CompressionCodec$.createCodec(
CompressionCodec.scala:72)
at org.apache.spark.io.CompressionCodec$.createCodec(
CompressionCodec.scala:65)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$
TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(
TorrentBroadcast.scala:80)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(
TorrentBroadcastFactory.scala:34)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(
BroadcastManager.scala:63)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1334)
at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(
DAGScheduler.scala:1006)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$
scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:921)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$
apache$spark$scheduler$DAGScheduler$$submitStage$4.
apply(DAGScheduler.scala:924)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$
apache$spark$scheduler$DAGScheduler$$submitStage$4.
apply(DAGScheduler.scala:923)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$
scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:923)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(
DAGScheduler.scala:861)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
doOnReceive(DAGScheduler.scala:1611)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
onReceive(DAGScheduler.scala:1603)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
onReceive(DAGScheduler.scala:1592)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in
java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at org.xerial.snappy.SnappyNativeLoader.loadLibrary(
SnappyNativeLoader.java:52)
... 33 more
- should put max rev ts when no page state match *** FAILED ***
org.apache.spark.SparkException: Job aborted due to stage failure: Task
serialization failed: java.lang.reflect.InvocationTargetException
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics