Hiya! It works for me if I package on stat1002. I was going to help out on
but that's not urgent.
On Fri, Mar 31, 2017 at 2:22 PM, Dan Andreescu <dandreescu(a)wikimedia.org>
wrote:
Mikhail, I know Joseph was working with that job whose
test is failing,
I'll cc him personally to see if he has any idea. Does this work if you do
it on the cluster? Since it's the weekend, you probably won't get a
response until Monday, let me know if there's anything urgent and we can
try to find a work-around.
On Fri, Mar 31, 2017 at 4:45 PM, Mikhail Popov <mpopov(a)wikimedia.org>
wrote:
Hello! Has anyone experienced issues `mvn
package`-ing
analytics/refinery/source on a local machine?
Wikimedia Analytics Refinery Jobs fails for me as of "Add mediawiki
history spark jobs to refinery-job" (
https://gerrit.wikimedia.org/
r/#/c/325312/)
Here's my `mvn --version`:
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T08:41:47-08:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/
Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.4", arch: "x86_64",
family: "mac"
When I set HEAD to the commit prior to that everything succeeds. Any
commit after that one makes Jobs tests fail with warnings and errors like:
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING] org.spark-project.akka:akka-actor_2.10:2.2.3-shaded-protobuf
requires scala version: 2.10.4
[WARNING] org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
requires scala version: 2.10.4
[WARNING] org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf
requires scala version: 2.10.4
[WARNING] org.apache.spark:spark-core_2.10:1.6.0-cdh5.10.0 requires
scala version: 2.10.4
[WARNING] org.json4s:json4s-jackson_2.10:3.2.10 requires scala version:
2.10.4
[WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version:
2.10.4
[WARNING] org.json4s:json4s-ast_2.10:3.2.10 requires scala version:
2.10.4
[WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version:
2.10.0
[WARNING] Multiple versions of scala libraries detected!
TestDenormalizedRevisionsBuilder:
17/03/31 13:38:41 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
populateDeleteTime
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
ssorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
thodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoade
r.java:317)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
at org.apache.spark.io.SnappyCompressionCodec$.liftedTree1$1(Co
mpressionCodec.scala:169)
at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark
$io$SnappyCompressionCodec$$version$lzycompute(Compression
Codec.scala:168)
at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark
$io$SnappyCompressionCodec$$version(CompressionCodec.scala:168)
at org.apache.spark.io.SnappyCompressionCodec.<init>(Compressio
nCodec.scala:152)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
ConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
legatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.io.CompressionCodec$.createCodec(Compressio
nCodec.scala:72)
at org.apache.spark.io.CompressionCodec$.createCodec(Compressio
nCodec.scala:65)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark
$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBr
oadcast.scala:80)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadc
ast(TorrentBroadcastFactory.scala:34)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(Bro
adcastManager.scala:63)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1334)
at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(D
AGScheduler.scala:1006)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$sch
eduler$DAGScheduler$$submitStage(DAGScheduler.scala:921)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$
spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:924)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$
spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:923)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$sch
eduler$DAGScheduler$$submitStage(DAGScheduler.scala:923)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(D
AGScheduler.scala:861)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOn
Receive(DAGScheduler.scala:1611)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onRe
ceive(DAGScheduler.scala:1603)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onRe
ceive(DAGScheduler.scala:1592)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in
java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativ
eLoader.java:52)
... 33 more
- should put max rev ts when no page state match *** FAILED ***
org.apache.spark.SparkException: Job aborted due to stage failure:
Task serialization failed: java.lang.reflect.InvocationTargetException
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics