Mikhail, I know Joseph was working with that job whose test is failing, I'll cc him personally to see if he has any idea. Does this work if you do it on the cluster? Since it's the weekend, you probably won't get a response until Monday, let me know if there's anything urgent and we can try to find a work-around.
On Fri, Mar 31, 2017 at 4:45 PM, Mikhail Popov mpopov@wikimedia.org wrote:
Hello! Has anyone experienced issues `mvn package`-ing analytics/refinery/source on a local machine?
Wikimedia Analytics Refinery Jobs fails for me as of "Add mediawiki history spark jobs to refinery-job" (https://gerrit.wikimedia.org/ r/#/c/325312/)
Here's my `mvn --version`:
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00) Maven home: /usr/local/Cellar/maven/3.3.9/libexec Java version: 1.8.0_121, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_ 121.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.12.4", arch: "x86_64", family: "mac"
When I set HEAD to the commit prior to that everything succeeds. Any commit after that one makes Jobs tests fail with warnings and errors like:
[INFO] Checking for multiple versions of scala [WARNING] Expected all dependencies to require Scala version: 2.10.4 [WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4 [WARNING] org.spark-project.akka:akka-actor_2.10:2.2.3-shaded-protobuf requires scala version: 2.10.4 [WARNING] org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf requires scala version: 2.10.4 [WARNING] org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf requires scala version: 2.10.4 [WARNING] org.apache.spark:spark-core_2.10:1.6.0-cdh5.10.0 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-jackson_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-ast_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected!
TestDenormalizedRevisionsBuilder: 17/03/31 13:38:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
populateDeleteTime java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44) at org.apache.spark.io.SnappyCompressionCodec$.liftedTree1$1( CompressionCodec.scala:169) at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark$io$ SnappyCompressionCodec$$version$lzycompute(CompressionCodec.scala:168) at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark$io$ SnappyCompressionCodec$$version(CompressionCodec.scala:168) at org.apache.spark.io.SnappyCompressionCodec.<init>( CompressionCodec.scala:152) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance( NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec( CompressionCodec.scala:72) at org.apache.spark.io.CompressionCodec$.createCodec( CompressionCodec.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$ TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) at org.apache.spark.broadcast.TorrentBroadcast.<init>( TorrentBroadcast.scala:80) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast( TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.BroadcastManager.newBroadcast( BroadcastManager.scala:63) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1334) at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks( DAGScheduler.scala:1006) at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:921) at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$ apache$spark$scheduler$DAGScheduler$$submitStage$4. apply(DAGScheduler.scala:924) at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$ apache$spark$scheduler$DAGScheduler$$submitStage$4. apply(DAGScheduler.scala:923) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:923) at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted( DAGScheduler.scala:861) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. doOnReceive(DAGScheduler.scala:1611) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. onReceive(DAGScheduler.scala:1603) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. onReceive(DAGScheduler.scala:1592) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at org.xerial.snappy.SnappyNativeLoader.loadLibrary( SnappyNativeLoader.java:52) ... 33 more
- should put max rev ts when no page state match *** FAILED *** org.apache.spark.SparkException: Job aborted due to stage failure: Task
serialization failed: java.lang.reflect.InvocationTargetException
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics