Pfthb

I am using 2.4.0 Spark-core and Spark-sql.

I am trying to create sparkSession and then creating a range and then writing it to table.

Using Scala following code works

val sparkSession = SparkSession
 .builder()
 .appName("Java Spark SQL basic example")
 .config("spark.master", "local")
 .getOrCreate();
 sparkSession.range(10).write.option("path", "/tmp/test").saveAsTable("testData")

But when I perform same steps using java, it fails.

 SparkSession sparkSession = SparkSession.builder()
 .appName("Java Spark SQL basic example")
 .config("spark.master", "local")
 .getOrCreate();

 //create table
 sparkSession.range(10).write().mode(SaveMode.Overwrite).option("path", "/tmp/test").saveAsTable("testData");

Th error stack trace is:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 10582
 at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563)
 at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338)
 at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103)
 at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)
 at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
 at scala.collection.Iterator.foreach(Iterator.scala:937)
 at scala.collection.Iterator.foreach$(Iterator.scala:937)
 at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
 at scala.collection.IterableLike.foreach(IterableLike.scala:70)
 at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
 at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
 at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
 at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
 at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)
 at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
 at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
 at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)
 at scala.collection.TraversableLike.map(TraversableLike.scala:233)
 at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
 at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:194)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)
 at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
 at scala.collection.immutable.List.foreach(List.scala:388)
 at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
 at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
 at scala.collection.immutable.List.flatMap(List.scala:351)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)
 at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:22)
 at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:30)
 at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:78)
 at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:467)
 at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:351)
 at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:283)
 at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueMethod(POJOPropertiesCollector.java:169)
 at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueMethod(BasicBeanDescription.java:223)
 at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:348)
 at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:210)
 at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:153)
 at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1203)
 at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1157)
 at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:481)
 at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:679)
 at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:107)
 at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
 at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
 at org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)
 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:142)
 at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
 at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
 at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
 at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
 at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:668)
 at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:78)
 at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
 at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
 at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
 at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:465)
 at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:444)
 at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:400)
 at org.amir.spark.FirstSparkApp.main(FirstSparkApp.java:16)

It is an issue with paranamer version. Add following dependency before the spark-core/ spark-sql.

<dependency>
 <groupId>com.thoughtworks.paranamer</groupId>
 <artifactId>paranamer</artifactId>
 <version>2.8</version>
</dependency>

Try this:

 sparkSession.range(10).write().mode("append").insertInto("testData");

Try to use 2.2.0 version instead of 2.4.0 because it isn't stable.

answer is

<groupId>org.apache.spark</groupId>
 <artifactId>spark-core_2.11</artifactId>
 <version>2.2.0</version>
</dependency>`