Pfthb

Trying to read a simple csv file and load it in a dataframe throw a java.lang.ArrayIndexOutOfBoundsException.

As I am new to Scala I may have missed something trivial, however a thorough search both in google and stackoverflow lead nothing.

The code is the following:

 import org.apache.spark.sql.SparkSession


 object TransformInitial 
 def main(args: Array[String]): Unit = 

 val session = SparkSession.builder.master("local").appName("test").getOrCreate()
 val df = session.read.format("csv").option("header", "true").option("inferSchema", "true").option("delimiter",",").load("data_sets/small_test.csv")

 df.show()
 
 

small_test.csv is as simple as possible:

v1,v2,v3
0,1,2
3,4,5

Here is the actual pom of this Maven project:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>Scala_tests</groupId>
 <artifactId>Scala_tests</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <build>
 <sourceDirectory>src</sourceDirectory>
 <resources>
 <resource>
 <directory>src</directory>
 <excludes>
 <exclude>**/*.java</exclude>
 </excludes>
 </resource>
 </resources>
 <plugins>
 <plugin>
 <artifactId>maven-compiler-plugin</artifactId>
 <version>3.8.0</version>
 <configuration>
 <source>1.8</source>
 <target>1.8</target>
 </configuration>
 </plugin>
 <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->


 </plugins>
 </build>
 <dependencies>
 <dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-core_2.12</artifactId>
 <version>2.4.0</version>
 </dependency>
 <dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-sql_2.12</artifactId>
 <version>2.4.0</version>
 </dependency>

 </dependencies>
</project>

Execution of the code throw the following

java.lang.ArrayIndexOutOfBoundsException:

18/11/09 12:03:31 INFO FileSourceStrategy: Pruning directories with: 
18/11/09 12:03:31 INFO FileSourceStrategy: Post-Scan Filters: (length(trim(value#0, None)) > 0)
18/11/09 12:03:31 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
18/11/09 12:03:31 INFO FileSourceScanExec: Pushed Filters: 
18/11/09 12:03:31 INFO CodeGenerator: Code generated in 413.859722 ms
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 10582
 at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563)
 at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338)
 at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103)
 at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:90)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58)
 at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
 at scala.collection.Iterator.foreach(Iterator.scala:929)
 at scala.collection.Iterator.foreach$(Iterator.scala:929)
 at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
 at scala.collection.IterableLike.foreach(IterableLike.scala:71)
 at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
 at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
 at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
 at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
 at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:58)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:176)
 at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
 at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
 at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
 at scala.collection.TraversableLike.map(TraversableLike.scala:234)
 at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
 at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:191)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:170)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:169)
 at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
 at scala.collection.immutable.List.foreach(List.scala:389)
 at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
 at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
 at scala.collection.immutable.List.flatMap(List.scala:352)
 at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)
 at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:22)
 at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:30)
 at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.findImplicitPropertyName(ScalaAnnotationIntrospectorModule.scala:78)
 at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:467)
 at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:351)
 at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:283)
 at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueMethod(POJOPropertiesCollector.java:169)
 at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueMethod(BasicBeanDescription.java:223)
 at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:348)
 at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:210)
 at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:153)
 at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1203)
 at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1157)
 at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:481)
 at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:679)
 at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:107)
 at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
 at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
 at org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:52)
 at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:142)
 at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
 at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
 at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:247)
 at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:339)
 at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
 at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3384)
 at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2545)
 at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3365)
 at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:78)
 at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
 at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
 at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3365)
 at org.apache.spark.sql.Dataset.head(Dataset.scala:2545)
 at org.apache.spark.sql.Dataset.take(Dataset.scala:2759)
 at org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.infer(CSVDataSource.scala:232)
 at org.apache.spark.sql.execution.datasources.csv.CSVDataSource.inferSchema(CSVDataSource.scala:68)
 at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:63)
 at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$12(DataSource.scala:183)
 at scala.Option.orElse(Option.scala:289)
 at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:180)
 at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
 at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
 at TransformInitial$.main(TransformInitial.scala:9)
 at TransformInitial.main(TransformInitial.scala)

For the record eclipse version is 2018-09 (4.9.0).

I've hunted for special characters in the csv with a cat -A. It yield nothing.
I'm out of options, something trivial must be missing but I can't put a finger on it.

I'm not sure exactly what is causing your error, since the code works for me. It could be related to the version of the Scala compiler that you are using, since there's no information about that in your Maven file.

I have posted my complete solution—using SBT— to GitHub. To exectute the code, you'll need to install SBT, cd to the checked out source's root folder, then run the following command:

$ sbt run

BTW, I changed your code to take advantage of a more idiomatic Scala conventions, and also used the csv function to load your file. The new Scala code looks like this:

import org.apache.spark.sql.SparkSession

// Extending App is more idiomatic than writing a "main" function.
object TransformInitial
extends App 

 val session = SparkSession.builder.master("local").appName("test").getOrCreate()

 // As of Spark 2.0, it's easier to read CSV files.
 val df = session.read.option("header", "true").option("inferSchema", "true").csv("data_sets/small_test.csv")

 df.show()

 // Shutdown gracefully.
 session.stop()

Note that I also removed the redundant delimiter option.