Pfthb

I am following the example from spark-redis connector documentation to create Dataframes from a an existing hash.

The hash built as follows:

127.0.0.1:6379> hset person:1 name John age 30

(integer) 2

127.0.0.1:6379> hset person:2 name Peter age 45

(integer) 2

127.0.0.1:6379> hset person:3 name James age 40

The code to read the hash is as follows -

 SparkSession spark = SparkSession
 .builder()
 .appName("MyApp")
 .master("local[*]")
 .config("spark.redis.host", "localhost")
 .config("spark.redis.port", "6379")
 .getOrCreate();
 Dataset<Row> df = spark.read()
 .format("org.apache.spark.sql.redis")
 .schema(new StructType(new StructField 
 DataTypes.createStructField("id", DataTypes.StringType, true),
 DataTypes.createStructField("name", DataTypes.StringType, false),
 DataTypes.createStructField("age", DataTypes.IntegerType, false)
 )
 )
 .option("keys.pattern", "person:*")
 .option("key.column", "id")
 .load();

 df.show();
 df.printSchema(); 

Output

+----+-----+---+
| id| name|age|
+----+-----+---+
|null| John| 30|
|null|James| 40|
|null|Peter| 45|
+----+-----+---+

root
 |-- id: string (nullable = true)
 |-- name: string (nullable = false)
 |-- age: integer (nullable = false)

I was expecting 1, 2, 3 respectively in the Id column, but get null instead. Any pointers in this regard will help. Also this code is in Java so I am not sure if there an issue with data types.

The version you're using doesn't support it - use the master tip until a new version is released.

xref: https://github.com/RedisLabs/spark-redis/issues/114