Pfthb

I want to create a new dataframe from existing dataframe in pyspark. The dataframe "df" contains a column named "data" which has rows of dictionary and has a schema as string. And the keys of each dictionary are not fixed.For example the name and address are the keys for the first row dictionary but that would not be the case for other rows they may be different. following is the example for that;

........................................................
 data 
........................................................
 "name": "sam", "address":"uk"
........................................................
"name":"jack" , "address":"aus", "occupation":"job"
.........................................................

How do I convert into the dataframe with individual columns like following.

 name address occupation
 sam uk 
 jack aus job

Convert data to an RDD, then use spark.read.json to convert the RDD into a dataFrame with the schema.

data = [
 "name": "sam", "address":"uk", 
 "name":"jack" , "address":"aus", "occupation":"job"
]

spark = SparkSession.builder.getOrCreate()
df = spark.read.json(sc.parallelize(data)).na.fill('') 
df.show()
+-------+----+----------+
|address|name|occupation|
+-------+----+----------+
| uk| sam| |
| aus|jack| job|
+-------+----+----------+

If the order of rows is not important, this is another way you can do this:

from pyspark import SparkContext
sc = SparkContext()

df = sc.parallelize([
 "name":"jack" , "address":"aus", "occupation":"job",
 "name": "sam", "address":"uk" 
 ]).toDF()

df = df.na.fill('')

df.show()

+-------+----+----------+
|address|name|occupation|
+-------+----+----------+
| aus|jack| job|
| uk| sam| |
+-------+----+----------+