Transform CSV Column Values into Single Row
Multi tool use
up vote
0
down vote
favorite
My data in CSV is like this(Expected Image):
Actual Data
And I want to convert this Data into:
Expected Data
(hivetablename.hivecolumnname = dbtablename.dbtablecolumn)
By joining the multiple Row values into a Single row value like above.
Please note that 'AND' is a Literal between the condition to be built, which would appear until the second last record.
Once the Last Record is reached, Only the condition would appear(xx=yy)
I wish the result to be in SCALA SPARK.
Many thanks in advance!
scala apache-spark-2.0
asked Nov 9 at 13:54
Farhan Soomro
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
up vote
0
down vote
favorite
My data in CSV is like this(Expected Image):
Actual Data
And I want to convert this Data into:
Expected Data
(hivetablename.hivecolumnname = dbtablename.dbtablecolumn)
By joining the multiple Row values into a Single row value like above.
Please note that 'AND' is a Literal between the condition to be built, which would appear until the second last record.
Once the Last Record is reached, Only the condition would appear(xx=yy)
I wish the result to be in SCALA SPARK.
Many thanks in advance!
scala apache-spark-2.0
asked Nov 9 at 13:54
Farhan Soomro
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Could you please share what have you tried so far?
– ulubeyn
Nov 9 at 14:40
Many thanks for the response. Im a learner so couldnt go much further. Heres what Ive been able to achieve val dfconf= spark.read.option("header", "true").csv("file:///conf.csv").alias("conf"); dfconf.show(10, false) val newDF1 = dfconf.withColumn("Y",lit("1")) val newDF2 = newDF1.withColumn("Z",lit(",")) val newDF3 = newDF2.withColumn("comb",concat($"hivetablename",lit("."),$"hivetablecolumn",lit("="),$"dbtablename",lit("."),$"dbtablecolumn")) newDF3.show() val newDF4 = newDF3.groupBy("Y").agg(collect_set("z").as("combined"))
– Farhan Soomro
Nov 9 at 14:51
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
My data in CSV is like this(Expected Image):
Actual Data
And I want to convert this Data into:
Expected Data
(hivetablename.hivecolumnname = dbtablename.dbtablecolumn)
By joining the multiple Row values into a Single row value like above.
Please note that 'AND' is a Literal between the condition to be built, which would appear until the second last record.
Once the Last Record is reached, Only the condition would appear(xx=yy)
I wish the result to be in SCALA SPARK.
Many thanks in advance!
scala apache-spark-2.0
asked Nov 9 at 13:54
Farhan Soomro
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
My data in CSV is like this(Expected Image):
Actual Data
And I want to convert this Data into:
Expected Data
(hivetablename.hivecolumnname = dbtablename.dbtablecolumn)
By joining the multiple Row values into a Single row value like above.
Please note that 'AND' is a Literal between the condition to be built, which would appear until the second last record.
Once the Last Record is reached, Only the condition would appear(xx=yy)
I wish the result to be in SCALA SPARK.
Many thanks in advance!
scala apache-spark-2.0
scala apache-spark-2.0
asked Nov 9 at 13:54
Farhan Soomro
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 9 at 13:54
Farhan Soomro
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Nov 9 at 14:28
asked Nov 9 at 13:54
Farhan Soomro
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 9 at 13:54
Farhan Soomro
12
asked Nov 9 at 13:54
Farhan Soomro
12
12
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Farhan Soomro is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Could you please share what have you tried so far?
– ulubeyn
Nov 9 at 14:40
Many thanks for the response. Im a learner so couldnt go much further. Heres what Ive been able to achieve val dfconf= spark.read.option("header", "true").csv("file:///conf.csv").alias("conf"); dfconf.show(10, false) val newDF1 = dfconf.withColumn("Y",lit("1")) val newDF2 = newDF1.withColumn("Z",lit(",")) val newDF3 = newDF2.withColumn("comb",concat($"hivetablename",lit("."),$"hivetablecolumn",lit("="),$"dbtablename",lit("."),$"dbtablecolumn")) newDF3.show() val newDF4 = newDF3.groupBy("Y").agg(collect_set("z").as("combined"))
– Farhan Soomro
Nov 9 at 14:51
add a comment |
Could you please share what have you tried so far?
– ulubeyn
Nov 9 at 14:40
Many thanks for the response. Im a learner so couldnt go much further. Heres what Ive been able to achieve val dfconf= spark.read.option("header", "true").csv("file:///conf.csv").alias("conf"); dfconf.show(10, false) val newDF1 = dfconf.withColumn("Y",lit("1")) val newDF2 = newDF1.withColumn("Z",lit(",")) val newDF3 = newDF2.withColumn("comb",concat($"hivetablename",lit("."),$"hivetablecolumn",lit("="),$"dbtablename",lit("."),$"dbtablecolumn")) newDF3.show() val newDF4 = newDF3.groupBy("Y").agg(collect_set("z").as("combined"))
– Farhan Soomro
Nov 9 at 14:51
Could you please share what have you tried so far?
– ulubeyn
Nov 9 at 14:40
Could you please share what have you tried so far?
– ulubeyn
Nov 9 at 14:40
Many thanks for the response. Im a learner so couldnt go much further. Heres what Ive been able to achieve val dfconf= spark.read.option("header", "true").csv("file:///conf.csv").alias("conf"); dfconf.show(10, false) val newDF1 = dfconf.withColumn("Y",lit("1")) val newDF2 = newDF1.withColumn("Z",lit(",")) val newDF3 = newDF2.withColumn("comb",concat($"hivetablename",lit("."),$"hivetablecolumn",lit("="),$"dbtablename",lit("."),$"dbtablecolumn")) newDF3.show() val newDF4 = newDF3.groupBy("Y").agg(collect_set("z").as("combined"))
– Farhan Soomro
Nov 9 at 14:51
Many thanks for the response. Im a learner so couldnt go much further. Heres what Ive been able to achieve val dfconf= spark.read.option("header", "true").csv("file:///conf.csv").alias("conf"); dfconf.show(10, false) val newDF1 = dfconf.withColumn("Y",lit("1")) val newDF2 = newDF1.withColumn("Z",lit(",")) val newDF3 = newDF2.withColumn("comb",concat($"hivetablename",lit("."),$"hivetablecolumn",lit("="),$"dbtablename",lit("."),$"dbtablecolumn")) newDF3.show() val newDF4 = newDF3.groupBy("Y").agg(collect_set("z").as("combined"))
– Farhan Soomro
Nov 9 at 14:51
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Farhan Soomro is a new contributor. Be nice, and check out our Code of Conduct.
Farhan Soomro is a new contributor. Be nice, and check out our Code of Conduct.
Farhan Soomro is a new contributor. Be nice, and check out our Code of Conduct.
Farhan Soomro is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53227060%2ftransform-csv-column-values-into-single-row%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
aet17mp9P0Tt0FC
Could you please share what have you tried so far?
– ulubeyn
Nov 9 at 14:40
Many thanks for the response. Im a learner so couldnt go much further. Heres what Ive been able to achieve val dfconf= spark.read.option("header", "true").csv("file:///conf.csv").alias("conf"); dfconf.show(10, false) val newDF1 = dfconf.withColumn("Y",lit("1")) val newDF2 = newDF1.withColumn("Z",lit(",")) val newDF3 = newDF2.withColumn("comb",concat($"hivetablename",lit("."),$"hivetablecolumn",lit("="),$"dbtablename",lit("."),$"dbtablecolumn")) newDF3.show() val newDF4 = newDF3.groupBy("Y").agg(collect_set("z").as("combined"))
– Farhan Soomro
Nov 9 at 14:51