How do I use MapElements and KV in together in Apache Beam?
up vote
0
down vote
favorite
I wanted to do something like:
PCollection<String> a = whatever;
PCollection<KV<String, User>> b = a.apply(
MapElements.into(TypeDescriptor.of(KV<String, User>.class))
.via(s -> KV.of(s, new User(s))));
Where User is a custom datatype with Arvo coder and a constructor that takes a string into account.
However, I get the following error:
Cannot select from parameterized type
I tried to change it to TypeDescriptor.of(KV.class) instead, but then I get:
Incompatible types; Required PCollection> but 'apply' was inferred to OutputT: no instance(s) of type variable(s) exists so that PCollection conforms to PCollection>
So how am I suppose to use KV with MapElements?
I know that what I want to do is doable using ParDo where I could explicitly specify how to do Type Erasure by declearing new DoFn<String, KV<String, User>> but ParDo does not support lambda function. As we are using Java 8, this seems less elegant....
java java-8 apache-beam
asked Nov 10 at 2:35
Tamaki Sakura
237215
add a comment |
up vote
0
down vote
favorite
I wanted to do something like:
PCollection<String> a = whatever;
PCollection<KV<String, User>> b = a.apply(
MapElements.into(TypeDescriptor.of(KV<String, User>.class))
.via(s -> KV.of(s, new User(s))));
Where User is a custom datatype with Arvo coder and a constructor that takes a string into account.
However, I get the following error:
Cannot select from parameterized type
I tried to change it to TypeDescriptor.of(KV.class) instead, but then I get:
Incompatible types; Required PCollection> but 'apply' was inferred to OutputT: no instance(s) of type variable(s) exists so that PCollection conforms to PCollection>
So how am I suppose to use KV with MapElements?
I know that what I want to do is doable using ParDo where I could explicitly specify how to do Type Erasure by declearing new DoFn<String, KV<String, User>> but ParDo does not support lambda function. As we are using Java 8, this seems less elegant....
java java-8 apache-beam
asked Nov 10 at 2:35
Tamaki Sakura
237215
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I wanted to do something like:
PCollection<String> a = whatever;
PCollection<KV<String, User>> b = a.apply(
MapElements.into(TypeDescriptor.of(KV<String, User>.class))
.via(s -> KV.of(s, new User(s))));
Where User is a custom datatype with Arvo coder and a constructor that takes a string into account.
However, I get the following error:
Cannot select from parameterized type
I tried to change it to TypeDescriptor.of(KV.class) instead, but then I get:
Incompatible types; Required PCollection> but 'apply' was inferred to OutputT: no instance(s) of type variable(s) exists so that PCollection conforms to PCollection>
So how am I suppose to use KV with MapElements?
I know that what I want to do is doable using ParDo where I could explicitly specify how to do Type Erasure by declearing new DoFn<String, KV<String, User>> but ParDo does not support lambda function. As we are using Java 8, this seems less elegant....
java java-8 apache-beam
asked Nov 10 at 2:35
Tamaki Sakura
237215
I wanted to do something like:
PCollection<String> a = whatever;
PCollection<KV<String, User>> b = a.apply(
MapElements.into(TypeDescriptor.of(KV<String, User>.class))
.via(s -> KV.of(s, new User(s))));
Where User is a custom datatype with Arvo coder and a constructor that takes a string into account.
However, I get the following error:
Cannot select from parameterized type
I tried to change it to TypeDescriptor.of(KV.class) instead, but then I get:
Incompatible types; Required PCollection> but 'apply' was inferred to OutputT: no instance(s) of type variable(s) exists so that PCollection conforms to PCollection>
So how am I suppose to use KV with MapElements?
I know that what I want to do is doable using ParDo where I could explicitly specify how to do Type Erasure by declearing new DoFn<String, KV<String, User>> but ParDo does not support lambda function. As we are using Java 8, this seems less elegant....
java java-8 apache-beam
java java-8 apache-beam
asked Nov 10 at 2:35
Tamaki Sakura
237215
asked Nov 10 at 2:35
Tamaki Sakura
237215
asked Nov 10 at 2:35
Tamaki Sakura
237215
asked Nov 10 at 2:35
Tamaki Sakura
237215
asked Nov 10 at 2:35
Tamaki Sakura
237215
237215
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53235553%2fhow-do-i-use-mapelements-and-kv-in-together-in-apache-beam%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
