Schema registry on AWS
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:
- schema upgrade - validate compatibility with the previous version
- version avro schemas in a similar way as schema registry does
What are the options to handle 2 above? The only thing that I found was glue catalogue, but it doesn't seem to
At the end I also want to use firehose (output to redshift), but from what I understand this is not possible and will require writing custom lambda.
avro amazon-kinesis confluent-schema-registry
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
asked Nov 15 '18 at 9:50
czajekczajek
295319
add a comment |
I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:
- schema upgrade - validate compatibility with the previous version
- version avro schemas in a similar way as schema registry does
What are the options to handle 2 above? The only thing that I found was glue catalogue, but it doesn't seem to
At the end I also want to use firehose (output to redshift), but from what I understand this is not possible and will require writing custom lambda.
avro amazon-kinesis confluent-schema-registry
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
asked Nov 15 '18 at 9:50
czajekczajek
295319
Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry
– cricket_007
Nov 15 '18 at 23:01
add a comment |
I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:
- schema upgrade - validate compatibility with the previous version
- version avro schemas in a similar way as schema registry does
What are the options to handle 2 above? The only thing that I found was glue catalogue, but it doesn't seem to
At the end I also want to use firehose (output to redshift), but from what I understand this is not possible and will require writing custom lambda.
avro amazon-kinesis confluent-schema-registry
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
asked Nov 15 '18 at 9:50
czajekczajek
295319
I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:
- schema upgrade - validate compatibility with the previous version
- version avro schemas in a similar way as schema registry does
What are the options to handle 2 above? The only thing that I found was glue catalogue, but it doesn't seem to
At the end I also want to use firehose (output to redshift), but from what I understand this is not possible and will require writing custom lambda.
avro amazon-kinesis confluent-schema-registry
avro amazon-kinesis confluent-schema-registry
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
asked Nov 15 '18 at 9:50
czajekczajek
295319
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
asked Nov 15 '18 at 9:50
czajekczajek
295319
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
edited Nov 15 '18 at 23:02
cricket_007
84.4k1147119
84.4k1147119
asked Nov 15 '18 at 9:50
czajekczajek
295319
asked Nov 15 '18 at 9:50
czajekczajek
295319
asked Nov 15 '18 at 9:50
czajekczajek
295319
295319
Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry
– cricket_007
Nov 15 '18 at 23:01
add a comment |
Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry
– cricket_007
Nov 15 '18 at 23:01
Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry
– cricket_007
Nov 15 '18 at 23:01
Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry
– cricket_007
Nov 15 '18 at 23:01
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53316618%2fschema-registry-on-aws%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53316618%2fschema-registry-on-aws%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry
– cricket_007
Nov 15 '18 at 23:01