Azure Databricks to Event Hub
I am very new to Databricks. So, pardon me please. Here is my requiremnt
- I have data stored in Azure DataLake
- As per the requirement, we can only access data via Azure Databricks notebook
- We have to pull the data from certain tables, join with other tables, aggregate
- Send the data to an Event Hub
How can I perform this activity. I assume there is not one shot process. I was planning to create a notebook and run it via Azure Data Factory. Pump the data in Blob and then using .Net send it to Event Hub. But, from Azure Data Factory we can only run the Azure Databricks notebook not store anywhere
azure azure-data-factory azure-data-lake databricks asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
add a comment |
I am very new to Databricks. So, pardon me please. Here is my requiremnt
- I have data stored in Azure DataLake
- As per the requirement, we can only access data via Azure Databricks notebook
- We have to pull the data from certain tables, join with other tables, aggregate
- Send the data to an Event Hub
How can I perform this activity. I assume there is not one shot process. I was planning to create a notebook and run it via Azure Data Factory. Pump the data in Blob and then using .Net send it to Event Hub. But, from Azure Data Factory we can only run the Azure Databricks notebook not store anywhere
azure azure-data-factory azure-data-lake databricks asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
add a comment |
I am very new to Databricks. So, pardon me please. Here is my requiremnt
- I have data stored in Azure DataLake
- As per the requirement, we can only access data via Azure Databricks notebook
- We have to pull the data from certain tables, join with other tables, aggregate
- Send the data to an Event Hub
How can I perform this activity. I assume there is not one shot process. I was planning to create a notebook and run it via Azure Data Factory. Pump the data in Blob and then using .Net send it to Event Hub. But, from Azure Data Factory we can only run the Azure Databricks notebook not store anywhere
azure azure-data-factory azure-data-lake databricks asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
I am very new to Databricks. So, pardon me please. Here is my requiremnt
- I have data stored in Azure DataLake
- As per the requirement, we can only access data via Azure Databricks notebook
- We have to pull the data from certain tables, join with other tables, aggregate
- Send the data to an Event Hub
How can I perform this activity. I assume there is not one shot process. I was planning to create a notebook and run it via Azure Data Factory. Pump the data in Blob and then using .Net send it to Event Hub. But, from Azure Data Factory we can only run the Azure Databricks notebook not store anywhere
azure azure-data-factory azure-data-lake databricks azure azure-data-factory azure-data-lake databricks asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
asked Nov 14 '18 at 4:36
Hillol SahaHillol Saha
183
183
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Azure Databricks do support Azure Event Hubs as source and sink. Understand Structured Streaming - it is a stream processing engine in Apache Spark (available in Azure Databricks as well)
Create a notebook to do all your transformation (join, aggregation...) - assuming you are doing batch write to azure event hub.
PySpark code:
val connectionString = "Valid EventHubs connection string."
val ehWriteConf = EventHubsConf(connectionString)
df.select("body")
.write
.format("eventhubs")
.options(ehWriteConf.toMap)
.save()
Replace .write to .writeStream if your queries are streaming.
More things to consider when working with Azure Event Hubs is regarding partitions - it is optional, you can just send the body alone (which will do round robin model)
More information here
answered Nov 14 '18 at 13:50
databashdatabash
372313
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53293264%2fazure-databricks-to-event-hub%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Azure Databricks do support Azure Event Hubs as source and sink. Understand Structured Streaming - it is a stream processing engine in Apache Spark (available in Azure Databricks as well)
Create a notebook to do all your transformation (join, aggregation...) - assuming you are doing batch write to azure event hub.
PySpark code:
val connectionString = "Valid EventHubs connection string."
val ehWriteConf = EventHubsConf(connectionString)
df.select("body")
.write
.format("eventhubs")
.options(ehWriteConf.toMap)
.save()
Replace .write to .writeStream if your queries are streaming.
More things to consider when working with Azure Event Hubs is regarding partitions - it is optional, you can just send the body alone (which will do round robin model)
More information here
answered Nov 14 '18 at 13:50
databashdatabash
372313
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
add a comment |
Azure Databricks do support Azure Event Hubs as source and sink. Understand Structured Streaming - it is a stream processing engine in Apache Spark (available in Azure Databricks as well)
Create a notebook to do all your transformation (join, aggregation...) - assuming you are doing batch write to azure event hub.
PySpark code:
val connectionString = "Valid EventHubs connection string."
val ehWriteConf = EventHubsConf(connectionString)
df.select("body")
.write
.format("eventhubs")
.options(ehWriteConf.toMap)
.save()
Replace .write to .writeStream if your queries are streaming.
More things to consider when working with Azure Event Hubs is regarding partitions - it is optional, you can just send the body alone (which will do round robin model)
More information here
answered Nov 14 '18 at 13:50
databashdatabash
372313
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
add a comment |
Azure Databricks do support Azure Event Hubs as source and sink. Understand Structured Streaming - it is a stream processing engine in Apache Spark (available in Azure Databricks as well)
Create a notebook to do all your transformation (join, aggregation...) - assuming you are doing batch write to azure event hub.
PySpark code:
val connectionString = "Valid EventHubs connection string."
val ehWriteConf = EventHubsConf(connectionString)
df.select("body")
.write
.format("eventhubs")
.options(ehWriteConf.toMap)
.save()
Replace .write to .writeStream if your queries are streaming.
More things to consider when working with Azure Event Hubs is regarding partitions - it is optional, you can just send the body alone (which will do round robin model)
More information here
answered Nov 14 '18 at 13:50
databashdatabash
372313
Azure Databricks do support Azure Event Hubs as source and sink. Understand Structured Streaming - it is a stream processing engine in Apache Spark (available in Azure Databricks as well)
Create a notebook to do all your transformation (join, aggregation...) - assuming you are doing batch write to azure event hub.
PySpark code:
val connectionString = "Valid EventHubs connection string."
val ehWriteConf = EventHubsConf(connectionString)
df.select("body")
.write
.format("eventhubs")
.options(ehWriteConf.toMap)
.save()
Replace .write to .writeStream if your queries are streaming.
More things to consider when working with Azure Event Hubs is regarding partitions - it is optional, you can just send the body alone (which will do round robin model)
More information here
answered Nov 14 '18 at 13:50
databashdatabash
372313
answered Nov 14 '18 at 13:50
databashdatabash
372313
answered Nov 14 '18 at 13:50
databashdatabash
372313
answered Nov 14 '18 at 13:50
databashdatabash
372313
372313
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
add a comment |
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
thanks for your suggestion. We did go through the docs and yes, it seems possible. We have two further questions. 1. there is size limit of 256 KB in Event Hub to handle flies, how can we break the file 2. We have to add custom properties to the file. Can we do that here
– Hillol Saha
Nov 28 '18 at 7:36
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53293264%2fazure-databricks-to-event-hub%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
