Timeout error: Error with 400 StatusCode: “requirement failed: Session isn't active.”
I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Number of rows of dataframe is 99,452,829. When the script reaches 1 hour, an error is encountered -
Error with 400 StatusCode: "requirement failed: Session isn't
active.
My default interpreter for the notebook is jdbc. I have read about timeoutlifecyclemanager and added in the interpreter setting zeppelin.interpreter.lifecyclemanager.timeout.threshold and set it to 7200000 but still encountered the error after it reaches 1 hour runtime at 33% processing completion.
I checked the Blob folder after the 1 hr timeout and parquet files were successfully written to Blob which are indeed partitioned per country.
The script I am running to write DF to parquet Blob is below:
trdpn_cntry_fct_denom_df.write.format("parquet").partitionBy("CNTRY_ID").mode("overwrite").save("wasbs://tradepanelpoc@blobasbackupx2066561.blob.core.windows.net/cbls/hdi/trdpn_cntry_fct_denom_df.parquet")
Is this Zeppelin timeout issue? How can it be extended for more than 1 hour runtime? Thanks for the help.
python apache-spark apache-zeppelin livy
edited Nov 15 '18 at 5:49
Max Belousov
18918
asked Nov 13 '18 at 7:14
user10644448user10644448
12
add a comment |
I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Number of rows of dataframe is 99,452,829. When the script reaches 1 hour, an error is encountered -
Error with 400 StatusCode: "requirement failed: Session isn't
active.
My default interpreter for the notebook is jdbc. I have read about timeoutlifecyclemanager and added in the interpreter setting zeppelin.interpreter.lifecyclemanager.timeout.threshold and set it to 7200000 but still encountered the error after it reaches 1 hour runtime at 33% processing completion.
I checked the Blob folder after the 1 hr timeout and parquet files were successfully written to Blob which are indeed partitioned per country.
The script I am running to write DF to parquet Blob is below:
trdpn_cntry_fct_denom_df.write.format("parquet").partitionBy("CNTRY_ID").mode("overwrite").save("wasbs://tradepanelpoc@blobasbackupx2066561.blob.core.windows.net/cbls/hdi/trdpn_cntry_fct_denom_df.parquet")
Is this Zeppelin timeout issue? How can it be extended for more than 1 hour runtime? Thanks for the help.
python apache-spark apache-zeppelin livy
edited Nov 15 '18 at 5:49
Max Belousov
18918
asked Nov 13 '18 at 7:14
user10644448user10644448
12
add a comment |
I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Number of rows of dataframe is 99,452,829. When the script reaches 1 hour, an error is encountered -
Error with 400 StatusCode: "requirement failed: Session isn't
active.
My default interpreter for the notebook is jdbc. I have read about timeoutlifecyclemanager and added in the interpreter setting zeppelin.interpreter.lifecyclemanager.timeout.threshold and set it to 7200000 but still encountered the error after it reaches 1 hour runtime at 33% processing completion.
I checked the Blob folder after the 1 hr timeout and parquet files were successfully written to Blob which are indeed partitioned per country.
The script I am running to write DF to parquet Blob is below:
trdpn_cntry_fct_denom_df.write.format("parquet").partitionBy("CNTRY_ID").mode("overwrite").save("wasbs://tradepanelpoc@blobasbackupx2066561.blob.core.windows.net/cbls/hdi/trdpn_cntry_fct_denom_df.parquet")
Is this Zeppelin timeout issue? How can it be extended for more than 1 hour runtime? Thanks for the help.
python apache-spark apache-zeppelin livy
edited Nov 15 '18 at 5:49
Max Belousov
18918
asked Nov 13 '18 at 7:14
user10644448user10644448
12
I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Number of rows of dataframe is 99,452,829. When the script reaches 1 hour, an error is encountered -
Error with 400 StatusCode: "requirement failed: Session isn't
active.
My default interpreter for the notebook is jdbc. I have read about timeoutlifecyclemanager and added in the interpreter setting zeppelin.interpreter.lifecyclemanager.timeout.threshold and set it to 7200000 but still encountered the error after it reaches 1 hour runtime at 33% processing completion.
I checked the Blob folder after the 1 hr timeout and parquet files were successfully written to Blob which are indeed partitioned per country.
The script I am running to write DF to parquet Blob is below:
trdpn_cntry_fct_denom_df.write.format("parquet").partitionBy("CNTRY_ID").mode("overwrite").save("wasbs://tradepanelpoc@blobasbackupx2066561.blob.core.windows.net/cbls/hdi/trdpn_cntry_fct_denom_df.parquet")
Is this Zeppelin timeout issue? How can it be extended for more than 1 hour runtime? Thanks for the help.
python apache-spark apache-zeppelin livy
python apache-spark apache-zeppelin livy
edited Nov 15 '18 at 5:49
Max Belousov
18918
asked Nov 13 '18 at 7:14
user10644448user10644448
12
edited Nov 15 '18 at 5:49
Max Belousov
18918
asked Nov 13 '18 at 7:14
user10644448user10644448
12
edited Nov 15 '18 at 5:49
Max Belousov
18918
edited Nov 15 '18 at 5:49
Max Belousov
18918
edited Nov 15 '18 at 5:49
Max Belousov
18918
18918
asked Nov 13 '18 at 7:14
user10644448user10644448
12
asked Nov 13 '18 at 7:14
user10644448user10644448
12
asked Nov 13 '18 at 7:14
user10644448user10644448
12
12
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The timeout lifecycle manager is available since version 0.8.
Seems there is problem with pyspark. Try this solution
Pyspark socket timeout exception after application running for a while
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53275693%2ftimeout-error-error-with-400-statuscode-requirement-failed-session-isnt-act%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The timeout lifecycle manager is available since version 0.8.
Seems there is problem with pyspark. Try this solution
Pyspark socket timeout exception after application running for a while
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
add a comment |
The timeout lifecycle manager is available since version 0.8.
Seems there is problem with pyspark. Try this solution
Pyspark socket timeout exception after application running for a while
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
add a comment |
The timeout lifecycle manager is available since version 0.8.
Seems there is problem with pyspark. Try this solution
Pyspark socket timeout exception after application running for a while
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
The timeout lifecycle manager is available since version 0.8.
Seems there is problem with pyspark. Try this solution
Pyspark socket timeout exception after application running for a while
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
answered Nov 14 '18 at 18:48
Max BelousovMax Belousov
18918
18918
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53275693%2ftimeout-error-error-with-400-statuscode-requirement-failed-session-isnt-act%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
