Perfromance issue when writing ORC file output from Hadoop map/reduce task
I output orc file from my hadoop task. My schema contains nested structs and some lists(about four lists). The length of each list is between 0 to 200. My task input is also orc file of simple struct.
The situation is when mapper begin to run, all mapper will be stuck at progress 1.67%, after about 20 minutes, the process starts to move forward.
I tried to figure out the reason.
- I commented out the
context.write, the whole map reduce task finished within 10 minutes, and stuck situation is disappeared. - Output empty list, The task is still finished quickly.
So it seems the big list is the reason of the problem. However, the questions are:
- Whether ORC file with list of length 200 will cause perfromance problem? If it is, any solutions?
- why the progress stucks at begging and starts to run after 20 minutes?
the version of orc is: orc-mapreduce 1.5.2, hadoop-mapreduce-client-core 2.8.0
hadoop orc
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
add a comment |
I output orc file from my hadoop task. My schema contains nested structs and some lists(about four lists). The length of each list is between 0 to 200. My task input is also orc file of simple struct.
The situation is when mapper begin to run, all mapper will be stuck at progress 1.67%, after about 20 minutes, the process starts to move forward.
I tried to figure out the reason.
- I commented out the
context.write, the whole map reduce task finished within 10 minutes, and stuck situation is disappeared. - Output empty list, The task is still finished quickly.
So it seems the big list is the reason of the problem. However, the questions are:
- Whether ORC file with list of length 200 will cause perfromance problem? If it is, any solutions?
- why the progress stucks at begging and starts to run after 20 minutes?
the version of orc is: orc-mapreduce 1.5.2, hadoop-mapreduce-client-core 2.8.0
hadoop orc
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
add a comment |
I output orc file from my hadoop task. My schema contains nested structs and some lists(about four lists). The length of each list is between 0 to 200. My task input is also orc file of simple struct.
The situation is when mapper begin to run, all mapper will be stuck at progress 1.67%, after about 20 minutes, the process starts to move forward.
I tried to figure out the reason.
- I commented out the
context.write, the whole map reduce task finished within 10 minutes, and stuck situation is disappeared. - Output empty list, The task is still finished quickly.
So it seems the big list is the reason of the problem. However, the questions are:
- Whether ORC file with list of length 200 will cause perfromance problem? If it is, any solutions?
- why the progress stucks at begging and starts to run after 20 minutes?
the version of orc is: orc-mapreduce 1.5.2, hadoop-mapreduce-client-core 2.8.0
hadoop orc
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
I output orc file from my hadoop task. My schema contains nested structs and some lists(about four lists). The length of each list is between 0 to 200. My task input is also orc file of simple struct.
The situation is when mapper begin to run, all mapper will be stuck at progress 1.67%, after about 20 minutes, the process starts to move forward.
I tried to figure out the reason.
- I commented out the
context.write, the whole map reduce task finished within 10 minutes, and stuck situation is disappeared. - Output empty list, The task is still finished quickly.
So it seems the big list is the reason of the problem. However, the questions are:
- Whether ORC file with list of length 200 will cause perfromance problem? If it is, any solutions?
- why the progress stucks at begging and starts to run after 20 minutes?
the version of orc is: orc-mapreduce 1.5.2, hadoop-mapreduce-client-core 2.8.0
hadoop orc
hadoop orc
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
asked Nov 14 '18 at 9:46
living zhangliving zhang
77119
77119
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53297177%2fperfromance-issue-when-writing-orc-file-output-from-hadoop-map-reduce-task%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53297177%2fperfromance-issue-when-writing-orc-file-output-from-hadoop-map-reduce-task%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
