How to import a fixed number of rows in incremental import in sqoop?
I am learning sqoop on my own and tried to locally test the incremental import functionality by using the below mentioned code
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 1000'
-m 1
--incremental append
--check-column emp_no
--target-dir sqoop/import_data/employee_db_import_incremental
The above piece of script imported the first 1000 records and now I want to append the next 1000 records to the existing data. How to do it ?
mysql hadoop mapreduce sqoop
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
add a comment |
I am learning sqoop on my own and tried to locally test the incremental import functionality by using the below mentioned code
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 1000'
-m 1
--incremental append
--check-column emp_no
--target-dir sqoop/import_data/employee_db_import_incremental
The above piece of script imported the first 1000 records and now I want to append the next 1000 records to the existing data. How to do it ?
mysql hadoop mapreduce sqoop
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
add a comment |
I am learning sqoop on my own and tried to locally test the incremental import functionality by using the below mentioned code
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 1000'
-m 1
--incremental append
--check-column emp_no
--target-dir sqoop/import_data/employee_db_import_incremental
The above piece of script imported the first 1000 records and now I want to append the next 1000 records to the existing data. How to do it ?
mysql hadoop mapreduce sqoop
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
I am learning sqoop on my own and tried to locally test the incremental import functionality by using the below mentioned code
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 1000'
-m 1
--incremental append
--check-column emp_no
--target-dir sqoop/import_data/employee_db_import_incremental
The above piece of script imported the first 1000 records and now I want to append the next 1000 records to the existing data. How to do it ?
mysql hadoop mapreduce sqoop
mysql hadoop mapreduce sqoop
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
asked Nov 13 '18 at 2:41
Sarvagya DubeySarvagya Dubey
69110
69110
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You must specify the last value, from which record you need to append, suppose your last record is 1000 then your command must be like following:
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 2000'
-m 1
--incremental append
--check-column emp_no
--last-value 1000
--target-dir sqoop/import_data/employee_db_import_incremental
edited Nov 13 '18 at 10:46
rdj7
8011818
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273007%2fhow-to-import-a-fixed-number-of-rows-in-incremental-import-in-sqoop%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You must specify the last value, from which record you need to append, suppose your last record is 1000 then your command must be like following:
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 2000'
-m 1
--incremental append
--check-column emp_no
--last-value 1000
--target-dir sqoop/import_data/employee_db_import_incremental
edited Nov 13 '18 at 10:46
rdj7
8011818
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
add a comment |
You must specify the last value, from which record you need to append, suppose your last record is 1000 then your command must be like following:
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 2000'
-m 1
--incremental append
--check-column emp_no
--last-value 1000
--target-dir sqoop/import_data/employee_db_import_incremental
edited Nov 13 '18 at 10:46
rdj7
8011818
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
add a comment |
You must specify the last value, from which record you need to append, suppose your last record is 1000 then your command must be like following:
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 2000'
-m 1
--incremental append
--check-column emp_no
--last-value 1000
--target-dir sqoop/import_data/employee_db_import_incremental
edited Nov 13 '18 at 10:46
rdj7
8011818
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
You must specify the last value, from which record you need to append, suppose your last record is 1000 then your command must be like following:
sqoop import
--connect jdbc:mysql://localhost/employees
--username root
-P
--query 'select * from employees WHERE $CONDITIONS ORDER BY emp_no LIMIT 2000'
-m 1
--incremental append
--check-column emp_no
--last-value 1000
--target-dir sqoop/import_data/employee_db_import_incremental
edited Nov 13 '18 at 10:46
rdj7
8011818
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
edited Nov 13 '18 at 10:46
rdj7
8011818
edited Nov 13 '18 at 10:46
rdj7
8011818
edited Nov 13 '18 at 10:46
rdj7
8011818
8011818
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
answered Nov 13 '18 at 10:17
Harika KrishnaHarika Krishna
262
262
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273007%2fhow-to-import-a-fixed-number-of-rows-in-incremental-import-in-sqoop%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
