Paramiko Download, process and re-upload the same file
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
add a comment |
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingput
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.
– Martin Prikryl
Nov 11 '18 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54
add a comment |
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
python sftp paramiko
edited Nov 12 '18 at 7:00
Martin Prikryl
85.5k22163355
85.5k22163355
asked Nov 11 '18 at 19:09
canofwhoopass
155
155
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingput
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.
– Martin Prikryl
Nov 11 '18 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54
add a comment |
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingput
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.
– Martin Prikryl
Nov 11 '18 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put
and putfo
in this respect. Actually all that put
does is that it calls open
and putfo
, just like your code does.– Martin Prikryl
Nov 11 '18 at 22:39
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put
and putfo
in this respect. Actually all that put
does is that it calls open
and putfo
, just like your code does.– Martin Prikryl
Nov 11 '18 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54
add a comment |
1 Answer
1
active
oldest
votes
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
add a comment |
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
add a comment |
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 '18 at 19:53
Martin Prikryl
85.5k22163355
85.5k22163355
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
add a comment |
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.– Martin Prikryl
Nov 11 '18 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54