Paramiko Download, process and re-upload the same file










1














I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.










share|improve this question























  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 '18 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 '18 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 '18 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 '18 at 22:39











  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 '18 at 2:54
















1














I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.










share|improve this question























  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 '18 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 '18 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 '18 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 '18 at 22:39











  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 '18 at 2:54














1












1








1


1





I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.










share|improve this question















I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.







python sftp paramiko






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 12 '18 at 7:00









Martin Prikryl

85.5k22163355




85.5k22163355










asked Nov 11 '18 at 19:09









canofwhoopass

155




155











  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 '18 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 '18 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 '18 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 '18 at 22:39











  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 '18 at 2:54

















  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 '18 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 '18 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 '18 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 '18 at 22:39











  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 '18 at 2:54
















"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00




"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 '18 at 20:00












Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35




Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 '18 at 20:35












Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43




Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 '18 at 20:43












Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
– Martin Prikryl
Nov 11 '18 at 22:39





Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
– Martin Prikryl
Nov 11 '18 at 22:39













Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54





Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 '18 at 2:54













1 Answer
1






active

oldest

votes


















0














If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.




If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer




















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 '18 at 2:50










Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.




If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer




















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 '18 at 2:50















0














If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.




If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer




















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 '18 at 2:50













0












0








0






If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.




If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer












If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.




If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 11 '18 at 19:53









Martin Prikryl

85.5k22163355




85.5k22163355











  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 '18 at 2:50
















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 '18 at 2:50















f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50




f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 '18 at 2:50

















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Use pre created SQLite database for Android project in kotlin

Darth Vader #20

Ondo