How to use DistributedDataParallel() and init_process_group() in a HPC?









up vote
1
down vote

favorite












I will use HPC for my research and I don't know a lot about parallel or distributed computing.
I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
What is the meaning of initializing processes group? and what is




init_method : URL specifying how to initialize the package.




for example (I found those in the documentation):
'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

What are those URLs?



What is the Rank of the current process?

Is world_size the number of GPUs?



It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



I will use things like Slurm(sbatch) in the HPC.










share|improve this question



























    up vote
    1
    down vote

    favorite












    I will use HPC for my research and I don't know a lot about parallel or distributed computing.
    I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
    What is the meaning of initializing processes group? and what is




    init_method : URL specifying how to initialize the package.




    for example (I found those in the documentation):
    'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

    What are those URLs?



    What is the Rank of the current process?

    Is world_size the number of GPUs?



    It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



    I will use things like Slurm(sbatch) in the HPC.










    share|improve this question

























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I will use HPC for my research and I don't know a lot about parallel or distributed computing.
      I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
      What is the meaning of initializing processes group? and what is




      init_method : URL specifying how to initialize the package.




      for example (I found those in the documentation):
      'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

      What are those URLs?



      What is the Rank of the current process?

      Is world_size the number of GPUs?



      It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



      I will use things like Slurm(sbatch) in the HPC.










      share|improve this question















      I will use HPC for my research and I don't know a lot about parallel or distributed computing.
      I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
      What is the meaning of initializing processes group? and what is




      init_method : URL specifying how to initialize the package.




      for example (I found those in the documentation):
      'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

      What are those URLs?



      What is the Rank of the current process?

      Is world_size the number of GPUs?



      It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



      I will use things like Slurm(sbatch) in the HPC.







      pytorch






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 9 at 20:44

























      asked Nov 9 at 19:29









      floyd

      17910




      17910



























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













           

          draft saved


          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53232171%2fhow-to-use-distributeddataparallel-and-init-process-group-in-a-hpc%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes















           

          draft saved


          draft discarded















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53232171%2fhow-to-use-distributeddataparallel-and-init-process-group-in-a-hpc%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Use pre created SQLite database for Android project in kotlin

          Darth Vader #20

          Ondo