How to use DistributedDataParallel() and init_process_group() in a HPC?









up vote
1
down vote

favorite












I will use HPC for my research and I don't know a lot about parallel or distributed computing.
I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
What is the meaning of initializing processes group? and what is




init_method : URL specifying how to initialize the package.




for example (I found those in the documentation):
'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

What are those URLs?



What is the Rank of the current process?

Is world_size the number of GPUs?



It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



I will use things like Slurm(sbatch) in the HPC.






pytorch







share|improve this question



























    up vote
    1
    down vote

    favorite












    I will use HPC for my research and I don't know a lot about parallel or distributed computing.
    I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
    What is the meaning of initializing processes group? and what is




    init_method : URL specifying how to initialize the package.




    for example (I found those in the documentation):
    'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

    What are those URLs?



    What is the Rank of the current process?

    Is world_size the number of GPUs?



    It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



    I will use things like Slurm(sbatch) in the HPC.






    pytorch







    share|improve this question

























      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I will use HPC for my research and I don't know a lot about parallel or distributed computing.
      I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
      What is the meaning of initializing processes group? and what is




      init_method : URL specifying how to initialize the package.




      for example (I found those in the documentation):
      'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

      What are those URLs?



      What is the Rank of the current process?

      Is world_size the number of GPUs?



      It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



      I will use things like Slurm(sbatch) in the HPC.






      pytorch







      share|improve this question















      I will use HPC for my research and I don't know a lot about parallel or distributed computing.
      I really don't understand the DistributedDataParallel() in pytorch. Especially init_process_group().
      What is the meaning of initializing processes group? and what is




      init_method : URL specifying how to initialize the package.




      for example (I found those in the documentation):
      'tcp://10.1.1.20:23456' or 'file:///mnt/nfs/sharedfile'

      What are those URLs?



      What is the Rank of the current process?

      Is world_size the number of GPUs?



      It would be really appreciated if someone explained to me what is and How to use DistributedDataParallel() and init_process_group() because I don't know parallel or distributed computing.



      I will use things like Slurm(sbatch) in the HPC.






      pytorch




      pytorch






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 9 at 20:44

























      asked Nov 9 at 19:29









      floyd

      17910




      17910



























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













           

          draft saved


          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53232171%2fhow-to-use-distributeddataparallel-and-init-process-group-in-a-hpc%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes















           

          draft saved


          draft discarded















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53232171%2fhow-to-use-distributeddataparallel-and-init-process-group-in-a-hpc%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Use pre created SQLite database for Android project in kotlin

          Image
          0 I am trying to use an already existing sqLite database in my android app. I have verified that it is loaded into the project in the assets folder and I can browse it in DB browser when I open it directly from the project. This is the schema for the table I am trying to query CREATE TABLE "ACTIVITY_LIST_TBL" ( `_ID` INTEGER NOT NULL, `ACT_NAME` VARCHAR NOT NULL, `ACT_CODE` VARCHAR, `IS_ACTIVE` INTEGER DEFAULT -1, PRIMARY KEY(`_ID`) ) Below is the activity that creates the sqlite helper and then on a button click I execute a query on the ACTIVITY_LIST_TBL. When the query is executed this exception is thrown: android.database.sqlite.SQLiteException: no such table: ACTIVITY_LIST_TBL (code 1 SQLITE_ERROR): , while compiling: SELECT * FROM ACTIVITY_LIST_TBL class CalendarActivity : AppCompatActivity() { private var cursor:Cursor? = null override fun onCreate(savedInstanceState:Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.calendar_activity...
          Read more

          Darth Vader #20

          Image
          Wiki Characters Creators Teams Volumes Issues Publishers Locations Concepts Things Story Arcs Movies Series Episodes New Comics Forums Gen. Discussion Bug Reporting Delete/Combine Pages Artist Show-Off Off-Topic Contests Battles Fan-Fic RPG Comic Book Preview API Developers Editing & Tools Podcast Quests Community Top Users Activity Feed User Lists Community Promos Archives News Reviews Videos Podcasts Previews Login / Sign Up All Wiki     Arcs     Characters     Companies     Concepts     Issues     Locations     Movies     People     Teams     Things     Volumes     Series     Episodes Editorial     Videos     Articles     Reviews     Features ...
          Read more

          Ondo

          Image
          Der Titel dieses Artikels ist mehrdeutig. Zu weiteren Bedeutungen siehe Ondo (Begriffsklärung). Ondo Basisdaten Hauptstadt: Akure gegründet: 3. Februar 1976 Gouverneur: Rotimi Akeredolu ISO 3166-2: NG-ON Fläche Fläche: 15.500 km² Rang in Nigeria: 25 Bevölkerung Einwohner: 4.671.700 (2016) Bevölkerungsdichte: 320 Einw./km² (2016) Rang in Nigeria: 18 Lage von Ondo in Nigeria Ondo ist ein Bundesstaat des westafrikanischen Landes Nigeria mit der Hauptstadt Akure, die mit 420.623 Einwohnern (2005) auch die größte Stadt des Bundesstaates ist. Inhaltsverzeichnis 1 Geografie 2 Geschichte 2.1 Liste der Gouverneure und Administratoren 3 Verwaltung 4 Wirtschaft Geografie | Der Bundesstaat liegt im Südwesten des Landes und grenzt im Norden an den Bundesstaat Ekiti, im Nordwesten an den Bundesstaat Osun, im Nordosten an den Bundesstaat Kogi, im Süden an den Atlantik, im Südosten an den Bundesstaat Delta, im Westen an den Bundess...
          Read more