Dispatching chunks of work to backgroundworkers
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
add a comment |
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 '18 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 '18 at 2:19
add a comment |
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
Using C#.
I have 100,000+ pieces of test data that need to have some calculations run with. My actual data set will be in the millions of pieces of data. The test data currently runs sequentially and takes about a minute to process. I want to split this work up and have backgroundworkers process back to back so I will hopefully get the processing done quicker.
What I have in mind is to do a foreach loop with the data and start a backgroundworker with each piece of data. I know I need to limit the number of bw's to three as I have 4 cores on this machine. I have done some testing with simple bw's but not three at the same time.
I have no idea how to go about this. How would one execute three background workers to process this data?
c# backgroundworker
c# backgroundworker
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
asked Nov 15 '18 at 1:57
Eric SnyderEric Snyder
316313
316313
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 '18 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 '18 at 2:19
add a comment |
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 '18 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 '18 at 2:19
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 '18 at 2:02
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 '18 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 '18 at 2:19
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 '18 at 2:19
add a comment |
1 Answer
1
active
oldest
votes
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53311346%2fdispatching-chunks-of-work-to-backgroundworkers%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
add a comment |
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
add a comment |
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
The BackgroundWorker is designed for early learning work mostly. Maybe the odd alternative threading scenario. What you are doing sounds like a very advanced opeartion. You can still use BGW, but raw Threads, Tasks, Threadpools and the like would be better at this point.
There is also the general question if this operation can even be accelerated with Multithreading. I like to say "multithreading has to pick it's problems carefully". Pick it in the wrong scenario and you end with a programm that needs more memory, is more prone to errors and slower then a single BGW or sequential programm.
Your case could be one of the rare cases of a pleasingly paralell operation. Or it could be mostly memory bound. Wich means you run into Paralell slowdown almost instantly. Resist atempts at hardcoding the number of threads. Usually you can leave that load-balancing work to a ThreadPool. To get a better answer you need to get a lot more specific.
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
answered Nov 15 '18 at 3:20
ChristopherChristopher
2,9962824
2,9962824
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53311346%2fdispatching-chunks-of-work-to-backgroundworkers%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
could anyone explain how bw related to number of cores ?
– Z.R.T.
Nov 15 '18 at 2:02
It would be awesome if you could provide a Minimal, Complete, and Verifiable example of your progress so far.
– mjwills
Nov 15 '18 at 2:19