Limit the number of background workers per user in Rails

I have a Rails application where each user has a specific number of background workers.

Since users pay more to increase the number of workers available, I want to be able to add these workers dynamically.

I would like to use ActiveJob in combination with Sidekiq and I thought about the following solution:

when the user registers, I create a new queue in sidekiq with the id of the user.

I add a number of workers, dedicated to that specific queue, depending on how much the user is paying.

I have problems in implementing this solution with Sidekiq and I could not find documentation on how to add queues and workers dynamically.

asked Nov 11 '18 at 21:58

coorasse

3,18611931

Yeah, this juggling of queues/workers is not very easy or scalable.
– Sergio Tulentsev
Nov 11 '18 at 22:50

add a comment |

I have a Rails application where each user has a specific number of background workers.

Since users pay more to increase the number of workers available, I want to be able to add these workers dynamically.

I would like to use ActiveJob in combination with Sidekiq and I thought about the following solution:

when the user registers, I create a new queue in sidekiq with the id of the user.

I add a number of workers, dedicated to that specific queue, depending on how much the user is paying.

I have problems in implementing this solution with Sidekiq and I could not find documentation on how to add queues and workers dynamically.

asked Nov 11 '18 at 21:58

coorasse

3,18611931

Yeah, this juggling of queues/workers is not very easy or scalable.
– Sergio Tulentsev
Nov 11 '18 at 22:50

add a comment |

I have a Rails application where each user has a specific number of background workers.

Since users pay more to increase the number of workers available, I want to be able to add these workers dynamically.

I would like to use ActiveJob in combination with Sidekiq and I thought about the following solution:

when the user registers, I create a new queue in sidekiq with the id of the user.

I add a number of workers, dedicated to that specific queue, depending on how much the user is paying.

I have problems in implementing this solution with Sidekiq and I could not find documentation on how to add queues and workers dynamically.

asked Nov 11 '18 at 21:58

coorasse

3,18611931

I have a Rails application where each user has a specific number of background workers.

Since users pay more to increase the number of workers available, I want to be able to add these workers dynamically.

I would like to use ActiveJob in combination with Sidekiq and I thought about the following solution:

when the user registers, I create a new queue in sidekiq with the id of the user.

I add a number of workers, dedicated to that specific queue, depending on how much the user is paying.

I have problems in implementing this solution with Sidekiq and I could not find documentation on how to add queues and workers dynamically.

ruby-on-rails sidekiq rails-activejob

asked Nov 11 '18 at 21:58

coorasse

3,18611931

asked Nov 11 '18 at 21:58

coorasse

3,18611931

asked Nov 11 '18 at 21:58

coorasse

3,18611931

asked Nov 11 '18 at 21:58

coorasse

3,18611931

asked Nov 11 '18 at 21:58

coorasse

3,18611931

Yeah, this juggling of queues/workers is not very easy or scalable.
– Sergio Tulentsev
Nov 11 '18 at 22:50

add a comment |

Yeah, this juggling of queues/workers is not very easy or scalable.
– Sergio Tulentsev
Nov 11 '18 at 22:50

Yeah, this juggling of queues/workers is not very easy or scalable.
– Sergio Tulentsev
Nov 11 '18 at 22:50

add a comment |

1 Answer
1

active

oldest

votes

If I were to do this, here's what I'd try first:

Wrap all limited jobs in a counter.

on start/dequeue job checks if this user has capacity to run it.

If yes, job runs. If not, it reschedules itself.

Something along these lines:

class MyWorker
 def perform(user_id, *args)
 user = User.find(user_id)
 unless user.has_available_workers
 # re-enqueue with the same args. Possibly, with a delay.
 return 
 end

 user.checkout_worker
 # do work
 ensure
 user.release_worker
 end
end

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

I like your solution, but in this case all users would share the workers and not have dedicated ones. Which is not necessarly a bad idea: it could even scale much better, but it could be that there are no more workers available and the user has to wait (worst case, because he paid)
– coorasse
Nov 12 '18 at 8:20

@coorasse: indeed. In this case, you might want to overprovision the workers, so that there's always some spare capacity. Which is a good idea anyway, if you offer paid service.
– Sergio Tulentsev
Nov 12 '18 at 8:34

Agree. I still miss how I can add workers dynamically without restarting the application. This is not a killer feature, I can always restart the server, but it would be interesting to see if is possible with Sidekiq
– coorasse
Nov 12 '18 at 9:06

@coorasse: add workers? You can just start a new sidekiq process. No need to restart/touch the active ones. Interesting devops task.
– Sergio Tulentsev
Nov 12 '18 at 9:38

add a comment |

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53253662%2flimit-the-number-of-background-workers-per-user-in-rails%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

If I were to do this, here's what I'd try first:

Wrap all limited jobs in a counter.

on start/dequeue job checks if this user has capacity to run it.

If yes, job runs. If not, it reschedules itself.

Something along these lines:

class MyWorker
 def perform(user_id, *args)
 user = User.find(user_id)
 unless user.has_available_workers
 # re-enqueue with the same args. Possibly, with a delay.
 return 
 end

 user.checkout_worker
 # do work
 ensure
 user.release_worker
 end
end

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

I like your solution, but in this case all users would share the workers and not have dedicated ones. Which is not necessarly a bad idea: it could even scale much better, but it could be that there are no more workers available and the user has to wait (worst case, because he paid)
– coorasse
Nov 12 '18 at 8:20

@coorasse: indeed. In this case, you might want to overprovision the workers, so that there's always some spare capacity. Which is a good idea anyway, if you offer paid service.
– Sergio Tulentsev
Nov 12 '18 at 8:34

Agree. I still miss how I can add workers dynamically without restarting the application. This is not a killer feature, I can always restart the server, but it would be interesting to see if is possible with Sidekiq
– coorasse
Nov 12 '18 at 9:06

@coorasse: add workers? You can just start a new sidekiq process. No need to restart/touch the active ones. Interesting devops task.
– Sergio Tulentsev
Nov 12 '18 at 9:38

add a comment |

If I were to do this, here's what I'd try first:

Wrap all limited jobs in a counter.

on start/dequeue job checks if this user has capacity to run it.

If yes, job runs. If not, it reschedules itself.

Something along these lines:

class MyWorker
 def perform(user_id, *args)
 user = User.find(user_id)
 unless user.has_available_workers
 # re-enqueue with the same args. Possibly, with a delay.
 return 
 end

 user.checkout_worker
 # do work
 ensure
 user.release_worker
 end
end

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

I like your solution, but in this case all users would share the workers and not have dedicated ones. Which is not necessarly a bad idea: it could even scale much better, but it could be that there are no more workers available and the user has to wait (worst case, because he paid)
– coorasse
Nov 12 '18 at 8:20

@coorasse: indeed. In this case, you might want to overprovision the workers, so that there's always some spare capacity. Which is a good idea anyway, if you offer paid service.
– Sergio Tulentsev
Nov 12 '18 at 8:34

Agree. I still miss how I can add workers dynamically without restarting the application. This is not a killer feature, I can always restart the server, but it would be interesting to see if is possible with Sidekiq
– coorasse
Nov 12 '18 at 9:06

@coorasse: add workers? You can just start a new sidekiq process. No need to restart/touch the active ones. Interesting devops task.
– Sergio Tulentsev
Nov 12 '18 at 9:38

add a comment |

If I were to do this, here's what I'd try first:

Wrap all limited jobs in a counter.

on start/dequeue job checks if this user has capacity to run it.

If yes, job runs. If not, it reschedules itself.

Something along these lines:

class MyWorker
 def perform(user_id, *args)
 user = User.find(user_id)
 unless user.has_available_workers
 # re-enqueue with the same args. Possibly, with a delay.
 return 
 end

 user.checkout_worker
 # do work
 ensure
 user.release_worker
 end
end

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

If I were to do this, here's what I'd try first:

Wrap all limited jobs in a counter.

on start/dequeue job checks if this user has capacity to run it.

If yes, job runs. If not, it reschedules itself.

Something along these lines:

class MyWorker
 def perform(user_id, *args)
 user = User.find(user_id)
 unless user.has_available_workers
 # re-enqueue with the same args. Possibly, with a delay.
 return 
 end

 user.checkout_worker
 # do work
 ensure
 user.release_worker
 end
end

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

answered Nov 11 '18 at 22:57

Sergio Tulentsev

179k30289304

I like your solution, but in this case all users would share the workers and not have dedicated ones. Which is not necessarly a bad idea: it could even scale much better, but it could be that there are no more workers available and the user has to wait (worst case, because he paid)
– coorasse
Nov 12 '18 at 8:20

@coorasse: indeed. In this case, you might want to overprovision the workers, so that there's always some spare capacity. Which is a good idea anyway, if you offer paid service.
– Sergio Tulentsev
Nov 12 '18 at 8:34

Agree. I still miss how I can add workers dynamically without restarting the application. This is not a killer feature, I can always restart the server, but it would be interesting to see if is possible with Sidekiq
– coorasse
Nov 12 '18 at 9:06

@coorasse: add workers? You can just start a new sidekiq process. No need to restart/touch the active ones. Interesting devops task.
– Sergio Tulentsev
Nov 12 '18 at 9:38

add a comment |

I like your solution, but in this case all users would share the workers and not have dedicated ones. Which is not necessarly a bad idea: it could even scale much better, but it could be that there are no more workers available and the user has to wait (worst case, because he paid)
– coorasse
Nov 12 '18 at 8:20

@coorasse: indeed. In this case, you might want to overprovision the workers, so that there's always some spare capacity. Which is a good idea anyway, if you offer paid service.
– Sergio Tulentsev
Nov 12 '18 at 8:34

Agree. I still miss how I can add workers dynamically without restarting the application. This is not a killer feature, I can always restart the server, but it would be interesting to see if is possible with Sidekiq
– coorasse
Nov 12 '18 at 9:06

@coorasse: add workers? You can just start a new sidekiq process. No need to restart/touch the active ones. Interesting devops task.
– Sergio Tulentsev
Nov 12 '18 at 9:38

I like your solution, but in this case all users would share the workers and not have dedicated ones. Which is not necessarly a bad idea: it could even scale much better, but it could be that there are no more workers available and the user has to wait (worst case, because he paid)
– coorasse
Nov 12 '18 at 8:20

@coorasse: indeed. In this case, you might want to overprovision the workers, so that there's always some spare capacity. Which is a good idea anyway, if you offer paid service.
– Sergio Tulentsev
Nov 12 '18 at 8:34

Agree. I still miss how I can add workers dynamically without restarting the application. This is not a killer feature, I can always restart the server, but it would be interesting to see if is possible with Sidekiq
– coorasse
Nov 12 '18 at 9:06

@coorasse: add workers? You can just start a new sidekiq process. No need to restart/touch the active ones. Interesting devops task.
– Sergio Tulentsev
Nov 12 '18 at 9:38

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Pfthb