C# keeping track of daily tasks done
i'm trying to figure out a way to optimize the way i count daily actions done.
I have a software that runs a bunch of "task executors", which all have an id.
Each task executor logs all tasks done in a local mongo database, with executor id, task type and epoch timestamp.
In order to count daily actions of specific types for each of the task executors, i'm querying mongo to count logs of the action type, for the task executor id and with an epoch > now minus 24 hours.
This is done atleast once per minute by each task executor, but it's quite inefficient, when the collections are huge(100k-1mil).
How else could i count how many of a specific task, each executor id has done within the last 24 hours?
A simpler way to put it, is how would you keep count of daily activity in any application, that doesn't run 24/7? And then being able to access that count very frequently.
Any advice would be GREATLY appreciated!
Thanks
c# mongodb
asked Nov 12 '18 at 15:23
VictornorVictornor
385
|
show 8 more comments
i'm trying to figure out a way to optimize the way i count daily actions done.
I have a software that runs a bunch of "task executors", which all have an id.
Each task executor logs all tasks done in a local mongo database, with executor id, task type and epoch timestamp.
In order to count daily actions of specific types for each of the task executors, i'm querying mongo to count logs of the action type, for the task executor id and with an epoch > now minus 24 hours.
This is done atleast once per minute by each task executor, but it's quite inefficient, when the collections are huge(100k-1mil).
How else could i count how many of a specific task, each executor id has done within the last 24 hours?
A simpler way to put it, is how would you keep count of daily activity in any application, that doesn't run 24/7? And then being able to access that count very frequently.
Any advice would be GREATLY appreciated!
Thanks
c# mongodb
asked Nov 12 '18 at 15:23
VictornorVictornor
385
Maybe create a table containing aggregated results? I don't think I fully understood what you wanted though :/
– Haytam
Nov 12 '18 at 15:28
I rewrote the question, hopefully it's clearer now.
– Victornor
Nov 12 '18 at 15:39
Sounds like you need to add an index.
– Liam
Nov 12 '18 at 15:40
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
– Nick
Nov 12 '18 at 15:41
I do have indexes in the collection, but there's over 10k task executors running simultaneously, an they're each making this CountDocument query every 40-60 seconds.
– Victornor
Nov 12 '18 at 15:42
|
show 8 more comments
i'm trying to figure out a way to optimize the way i count daily actions done.
I have a software that runs a bunch of "task executors", which all have an id.
Each task executor logs all tasks done in a local mongo database, with executor id, task type and epoch timestamp.
In order to count daily actions of specific types for each of the task executors, i'm querying mongo to count logs of the action type, for the task executor id and with an epoch > now minus 24 hours.
This is done atleast once per minute by each task executor, but it's quite inefficient, when the collections are huge(100k-1mil).
How else could i count how many of a specific task, each executor id has done within the last 24 hours?
A simpler way to put it, is how would you keep count of daily activity in any application, that doesn't run 24/7? And then being able to access that count very frequently.
Any advice would be GREATLY appreciated!
Thanks
c# mongodb
asked Nov 12 '18 at 15:23
VictornorVictornor
385
i'm trying to figure out a way to optimize the way i count daily actions done.
I have a software that runs a bunch of "task executors", which all have an id.
Each task executor logs all tasks done in a local mongo database, with executor id, task type and epoch timestamp.
In order to count daily actions of specific types for each of the task executors, i'm querying mongo to count logs of the action type, for the task executor id and with an epoch > now minus 24 hours.
This is done atleast once per minute by each task executor, but it's quite inefficient, when the collections are huge(100k-1mil).
How else could i count how many of a specific task, each executor id has done within the last 24 hours?
A simpler way to put it, is how would you keep count of daily activity in any application, that doesn't run 24/7? And then being able to access that count very frequently.
Any advice would be GREATLY appreciated!
Thanks
c# mongodb
c# mongodb
asked Nov 12 '18 at 15:23
VictornorVictornor
385
asked Nov 12 '18 at 15:23
VictornorVictornor
385
edited Nov 12 '18 at 15:40
Victornor
asked Nov 12 '18 at 15:23
VictornorVictornor
385
asked Nov 12 '18 at 15:23
VictornorVictornor
385
asked Nov 12 '18 at 15:23
VictornorVictornor
385
385
Maybe create a table containing aggregated results? I don't think I fully understood what you wanted though :/
– Haytam
Nov 12 '18 at 15:28
I rewrote the question, hopefully it's clearer now.
– Victornor
Nov 12 '18 at 15:39
Sounds like you need to add an index.
– Liam
Nov 12 '18 at 15:40
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
– Nick
Nov 12 '18 at 15:41
I do have indexes in the collection, but there's over 10k task executors running simultaneously, an they're each making this CountDocument query every 40-60 seconds.
– Victornor
Nov 12 '18 at 15:42
|
show 8 more comments
Maybe create a table containing aggregated results? I don't think I fully understood what you wanted though :/
– Haytam
Nov 12 '18 at 15:28
I rewrote the question, hopefully it's clearer now.
– Victornor
Nov 12 '18 at 15:39
Sounds like you need to add an index.
– Liam
Nov 12 '18 at 15:40
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
– Nick
Nov 12 '18 at 15:41
I do have indexes in the collection, but there's over 10k task executors running simultaneously, an they're each making this CountDocument query every 40-60 seconds.
– Victornor
Nov 12 '18 at 15:42
Maybe create a table containing aggregated results? I don't think I fully understood what you wanted though :/
– Haytam
Nov 12 '18 at 15:28
Maybe create a table containing aggregated results? I don't think I fully understood what you wanted though :/
– Haytam
Nov 12 '18 at 15:28
I rewrote the question, hopefully it's clearer now.
– Victornor
Nov 12 '18 at 15:39
I rewrote the question, hopefully it's clearer now.
– Victornor
Nov 12 '18 at 15:39
Sounds like you need to add an index.
– Liam
Nov 12 '18 at 15:40
Sounds like you need to add an index.
– Liam
Nov 12 '18 at 15:40
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
– Nick
Nov 12 '18 at 15:41
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
– Nick
Nov 12 '18 at 15:41
I do have indexes in the collection, but there's over 10k task executors running simultaneously, an they're each making this CountDocument query every 40-60 seconds.
– Victornor
Nov 12 '18 at 15:42
I do have indexes in the collection, but there's over 10k task executors running simultaneously, an they're each making this CountDocument query every 40-60 seconds.
– Victornor
Nov 12 '18 at 15:42
|
show 8 more comments
1 Answer
1
active
oldest
votes
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
answered Nov 12 '18 at 15:44
NickNick
1,723912
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53265201%2fc-sharp-keeping-track-of-daily-tasks-done%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
answered Nov 12 '18 at 15:44
NickNick
1,723912
add a comment |
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
answered Nov 12 '18 at 15:44
NickNick
1,723912
add a comment |
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
answered Nov 12 '18 at 15:44
NickNick
1,723912
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
answered Nov 12 '18 at 15:44
NickNick
1,723912
answered Nov 12 '18 at 15:44
NickNick
1,723912
answered Nov 12 '18 at 15:44
NickNick
1,723912
answered Nov 12 '18 at 15:44
NickNick
1,723912
1,723912
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53265201%2fc-sharp-keeping-track-of-daily-tasks-done%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Maybe create a table containing aggregated results? I don't think I fully understood what you wanted though :/
– Haytam
Nov 12 '18 at 15:28
I rewrote the question, hopefully it's clearer now.
– Victornor
Nov 12 '18 at 15:39
Sounds like you need to add an index.
– Liam
Nov 12 '18 at 15:40
If I were you, I would count the tasks every hour and store the intermediate results. Then, for one day, you will just collect the 24 entries and sum the result.
– Nick
Nov 12 '18 at 15:41
I do have indexes in the collection, but there's over 10k task executors running simultaneously, an they're each making this CountDocument query every 40-60 seconds.
– Victornor
Nov 12 '18 at 15:42