scaling django app that relies heavily on database queries
Hello I was load testing my social media platform with LocustIO that uses django 2.1 and SQlite3, and these are my results. Load Testing Results
I need to make sure it can run for 10000+ visitors and have looked into caching options like redis. It is a real time app so it involves heavy database usage and power. What road should I go down to make it quicker and scalable?
python django locust
asked Nov 13 '18 at 22:54
Ty CTy C
399
|
show 2 more comments
Hello I was load testing my social media platform with LocustIO that uses django 2.1 and SQlite3, and these are my results. Load Testing Results
I need to make sure it can run for 10000+ visitors and have looked into caching options like redis. It is a real time app so it involves heavy database usage and power. What road should I go down to make it quicker and scalable?
python django locust
asked Nov 13 '18 at 22:54
Ty CTy C
399
Welcome to SO! SQlite3 is not really meant for a production environment, at least not for busy sites. And yes, caching is important and Memcached etc. is your best friend. BTW 10K visitors a day is not much while 10K concurrent visitors is a slightly different story. Which one is it?
– WayBehind
Nov 14 '18 at 1:44
@WayBehind thanks for the response. I am planning for 10K concurrent visitors. I recently switched my database to Postgres. I’ve looked up possible solutions such as web sockets and pusher. It’s an app where tokens are being bought and spent for context.
– Ty C
Nov 14 '18 at 1:47
Just to give you an idea. On AWS I run Django app on EalactiBeanstalk with two T2.small servers behind a Loadbalancer, and just one cache.t2.small Memcached server along t2.medium MySQL RDS (Multi AZ) and its all running at about at 10% capacity while having about 2M requests per month through CloudFlare.
– WayBehind
Nov 14 '18 at 1:56
@WayBehind Do you use a classic load balancer through AWS? Also what do you use CloudFare for if you are using EalactiBeanstock and AWS?
– Ty C
Nov 14 '18 at 6:12
Yes, classic LoadBalancer. As per CloudFlare, there is a lot of benefits to use their DNS etc but one of the main reasons is that I dont have to deal with SSL certificates anymore as those are included.
– WayBehind
Nov 14 '18 at 9:09
|
show 2 more comments
Hello I was load testing my social media platform with LocustIO that uses django 2.1 and SQlite3, and these are my results. Load Testing Results
I need to make sure it can run for 10000+ visitors and have looked into caching options like redis. It is a real time app so it involves heavy database usage and power. What road should I go down to make it quicker and scalable?
python django locust
asked Nov 13 '18 at 22:54
Ty CTy C
399
Hello I was load testing my social media platform with LocustIO that uses django 2.1 and SQlite3, and these are my results. Load Testing Results
I need to make sure it can run for 10000+ visitors and have looked into caching options like redis. It is a real time app so it involves heavy database usage and power. What road should I go down to make it quicker and scalable?
python django locust
python django locust
asked Nov 13 '18 at 22:54
Ty CTy C
399
asked Nov 13 '18 at 22:54
Ty CTy C
399
asked Nov 13 '18 at 22:54
Ty CTy C
399
asked Nov 13 '18 at 22:54
Ty CTy C
399
asked Nov 13 '18 at 22:54
Ty CTy C
399
399
Welcome to SO! SQlite3 is not really meant for a production environment, at least not for busy sites. And yes, caching is important and Memcached etc. is your best friend. BTW 10K visitors a day is not much while 10K concurrent visitors is a slightly different story. Which one is it?
– WayBehind
Nov 14 '18 at 1:44
@WayBehind thanks for the response. I am planning for 10K concurrent visitors. I recently switched my database to Postgres. I’ve looked up possible solutions such as web sockets and pusher. It’s an app where tokens are being bought and spent for context.
– Ty C
Nov 14 '18 at 1:47
Just to give you an idea. On AWS I run Django app on EalactiBeanstalk with two T2.small servers behind a Loadbalancer, and just one cache.t2.small Memcached server along t2.medium MySQL RDS (Multi AZ) and its all running at about at 10% capacity while having about 2M requests per month through CloudFlare.
– WayBehind
Nov 14 '18 at 1:56
@WayBehind Do you use a classic load balancer through AWS? Also what do you use CloudFare for if you are using EalactiBeanstock and AWS?
– Ty C
Nov 14 '18 at 6:12
Yes, classic LoadBalancer. As per CloudFlare, there is a lot of benefits to use their DNS etc but one of the main reasons is that I dont have to deal with SSL certificates anymore as those are included.
– WayBehind
Nov 14 '18 at 9:09
|
show 2 more comments
Welcome to SO! SQlite3 is not really meant for a production environment, at least not for busy sites. And yes, caching is important and Memcached etc. is your best friend. BTW 10K visitors a day is not much while 10K concurrent visitors is a slightly different story. Which one is it?
– WayBehind
Nov 14 '18 at 1:44
@WayBehind thanks for the response. I am planning for 10K concurrent visitors. I recently switched my database to Postgres. I’ve looked up possible solutions such as web sockets and pusher. It’s an app where tokens are being bought and spent for context.
– Ty C
Nov 14 '18 at 1:47
Just to give you an idea. On AWS I run Django app on EalactiBeanstalk with two T2.small servers behind a Loadbalancer, and just one cache.t2.small Memcached server along t2.medium MySQL RDS (Multi AZ) and its all running at about at 10% capacity while having about 2M requests per month through CloudFlare.
– WayBehind
Nov 14 '18 at 1:56
@WayBehind Do you use a classic load balancer through AWS? Also what do you use CloudFare for if you are using EalactiBeanstock and AWS?
– Ty C
Nov 14 '18 at 6:12
Yes, classic LoadBalancer. As per CloudFlare, there is a lot of benefits to use their DNS etc but one of the main reasons is that I dont have to deal with SSL certificates anymore as those are included.
– WayBehind
Nov 14 '18 at 9:09
Welcome to SO! SQlite3 is not really meant for a production environment, at least not for busy sites. And yes, caching is important and Memcached etc. is your best friend. BTW 10K visitors a day is not much while 10K concurrent visitors is a slightly different story. Which one is it?
– WayBehind
Nov 14 '18 at 1:44
Welcome to SO! SQlite3 is not really meant for a production environment, at least not for busy sites. And yes, caching is important and Memcached etc. is your best friend. BTW 10K visitors a day is not much while 10K concurrent visitors is a slightly different story. Which one is it?
– WayBehind
Nov 14 '18 at 1:44
@WayBehind thanks for the response. I am planning for 10K concurrent visitors. I recently switched my database to Postgres. I’ve looked up possible solutions such as web sockets and pusher. It’s an app where tokens are being bought and spent for context.
– Ty C
Nov 14 '18 at 1:47
@WayBehind thanks for the response. I am planning for 10K concurrent visitors. I recently switched my database to Postgres. I’ve looked up possible solutions such as web sockets and pusher. It’s an app where tokens are being bought and spent for context.
– Ty C
Nov 14 '18 at 1:47
Just to give you an idea. On AWS I run Django app on EalactiBeanstalk with two T2.small servers behind a Loadbalancer, and just one cache.t2.small Memcached server along t2.medium MySQL RDS (Multi AZ) and its all running at about at 10% capacity while having about 2M requests per month through CloudFlare.
– WayBehind
Nov 14 '18 at 1:56
Just to give you an idea. On AWS I run Django app on EalactiBeanstalk with two T2.small servers behind a Loadbalancer, and just one cache.t2.small Memcached server along t2.medium MySQL RDS (Multi AZ) and its all running at about at 10% capacity while having about 2M requests per month through CloudFlare.
– WayBehind
Nov 14 '18 at 1:56
@WayBehind Do you use a classic load balancer through AWS? Also what do you use CloudFare for if you are using EalactiBeanstock and AWS?
– Ty C
Nov 14 '18 at 6:12
@WayBehind Do you use a classic load balancer through AWS? Also what do you use CloudFare for if you are using EalactiBeanstock and AWS?
– Ty C
Nov 14 '18 at 6:12
Yes, classic LoadBalancer. As per CloudFlare, there is a lot of benefits to use their DNS etc but one of the main reasons is that I dont have to deal with SSL certificates anymore as those are included.
– WayBehind
Nov 14 '18 at 9:09
Yes, classic LoadBalancer. As per CloudFlare, there is a lot of benefits to use their DNS etc but one of the main reasons is that I dont have to deal with SSL certificates anymore as those are included.
– WayBehind
Nov 14 '18 at 9:09
|
show 2 more comments
2 Answers
2
active
oldest
votes
So the first thing you'll want to do is figure out where your time is being taken up. Is most of the time taken up with database queries? other network operations? page rendering?
Django-debug-toolbar is a pretty good first step at answering that all-important first question.
Then once you have some idea of where the problem is, you can focus your brain power on solving that problem.
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
add a comment |
Sqlite 3 cant handle much concurrency.So migrate to more robust sql.
Also Check out silk.
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290736%2fscaling-django-app-that-relies-heavily-on-database-queries%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
So the first thing you'll want to do is figure out where your time is being taken up. Is most of the time taken up with database queries? other network operations? page rendering?
Django-debug-toolbar is a pretty good first step at answering that all-important first question.
Then once you have some idea of where the problem is, you can focus your brain power on solving that problem.
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
add a comment |
So the first thing you'll want to do is figure out where your time is being taken up. Is most of the time taken up with database queries? other network operations? page rendering?
Django-debug-toolbar is a pretty good first step at answering that all-important first question.
Then once you have some idea of where the problem is, you can focus your brain power on solving that problem.
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
add a comment |
So the first thing you'll want to do is figure out where your time is being taken up. Is most of the time taken up with database queries? other network operations? page rendering?
Django-debug-toolbar is a pretty good first step at answering that all-important first question.
Then once you have some idea of where the problem is, you can focus your brain power on solving that problem.
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
So the first thing you'll want to do is figure out where your time is being taken up. Is most of the time taken up with database queries? other network operations? page rendering?
Django-debug-toolbar is a pretty good first step at answering that all-important first question.
Then once you have some idea of where the problem is, you can focus your brain power on solving that problem.
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
answered Nov 14 '18 at 0:41
Chris CurveyChris Curvey
3,06642843
3,06642843
add a comment |
add a comment |
Sqlite 3 cant handle much concurrency.So migrate to more robust sql.
Also Check out silk.
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
add a comment |
Sqlite 3 cant handle much concurrency.So migrate to more robust sql.
Also Check out silk.
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
add a comment |
Sqlite 3 cant handle much concurrency.So migrate to more robust sql.
Also Check out silk.
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
Sqlite 3 cant handle much concurrency.So migrate to more robust sql.
Also Check out silk.
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
answered Nov 14 '18 at 3:52
Garry KevinGarry Kevin
114
114
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290736%2fscaling-django-app-that-relies-heavily-on-database-queries%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Welcome to SO! SQlite3 is not really meant for a production environment, at least not for busy sites. And yes, caching is important and Memcached etc. is your best friend. BTW 10K visitors a day is not much while 10K concurrent visitors is a slightly different story. Which one is it?
– WayBehind
Nov 14 '18 at 1:44
@WayBehind thanks for the response. I am planning for 10K concurrent visitors. I recently switched my database to Postgres. I’ve looked up possible solutions such as web sockets and pusher. It’s an app where tokens are being bought and spent for context.
– Ty C
Nov 14 '18 at 1:47
Just to give you an idea. On AWS I run Django app on EalactiBeanstalk with two T2.small servers behind a Loadbalancer, and just one cache.t2.small Memcached server along t2.medium MySQL RDS (Multi AZ) and its all running at about at 10% capacity while having about 2M requests per month through CloudFlare.
– WayBehind
Nov 14 '18 at 1:56
@WayBehind Do you use a classic load balancer through AWS? Also what do you use CloudFare for if you are using EalactiBeanstock and AWS?
– Ty C
Nov 14 '18 at 6:12
Yes, classic LoadBalancer. As per CloudFlare, there is a lot of benefits to use their DNS etc but one of the main reasons is that I dont have to deal with SSL certificates anymore as those are included.
– WayBehind
Nov 14 '18 at 9:09