Pfthb

I've a deep learning model. To query the model I use celery to make it asynchronous, as number of query requests is very large. The number of workers that spins up while starting the service is equal to number of cores that is available in the system. For example if I have a 8 core machine, it'll spin up 8 workers.

I use the following line to do asynchronous request using celery:

response = flask_with_celery.inference.apply_async((service_id, locale, query), queue='inference')

where inference is a celery task function. redis is the backend for celery here.

I do performance testing using jmeter. The testing is being done for 10 minutes for x concurrent users. I try the value of x here with 20, 30, 40, 50, 100.

After some time, at the end of 50 concurrent users or start of 100 concurrent users, the code throws error.

[2018-11-12 14:36:12,815] ERROR in app: Exception on /process_qna_query [POST]
 File "site-packages/flask/app.py", line 1982, in wsgi_app
 File "site-packages/flask/app.py", line 1614, in full_dispatch_request
 File "site-packages/flask_cors/extension.py", line 161, in wrapped_function
 File "site-packages/flask/app.py", line 1517, in handle_user_exception
 File "site-packages/flask/_compat.py", line 33, in reraise
 File "site-packages/flask/app.py", line 1612, in full_dispatch_request
 File "site-packages/flask/app.py", line 1598, in dispatch_request
 File "FAQ.py", line 103, in json_test
 File "site-packages/celery/result.py", line 194, in get
 File "site-packages/celery/backends/async.py", line 189, in wait_for_pending
 File "site-packages/celery/backends/async.py", line 256, in _wait_for_pending
 File "site-packages/celery/backends/async.py", line 57, in drain_events_until
 File "site-packages/celery/backends/async.py", line 66, in wait_for
 File "site-packages/celery/backends/redis.py", line 69, in drain_events
 File "site-packages/redis/client.py", line 2513, in get_message
 File "site-packages/redis/client.py", line 2428, in parse_response
 File "site-packages/redis/connection.py", line 619, in can_read
 ValueError: filedescriptor out of range in select()

If I start testing for 100 conc users first, then 50 and so on. I get the error when testing with 20 concurrent users. Clearly it's not about number of users. The number of request it gets for 100 users in 10 minutes is roughly the same as number of request it gets for 20 users.

Edit: I changed the celery results backend to memcached and I'm not getting any errors!