feat: add gevent as an openedx requirement #15
Conversation
| @@ -0,0 +1,2 @@ | |||
| RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ | |||
| pip install "gevent==24.11.1" | |||
There was a problem hiding this comment.
This works as is? Don't you have to add a flag to celery to enable gevent?
There was a problem hiding this comment.
Yes, you need to set pool=gevent and a high concurrency value such as 100 or 400. It depends on the workload. The default is prefork, which works well for CPU-bound tasks.
I've tested Aspects with a dedicated deployment and routing I/O bound tasks there, but it wasn't worth running Aspects like. However, there are advantages when running a dedicated I/O bound deployment but it would require a good mapping of tasks and that's a large effort this PR doesn't try to solve
There was a problem hiding this comment.
But if aspects is not a good use case for that then what would be?
I'm thinking that you add the io_bound deployment with the correct celery and put that and the python requirements behind a configuration setting, by default false. We can later enable it when needed and use CELERY_LMS_EXPLICIT_QUEUES to route the tasks that make sense to that deployment.
This PR adds
geventas an openedx requirement, allowing celery workers to run with a pool optimized for I/O bound tasks.It also adds an examples for the recommended settings on a production system and for running a specific (and optimized) celery queue for Aspects.