Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve parallelism using Cython OpenMP #8

Open
lrauschning opened this issue Nov 18, 2022 · 2 comments
Open

Improve parallelism using Cython OpenMP #8

lrauschning opened this issue Nov 18, 2022 · 2 comments

Comments

@lrauschning
Copy link
Collaborator

The parallelization in the reduce_parallel function could be improved by substituting python multiprocessing with the Cytoh OpenMP backend.
S. also https://cython.readthedocs.io/en/latest/src/userguide/parallelism.html

@mnshgl0110
Copy link
Member

I will be very careful using this. I am not sure how stable and platform/environment independent this is. It is possible, that in the long run, it might be more time-consuming and annoying to maintain it and to ensure that it works for everyone then to just make multiprocessing (a python standard library function) to work.

@lrauschning
Copy link
Collaborator Author

lrauschning commented Nov 25, 2022

Hm, maybe its worth making this a compile-time or runtime flag.
Another way to improve performance would be to make the input file reading async, which should be possible without using openmp or anything, and reading the files in is definitely a bottleneck.
I think I'll play around with the code in pansyn.pyx a bit next week and see if these speedups can be implemented quickly-ish and cleanly; having pansyri be a bit faster would be neat also for testing, but it's not terrible as it is right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants