Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

Akshay-Venkatesh
Copy link
Contributor

Why/What?

Export handles are used for VA recycling check today but this is not guaranteed to be unique for newer memory allocators like VMM, and CUDA memory pools. We need to switch to using buffer_id attribute to perform this check.

@Akshay-Venkatesh
Copy link
Contributor Author

@yosefe , @tvegas1 We got reports of VA recycling not being detected internally. This PR aims to address the issue for newer allocators. If the changes look good, is there an option to backport this to 1.18 at all?

src/uct/cuda/cuda_ipc/cuda_ipc_md.h Show resolved Hide resolved
arg1 = (const void *)&key->ph;
arg2 = (const void *)&region->key.ph;
#endif
if (memcmp(arg1, arg2, cmp_size) == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it makes sense to update the test to cover this behavior?

Comment on lines 496 to 504
#if HAVE_CUDA_FABRIC
cmp_size = sizeof(key->ph.buffer_id);
arg1 = (const void *)&key->ph.buffer_id;
arg2 = (const void *)&region->key.ph.buffer_id;
#else
cmp_size = sizeof(key->ph);
arg1 = (const void *)&key->ph;
arg2 = (const void *)&region->key.ph;
#endif
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. need to fix indent
  2. any reason to not compare buffer_id also for legacy, regardless of HAVE_CUDA_FABRIC?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason to not compare buffer_id also for legacy, regardless of HAVE_CUDA_FABRIC?

I had this to begin with but key->ph is of type CUipcMemHandle when HAVE_CUDA_FABRIC is not declared. So we would need to change the type if we want to also include buffer_id.

src/uct/cuda/cuda_ipc/cuda_ipc_md.h Show resolved Hide resolved
@tvegas1
Copy link
Contributor

tvegas1 commented Jan 7, 2025

Export handles are used for VA recycling check today but this is not guaranteed to be unique for newer memory allocators like VMM, and CUDA memory pools. We need to switch to using buffer_id attribute to perform this check.

This PR solves the case where cuda reused pointer value (alloc/free/alloc), and failed to detect that it's actually not same allocation instance, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants