fix rmm managed memory resource initialization to resolve some intermittent … #787
Conversation
…memory issuespin numpy < 1 in readme Signed-off-by: Erik Ordentlich <[email protected]>
|
build |
Signed-off-by: Erik Ordentlich <[email protected]>
eordentlich
changed the title
|
build |
| if not type(rmm.mr.get_current_device_resource()) == type( | ||
| rmm.mr.ManagedMemoryResource() | ||
| ): | ||
| rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource()) |
There was a problem hiding this comment.
Looks good.
One question: any idea on the difference of this line compared with "rmm.reinitialize(managed_memory=True)"?
There was a problem hiding this comment.
Yes. rmm.reinitialize also resets memory resources for previously specified devices to default mr's. This I believe was the source of our problems. It destroyed the original memory resources, replacing the device to mr mapping to point to the new defaults. However this is only done on the python side. On the c++ side, the corresponding map still points to the destroyed mrs (which are the same object), leading to dangling pointers. So this would lead to undefined behavior in the C++ rmm invocations, if the devices were changing in the same process, like between fit and transform, with python worker reuse on. The observations and code inspection were consistent with the above.
…memory issues
Looks like rmm.reinitialize destroys memory resources assigned to all other devices besides the current device. This way of initializing as in this PR and doing only once per process avoids this. I think the intermittent cuda errors were due to hit or miss uses of destroyed resources on the c++ api rmm operations. So this should address the issue.
Also, in umap, current device was set before the memory resource was initialized. Needs to be the other way around.
also pin numpy < 1 in readme