Add CUDA build support and some code refinements

[wenbingl]

No description provided.

[wenbingl]

@souptc , @RandySheriffH , it looks like we at least need 4 files to write a simplest cuda kernel due to the limitation of nvcc, is there any good idea for this?

[RandySheriffH]

Yes, let's keep it this way for bypass NVCC incompatibilities for now.

[wejoncy]

Hi @wenbingl Is it possible to support a general Cuda Op? This op is only responsible for converting tensor to DLPACK and collect output tensor.
The benefits are that we don't have to write a specific kernel for a specific op., and we could free the flexibility to user to support any kinds of customized operations just like what torch.extentions does.

[wenbingl]

Hi @wenbingl Is it possible to support a general Cuda Op? This op is only responsible for converting tensor to DLPACK and collect output tensor. The benefits are that we don't have to write a specific kernel for a specific op., and we could free the flexibility to user to support any kinds of customized operations just like what torch.extentions does.

The current goal is to implement a kernel just like we did in onnxruntime. Are you suggesting how we can support the CUDA custom op in Python, which hasn't been implemented yet?

[wejoncy]

Hi @wenbingl Is it possible to support a general Cuda Op? This op is only responsible for converting tensor to DLPACK and collect output tensor. The benefits are that we don't have to write a specific kernel for a specific op., and we could free the flexibility to user to support any kinds of customized operations just like what torch.extentions does.

The current goal is to implement a kernel just like we did in onnxruntime. Are you suggesting how we can support the CUDA custom op in Python, which hasn't been implemented yet?

Yes, It seems quite useful if we have such a mechanism to support user customized cuda kernel but no touch python inside the graph as what ort-training did (python-op).
With this, users can easily add a custom kernel support by registering a kernel function pointer to ORT, and ort can just call the function pointer (signautre is (DLTensor* tensors, int tensor_lengths, const char* function_name)). This will allow us easily to support all torch extension Ops, TVM ops or any user implemented ops without needing to merge these kernels to ORT/ORT-ext.

[wenbingl]

Hi @wenbingl Is it possible to support a general Cuda Op? This op is only responsible for converting tensor to DLPACK and collect output tensor. The benefits are that we don't have to write a specific kernel for a specific op., and we could free the flexibility to user to support any kinds of customized operations just like what torch.extentions does.

The current goal is to implement a kernel just like we did in onnxruntime. Are you suggesting how we can support the CUDA custom op in Python, which hasn't been implemented yet?

Yes, It seems quite useful if we have such a mechanism to support user customized cuda kernel but no touch python inside the graph as what ort-training did (python-op). With this, users can easily add a custom kernel support by registering a kernel function pointer to ORT, and ort can just call the function pointer (signautre is (DLTensor* tensors, int tensor_lengths, const char* function_name)). This will allow us easily to support all torch extension Ops, TVM ops or any user implemented ops without needing to merge these kernels to ORT/ORT-ext.

It's a good idea and it's another 'extensions' for the current custom op mechanism. Do you have any use case or a good torch extensions/TVM ops that can be integrated?

[wejoncy]

It's a good idea and it's another 'extensions' for the current custom op mechanism. Do you have any use case or a good torch extensions/TVM ops that can be integrated?

Yes, PagedAttention , Flashattention, memory_efficent_attn
highly optimized w4a16/w8a8 kernels for Matmul in TVM-MLC/TensorRT-LLM.
I know it's possible to port those cuda code into ORT, but it's really hard to maintain.

Especially, all these kernels can comunicate with ORT by DLPACK Tensor.