Question: Modify Resnet File structure and how to import it

[IzanCatalan]

Hi, I would like to modify the structure of the model Resnet50 . My goal is neither to add nor to remove layers, only to replace the convolutions that in the code are made by the pytorch nn.Conv function by convolutions made by the Nvidia CUTLASS library (https://github.com/NVIDIA/cutlass/blob/main/examples/python/02_pytorch_extension_grouped_gemm.ipynb).

I don't intend either to retrain or to modify weights, only to substitute the call to the convolutions with the call to a convolution of cutlass in a similar way to how I describe it in the pytorch forum: https://discuss.pytorch.org/t/using-diffetent-conv2d-ops-with-pre-trained-models/214367

My question is if it is possible, within the guidelines of the repository and then how can I import the file Resnet50 from another file or pytorch as it would be done with https://pytorch.org/vision/main/models.html.

Thanks.

[aisosalo]

Sounds interesting. And sounds like a notable modification to the model. Do you plan to take benefit from pre-trained model weights? Or is it enough for your purposes to have random weights and you will just train the new/modified model to suit your specific application? I suppose it is possible to initialize ResNet50 the normal way and loop trough the model layers and replace torch.nn.Conv2d layers with cutlass.Conv2d layers, but those will then receive random initialization, right. To my understanding, training is required. If you want to use ResNet50 weights, you might be able to freeze the weights of the unaltered layers and train the model for few epochs to get the randomly initialized weights to get more useful values. Always happy to learn if I am mistaken.

[IzanCatalan]

Hi @aisosalo, yes, I intend to use the weights from the pre-trained model, with no intention of making any significant modifications or adding additional layers.

Instead, as you have said, I will replace the torch.nn.Conv2d with cutlass.Conv2d , using one type of convolution for another.

I don't quite understand why re-training the network would be necessary, given that I intend to use the same weights as the original model. Could you please clarify this point?

Additionally, I have a question regarding the use of the model defined in the Resnet50 file. Specifically, could you please advise on importing it into another file with Python and PyTorch to view its metrics and results and perform the inference with that model? Is running python resnet.py with Python sufficient?

[aisosalo]

Replacing a layer with one that operates in a different way, e.g., changing the dynamics of the model, might be considered a significant change, but I do get your point, that it is a small task to perform programmatically.

Yes, since I am not familiar with the cutlass.Conv2d, I am just guessing that it might not be possible to adopt the original weights. This is where I might be wrong.

To implement what you ask would require to make a python file which we could name test_cutlass_conv.py:

import torchvision.models as models
import torch.nn as nn

import cutlass


def replace_conv2d(module):
    for name, child in module.named_children():
        # Replace original layer with CUTLASS layer
        if isinstance(child, nn.Conv2d):
            cutlass_conv = cutlass.Conv2d(  # FIXME: Is there a correspondence in naming the input parameters, and is there a difference in the layout (e.g., NCHW vs. NHWC)
                child.in_channels,
                child.out_channels,
                child.kernel_size,
                stride=child.stride,
                padding=child.padding,
                dilation=child.dilation,
                groups=child.groups,
                bias=child.bias is not None
            )
            # Copy weight from the original layer to the new CUTLASS layer
            cutlass_conv.weight.data = child.weight.data  # FIXME: How cutlass.Conv2d can be assigned a weight attribute?
            # Copy bias from the original layer to the new CUTLASS layer
            if child.bias is not None:
                cutlass_conv.bias.data = child.bias.data  # FIXME: How cutlass.Conv2d can be assigned a bias attribute?
            setattr(module, name, cutlass_conv)
        else:
            replace_conv2d(child)
    return module


if __name__ == "__main__":
    cutlass_model = replace_conv2d(models.resnet50(pretrained=True))
    print(cutlass_model)  # TODO: Replace with more suitable action

This is the idea, not a working solution.

[IzanCatalan]

@aisosalo Yes, I get your point. Thank you for your code example. However, instead of loading models.resnet50, why wouldn't it be better to change the calls to torch.nn.conv2d for cutlass.Conv2d in the same resnet.py file -> (https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py) and then import the model? It would avoid having to use replace_conv2d. As I commented, I don't know precisely how the resnet50 can be imported from the file resnet.py.

[aisosalo]

It will free you from the manual work if you study the cutlass.Conv2d documentation and try to make the custom replace_conv2d function to work. I am not saying that it would not be educational to do the replacements once by hand, you can surely do that. The PyTorch resnet.py implements the ResNet's in a very sophisticated way with many functionalities and I wouldn't know which parts of the script to copy-paste to get just the ResNet50. ChatGPT might be able to help with that. In any case, if you decide to do the replacing by hand, you can not avoid studying the CUTLASS conv.py and comparing cutlass.Conv2d with torch.nn.Conv2d documentation. Good luck! And if you manage to get it working, consider posting the solution here.

[IzanCatalan]

@aisosalo, Yes, I have studied cutlass conv2d. I will try to implement what we have discussed and let you know of any progress. Just to be sure, how do I execute Pytorch resnet.py?

And if I am successful, how can I use the new resnet.py? Do I need to build the Torhcvision repo from source and install it with pip? Or how do I import the new model to use it and load it like any other pre-trained model with Pytorch?

[aisosalo]

@IzanCatalan You could make a copy of the resnet.py, say cutlass_resnet.py, for modifications and then have a main.py (in the same file location) to be run by python main.py:

import torch
#from torchvision.models.resnet import ResNet, Bottleneck
from cutlass_resnet import ResNet, Bottleneck

def resnet50(pretrained=False, **kwargs):
    # Initialize the model
    model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
    
    # Load pretrained weights if needed
    if pretrained:  # TODO
        pass  # model.load_state_dict(torch.load('path_to_pretrained_weights.pth'))
    
    return model


if __name__ == "__main__":
    model = resnet50(pretrained=False)  # 'pretrained' is set as False as the feature is not implemented
    print(model)

If you wish to use your customized ResNet with pretrained weights, you'll need to ensure that the architecture matches the original ResNet structure, or to modify the weight loading process to take into account the changes you made.

pip install should be enough.

[IzanCatalan]

@aisosalo I will check it soon, and I will let you know of any updates.

But I wonder if when you say "pip install should be enough", you mean that in order to do from cutlass_resnet import ResNet, Bottleneck first, I need to do pip install cutlast_resnet.py or pip install should be done after running python main.py? If it is the second option, pip install ...'x', being 'x', what exactly? Sorry if I didn't understand you correctly.

[aisosalo]

@IzanCatalan You need to have torch, torchvision, and cutlass installed to do any of this, right.