Training failed

[renqianluo]

Hi, thanks for your implementation of the work! I see you only experiment on NASBench. Following the settings on Imagenet experiment in the original paper, I modify the GCN to be the same as in the paper (18 layers), but the training fails. The loss keeps constant and the output value is near zero. Do you have any idea on this?

[ultmaster]

I didn't do experiments on ImageNet because I don't have the training data. According to paper, they have trained up to 130 models on ImageNet and I don't have the resource to do that. Wondering how you obtain that data?

The model failing to converge can be caused by many things. First of all, you should check whether your learning rate, batch size and other hyper-parameters are appropriate. Then, you should check whether your data have any mistakes. If not, a simple way to figure it out is to downsize the training data to see if the model fits to several records.

[renqianluo]

@ultmaster 我也搞不到原始Paper的imagenet数据，我用的其实是one-shot模型，用shared-weights训练的1000个模型然后eval出准确率，作为训练数据来训练GCN的。这批数据我用其他的neural predictor（比如LSTM）都可以正常学习并且fit，但是用这个GCN就学不出来了，所有参数都和原始paper里一样

[ultmaster]

看起来像是模型有问题，你可以先看看模型对于一个样本收不收敛。

[cardwing]

@renqianluo Since the predictor uses the sigmoid layer to process the final output, you may need to use the binary cross-entropy loss instead of the l2 loss.

[ultmaster]

Let's continue the discussion in #2.