Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NaN or Inf found in input tensor. #94

Open
aman190202 opened this issue Nov 3, 2023 · 0 comments
Open

NaN or Inf found in input tensor. #94

aman190202 opened this issue Nov 3, 2023 · 0 comments

Comments

@aman190202
Copy link

2023-11-03 20:15:53: Checkpoint does not exist. Starting a new training run.
2023-11-03 20:15:53: Number of parameters being optimized: 77622581
2023-11-03 20:15:53: Begin training...
2023-11-03 20:15:56: NaN or Inf found in input tensor.
2023-11-03 20:15:56: Error!
Traceback (most recent call last):
File "/root/zipnerf-pytorch/train.py", line 387, in
app.run(main)
File "/root/miniconda3/envs/zipnerf/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/root/miniconda3/envs/zipnerf/lib/python3.9/site-packages/absl/app.py", line 254, in run_main
sys.exit(main(argv))
File "/root/zipnerf-pytorch/train.py", line 254, in main
summary_writer.add_histogram('train
' + k, v, step)
File "/root/miniconda3/envs/zipnerf/lib/python3.9/site-packages/tensorboardX/writer.py", line 562, in add_histogram
histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
File "/root/miniconda3/envs/zipnerf/lib/python3.9/site-packages/tensorboardX/summary.py", line 210, in histogram
hist = make_histogram(values.astype(float), bins, max_bins)
File "/root/miniconda3/envs/zipnerf/lib/python3.9/site-packages/tensorboardX/summary.py", line 248, in make_histogram
raise ValueError('The histogram is empty, please file a bug report.')
ValueError: The histogram is empty, please file a bug report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant