RuntimeError: Expected is_sm80 to be true, but got false.

[Qiang-HU]

╭───────────────────────── Traceback (most recent call last) ──────────────────────────╮
│ /Project/lq_ChatGLM2-6B/LLM-Tuning/chatglm2_lora_tuning.py:172 in │
│ │
│ 169 │
│ 170 │
│ 171 if name == "main": │
│ ❱ 172 │ main() │
│ 173 │
│ │
│ /Project/lq_ChatGLM2-6B/LLM-Tuning/chatglm2_lora_tuning.py:165 in main │
│ │
│ 162 │ │ callbacks=[TensorBoardCallback(writer)], │
│ 163 │ │ data_collator=data_collator │
│ 164 │ ) │
│ ❱ 165 │ trainer.train() │
│ 166 │ writer.close() │
│ 167 │ # save model │
│ 168 │ model.save_pretrained(training_args.output_dir) │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/transformers/trainer.py:1662 in │
│ train │
│ │
│ 1659 │ │ inner_training_loop = find_executable_batch_size( │
│ 1660 │ │ │ self._inner_training_loop, self._train_batch_size, args.auto_find │
│ 1661 │ │ ) │
│ ❱ 1662 │ │ return inner_training_loop( │
│ 1663 │ │ │ args=args, │
│ 1664 │ │ │ resume_from_checkpoint=resume_from_checkpoint, │
│ 1665 │ │ │ trial=trial, │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/transformers/trainer.py:1929 in │
│ _inner_training_loop │
│ │
│ 1926 │ │ │ │ │ with model.no_sync(): │
│ 1927 │ │ │ │ │ │ tr_loss_step = self.training_step(model, inputs) │
│ 1928 │ │ │ │ else: │
│ ❱ 1929 │ │ │ │ │ tr_loss_step = self.training_step(model, inputs) │
│ 1930 │ │ │ │ │
│ 1931 │ │ │ │ if ( │
│ 1932 │ │ │ │ │ args.logging_nan_inf_filter │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/transformers/trainer.py:2709 in │
│ training_step │
│ │
│ 2706 │ │ │ loss = loss / self.args.gradient_accumulation_steps │
│ 2707 │ │ │
│ 2708 │ │ if self.do_grad_scaling: │
│ ❱ 2709 │ │ │ self.scaler.scale(loss).backward() │
│ 2710 │ │ elif self.use_apex: │
│ 2711 │ │ │ with amp.scale_loss(loss, self.optimizer) as scaled_loss: │
│ 2712 │ │ │ │ scaled_loss.backward() │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/torch/_tensor.py:487 in backward │
│ │
│ 484 │ │ │ │ create_graph=create_graph, │
│ 485 │ │ │ │ inputs=inputs, │
│ 486 │ │ │ ) │
│ ❱ 487 │ │ torch.autograd.backward( │
│ 488 │ │ │ self, gradient, retain_graph, create_graph, inputs=inputs │
│ 489 │ │ ) │
│ 490 │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/torch/autograd/init.py:200 in │
│ backward │
│ │
│ 197 │ # The reason we repeat same the comment below is that │
│ 198 │ # some Python versions print out the first line of a multi-line function │
│ 199 │ # calls in the traceback and some print out the last line │
│ ❱ 200 │ Variable.execution_engine.run_backward( # Calls into the C++ engine to r │
│ 201 │ │ tensors, grad_tensors, retain_graph, create_graph, inputs, │
│ 202 │ │ allow_unreachable=True, accumulate_grad=True) # Calls into the C++ en │
│ 203 │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/torch/autograd/function.py:274 in │
│ apply │
│ │
│ 271 │ │ │ │ │ │ │ "Function is not allowed. You should only imple │
│ 272 │ │ │ │ │ │ │ "of them.") │
│ 273 │ │ user_fn = vjp_fn if vjp_fn is not Function.vjp else backward_fn │
│ ❱ 274 │ │ return user_fn(self, *args) │
│ 275 │ │
│ 276 │ def apply_jvp(self, *args): │
│ 277 │ │ # _forward_cls is defined by derived class │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/torch/utils/checkpoint.py:157 in │
│ backward │
│ │
│ 154 │ │ │ raise RuntimeError( │
│ 155 │ │ │ │ "none of output has requires_grad=True," │
│ 156 │ │ │ │ " this checkpoint() is not necessary") │
│ ❱ 157 │ │ torch.autograd.backward(outputs_with_grad, args_with_grad) │
│ 158 │ │ grads = tuple(inp.grad if isinstance(inp, torch.Tensor) else None │
│ 159 │ │ │ │ │ for inp in detached_inputs) │
│ 160 │
│ │
│ /usr/local/python3.8/lib/python3.8/site-packages/torch/autograd/init.py:200 in │
│ backward │
│ │
│ 197 │ # The reason we repeat same the comment below is that │
│ 198 │ # some Python versions print out the first line of a multi-line function │
│ 199 │ # calls in the traceback and some print out the last line │
│ ❱ 200 │ Variable.execution_engine.run_backward( # Calls into the C++ engine to r │
│ 201 │ │ tensors, grad_tensors, retain_graph, create_graph, inputs, │
│ 202 │ │ allow_unreachable=True, accumulate_grad=True) # Calls into the C++ en │
│ 203 │
╰──────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Expected is_sm80 to be true, but got false. (Could this error message be
improved? If so, please report an enhancement request to PyTorch.)

这个报错是为什么？

[beyondguo]

pytorch版本？

[Qiang-HU]

对，先用的torch2.0版本的，后面降到1.13.1版本就行了

[beyondguo]

奇怪，但我使用的torch2.0

[natureLanguageQing]

遇到了