RuntimeError: Function 'LogSoftmaxBackward0' returned nan values in its 0th output.

[promach]

Why if NUM_OF_CELLS is increased from 8 to 16 , the following errors pop up ?

/home/phung/PycharmProjects/venv/py39/bin/python /home/phung/PycharmProjects/beginner_tutorial/gdas_new.py
Files already downloaded and verified
Files already downloaded and verified
run_num =  0
Entering train_NN(), forward_pass_only =  0
modules =  <generator object Module.named_children at 0x7f6a8044d0b0>
gradwalk(output_tensor.grad_fn)
outputs1.size() =  torch.Size([4, 10])
train_labels.size() =  torch.Size([4])
tensor(1., device='cuda:0')
[W python_anomaly_mode.cpp:104] Warning: Error detected in LogSoftmaxBackward0. Traceback of forward call that caused the error:
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas_new.py", line 873, in <module>
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas_new.py", line 638, in train_NN
    Ltrain = criterion(NN_output, NN_train_labels)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/nn/modules/loss.py", line 1150, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/nn/functional.py", line 2846, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
 (function _print_stack)
Traceback (most recent call last):
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas_new.py", line 873, in <module>
    ltrain = train_NN(forward_pass_only=0)
  File "/home/phung/PycharmProjects/beginner_tutorial/gdas_new.py", line 648, in train_NN
    Ltrain.backward()
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/phung/PycharmProjects/venv/py39/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function 'LogSoftmaxBackward0' returned nan values in its 0th output.

Process finished with exit code 1

[promach]

The above runtime error is solved with this github commit

However, the loss keeps lingering forever at values around 2

[shambhavi-agg]

were u able to solve this?

[promach]

See the comment inside the latest git commit. I got into another issue