Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED code example

Example 1: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

$ export TF_FORCE_GPU_ALLOW_GROWTH=true

Example 2: could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

physical_devices = tf.config.experimental.list_physical_devices('GPU')
assert len(physical_devices) > 0, "Not enough GPU hardware devices available"
config = tf.config.experimental.set_memory_growth(physical_devices[0], True)

Example 3: cudnn error: cudnn_status_not_initialized

Finally, I solved the problem.

1.Somewhere in my code I use a CrossEntropyLoss function with ignore_index parameter as ignore_index = my_ignore_index. By mistake, I had my_ignore_index = -1 which as value, it is not a valid value for my data; -1 never appears in my data values. Updating correctly solved the problem. This solved the "... an illegal memory access was encou..." error.

2.The other thing that I did and helped to solve the problem was to use a newer version of anaconda3. This solved the CUDNN_STATUS_NOT_INITIALIZED error.