How to set up GoCD to handle private git repositories that use SSH keys for authentication?

At the time of writing this answer, it's not possible to manage SSH keys in GoCD directly. To make it work you have to generate SSH keys for a GoCD server and all agents and then add them to the server that's hosting the git repository. You can also copy an existing key to the server & nodes but that's obviously not recommended.

For example, with standard GoCD server installation you should have the "go" user in your system:

$ grep GoCD /etc/passwd
go:x:998:998:GoCD User:/var/go:/bin/bash

sudo as a "go" user and create the key

$ sudo su - go
$ ssh-keygen
...
$ ssh [server]
The authenticity of host '[server] ([1.3.3.7])' can't be established.
ECDSA key fingerprint is SHA256:Rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[server]' (ECDSA) to the list of known hosts.
Permission denied (publickey,keyboard-interactive).

The last step is important because if you don't make it, GoCD will give you the same error.

Now add your key to the git server and click "Check connection" in GoCD. It should print "Connection OK.".

Generate keys for each node and user that runs an agent.