How can I stop ssh from trying to create ~/.ssh?

The solution mentioned here is good, but I was looking for one which didn't require ~/.ssh to be present or required an attempt for it to be created (running on Windows, distributing a MSYS built ssh.exe, but running on other machines without MSYS/Cygwin installed).

Unfortunately, it appears the routine is hardcoded in the source code, regardless of what value you give to UserKnownHostsFile:

    r = snprintf(buf, sizeof buf, "%s%s%s", pw->pw_dir,
        strcmp(pw->pw_dir, "/") ? "/" : "", _PATH_SSH_USER_DIR);
    if (r > 0 && (size_t)r < sizeof(buf) && stat(buf, &st) < 0) {
#ifdef WITH_SELINUX
        ssh_selinux_setfscreatecon(buf);
#endif
        if (mkdir(buf, 0700) < 0)
            error("Could not create directory '%.200s'.",
                buf);
#ifdef WITH_SELINUX
        ssh_selinux_setfscreatecon(NULL);
#endif
    }

However, it's interesting to note that it attempts to expand the $HOME environment variable to determine the home directory. Since the buffer is around 256 bytes, we can actually bypass the if condition by overflowing the buffer (defining HOME to a string longer than 256 bytes), e.g.:

export HOME=$HOME/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.

Remount the root filesystem read-write temporarily and make a symbolic link for .ssh that points somewhere where ssh can write. This way you can also do things like add ssh keys in the future, or allow new known_hosts without having to go to extra strenuous steps. And you could get rid of your -o UserKnownHostsFile option at the same time.