How do you keep SourceTree/SSH from forgetting your SSH keys? (I have to manually re-run 'ssh-add' to get it to work again!)

Ok, I think I have all the parts figured out.

To help people get what they're after, here's the solution right up front:

1. Make sure the keys you want to work with are secured with a password or else they will not add to Keychain.
2. Make sure the keys you want to auto-load are configured in your config file and have the UseKeychain and AddKeysToAgent set
3. Make sure to connect to those config-defined hosts from terminal!!
4. Create a LaunchAgent to run ssh-add -A to automatically reload your Keychain-stored keys

Ok now that you know what to do, here's the 'why'.

The Meat

As explained in my question, lately, whenever I rebooted, I (incorrectly) thought the system was losing my private keys. It wasn't losing them, it was just ignoring them. This was because of a bunch of things that all came together in a perfect storm of confusion for someone like me who never uses the terminal for GIT.

1. In the latest versions of macOS, Apple changed how it's implemented SSH so that It better matches the implementation of OpenSSH
2. As a result of #1, ssh-add -K [privateKey] no longer stores the keys in the keychain (it essentially ignores the -K.) While they do get added to ssh for that session--and thus your connections will work again--as soon as you reboot, they will no longer work. (This is what's been driving me mad!)
3. Even for keys that are in the Keychain, Apple no longer loads them automatically meaning you manually have to call ssh-add -A from the terminal to reload them every time you reboot.
4. However, as stated above, ssh-add -K [privateKey] no longer adds the keys to keychain, so ssh-add -A is pointless anyway for keys added that way. (They can be added to Keychain another way. More on that in a minute.)

Because of the above, any keys manually added with the -K option prior to upgrading your OS will still be in your Keychain. However, keys added after Apple's change are not.

That said, Apple does still have the ability to store keys in the keychain, but not from ssh-add anymore. It now only works for hosts defined in your config file.

This is now the only way to get your keys in your Keychain.

Again, here's my config:

Host MarqueIV-Bitbucket
    HostName bitbucket.org
    User git <-- Make sure this is 'git', not what SourceTree puts here
    PreferredAuthentications publickey
    IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
    UseKeychain yes    <-- Note here
    AddKeysToAgent yes <-- ...and here

But wait! If you look in my config file, it does have those values set! So why didn't it work?

Two things.

1. I don't use Terminal, ever. I use SourceTree which doesn't use the host entry in that file
2. Apple technically only adds (and stores) the key on demand when that host is accessed, not when the file is (re)loaded meaning unless you explicitly access that host, nothing happens.

In my case, adding the keys via SourceTree would add them for that initial session, but as soon as I rebooted, the keys would again not be loaded and thus all connections would fail. ssh-add -A wouldn't fix it either because again, they weren't in the keychain, meaning I was back to manually adding each one on the command line with ssh-add [privateKey]. What a pain!!

Then it occurred to me... if that setting is in the config file, and that entry can be used from the command line, then shouldn't I be able to directly connect to that host, thus adding the keys to my keychain? Let's find out! I typed this...

ssh -T MarqueIV-BitBucket

And sure enough, not only was the key added to ssh, but it was also again added to my Keychain! I confirmed this by checking Keychain Access directly and it was there.

To further test, I ran this...

ssh-add -D

which deleted all my keys. Sure enough, my SourceTree connections all failed again.

Then I ran this...

ssh-add -A

and the keychain-stored keys magically came back and connections started working again! WOOT!!

Ok, almost there, but not quite! What about reboots? Again, Apple no longer automatically loads keys from Keychain. Sure, it's just a quick jaunt now to terminal to type ssh-add -A, but again, I shouldn't have to do that!

Enter LaunchAgents!

LaunchAgents and LaunchDaemons are beyond the discussion of this post, but in short, they allow you to execute something on reboot, on a schedule, when changes happen to the system, etc.

In my case, I wanted something that would run when I logged onto my mac, so a LaunchAgent was the best choice.

Here's my plist defining how to execute ssh-add -A every time I logged into my account (even if I never touched Terminal):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>ssh-add-a</string>
    <key>ProgramArguments</key>
    <array>
        <string>ssh-add</string>
        <string>-A</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Since I only want this for my particular user, I stored it here:

~/Library/LaunchAgents

Note: Make sure to change the permissions to allow it to be executed, or it won't start!

Sure enough, on reboot, all my keys came back and were active! Connections all worked, children played, grown men cried, and it was a good day in the Code-dom!

So to recap:

1. Apple changed how their SSH worked
2. Keys were no longer added to Keychain from the command line
3. Apple also no longer auto-loaded keys that were stored in the keychain
4. Using terminal to connect to config-defined hosts fixed #2
5. Using a LaunchAgent fixed #3

Hope this helps! Now time to go get some Icy-Hot for my sore shoulder that I've been patting myself on so hard for figuring this all out! ;)