Which is more secure Yubikey + Keepass using Challenge/Response or Yubikey + Keepass using OTP?

In my opinion "Keepass 2 using OTP" can not be "secure". Why?

OTP can not be used to encrypt data

Keepass encrypts data and the problem with encrypted data is, that you encrypt the data because you want to protect it against offline attacks. S.o. gets access to the data in the encrypted form and you want to avoid that the attacker can read it in clear text. But if the attacker has the encrypted data, he can run offline brute force attacks. And HOTP, creating 6 or 8 digit numbers do not realy add a big deal to protecting from offline attacks. It simply can add 6 digits to your password! This is fine for online authentication, when the account is blocked after a certain amount of failed tries. But there is no sense in using OTP to derivate encryption keys from.

The thing with OTP is, it is so short because it can be easily used for authentication, because everybody can type it in everywhere. This is cool - but only for the intended use cases!

I would not recommend to use OTP for keepass!

Using Challenge Response

Using challenge response does not have this limitation. Don't get me wrong, OTP has it's right to exist! And the problem with challenge response is: you need drivers! So there are authentication scenarios where OTP is better (not more secure!) than challenge response.

But back to keepass 2 and the dirvers. Well, with keepass this is no problem, because you obviously have them anyways. The Challenge Response mode of the Yubikey uses a symmetric secret key. You can send a challenge to the yubikey, it will create an hmac from the challenge and the secret key and respond with a 256bit return value. Now, you need the same challenge and the same secret key to always create the same return value. And obviously the return value is used as the encryption key (well actually the secret key is used as encryption key, but it is stored encrypted with the response value) But bottomline, you need the secret key to decrypt the data. As we are talking different key length here (than a 6 digit number), the secret key is as difficult to brute force as the AES encryption key itself. Read here: http://richardbenjaminrush.com/keechallenge/#using

I would recommend using Challenge/Response!

Disclaimer: Of course I did no code review or any in depth analysis. This is simply an evaluation from the basic official concepts of the two mechanism!

One important difference between OTPs and challenge/response schemes is that the first can be spoofed, and the second cannot. Suppose somebody sets up a phishing UI for KeePass, and you enter your password into it without realizing that it's not the legit one. The attacker realizes that the password isn't enough, you have MFA enabled. So far, so good.

If you use OTP, though, all the attacker needs to do is show the usual OTP entry box. You tap your Yubikey, it sends the OTP to the attacker, attacker forwards it to KeePass, and boom they've got access to your KeePass vault. If you instead use Challenge/Response, then the Yubikey's response is based on the challenge from the app. The attacker doesn't know the correct challenge to send for KeePass, so they can't spoof it.

Note that this distinction probably doesn't matter that much for a thick-client local app like KeePass, but it definitely matters for anything that does remote authentication. For example, if you're logging into a web app like Gmail, a phishing page can spoof the login UI and the OTP UI just fine. However, a phishing page cannot spoof the challenge/response flow, because that's based on the actual site being visited, and your browser (and therefore the Yubikey) will send a different challenge even if it looks the same to you.

In fact, challenge-response systems like this are the only method I know of that provides pretty strong security against phishing attacks. Everything else involving a true second factor - OTP, SMS, even push notifications - can be spoofed, and you will take the approve the authentication check just like normal because, as far as you know, you are logging in just like normal.