Is it possible to use SSH agent for generic data encryption?
No, it is not at least with OpenSSH -- there are no options to operate in this mode. It is for authenticating with remote servers (validating identity) and then encrypting the network traffic between ssh client application and the remote sshd server.
You could try compiling a library from OpenSSH that uses its encryption modes and create an executable based on that to use encryption, but the authors of OpenSSH did not try to do that. Here's a github repo that took openssh and tried converting it into a library that other applications can use.
OpenSSH uses well-known protocols that are implemented in OpenSSL; e.g., my ssh connection right now uses aes128-ctr to encrypt with hmac-md5 to verify integrity (the combination). Granted if you actually trace through to see where aes128-ctr comes from, you'll find it ultimately is taken from the OpenSSL library. (That is in libopenssh/cipher.c tells aes128_ctr to use EVP_aes_128_ctr which is going through the header file #include "cipher.h" (libOpenSSH which includes #include <openssl/evp.h> (note OpenSSL) which defines EVP_aes_128_ctr that ultimately traces back to being implemented (after some wrappers to deal with different modes/sizes) in openssl/crypto/aes/aes_core.c.
EDIT:
Sorry wasn't clear to me you were talking about ssh-agent.
As to using the private keys loaded into ssh-agent to encrypt or decrypt messages, no that is not possible. The entire purpose of the ssh-agent is to limit use of the private key to only the one acceptable function (authentication in ssh).
Look at the source code of ssh-agent and the description of the functionality in PROTOCOL.agent. For security, ssh-agent provides a very limited interface to using the stored ssh private keys.
For ssh2, the only use of the private key is to sign requests using your private key. With ssh1 there's also process authentication challenges, but you really shouldn't use ssh1 anymore. The other functions don't use the private key, but deal with key management.
Key challenges in SSH1 worked as fellows. First, the remote end sends a challenge -- a random bit-string encrypted with the user's public key (known from authorized_keys file). The user sends the challenge to ssh-agent, which uses the private key to decrypt the challenge message. Then, ssh-agent takes the random bit-string and calculates a cryptographic hash of it, and sends back the cryptographic hash. This verifies that the user was in possession of the private key, and only leaks the hash of the decrypted message -- but not the actual decrypted message.
Key challenges in SSH2 now work by sending signing requests. Again, the remote server sends a random bit-string to the user. Then ssh-agent takes that random-bit-string and uses their private key to cryptographically sign that message -- no decryption is performed. The signature is sent back and the server verifies the signature by using the public key to validate it.
In the context of RSA, the public key is (N = p*q, e) and the private key (N, d), where d is constructed so d*e ≡ 1 (mod (p-1)(q-1)) (d can be calculated efficiently knowing e, p and q via Euclid's extended algorithm). Then the method of signing is to hash (with SHA1) the random message h = sha1(r) and then raise the hash to a power with the private key S = h^d (mod N). Then the server validates the signature S with the user's public key by calculating S^e (mod N) which due to Euler's totient theorem will be the hash of the sent message, which it will compare it against S^e ≡ (h^d)^e ≡ h^(d*e) ≡ h^(1 + (q-1)(p-1)) ≡ h (mod N).
It is not possible to use the SSH Agent to asymmetrically decrypt data that was encrypted with the public key because the SSH Agent only exposes the ability to sign using the private key, not decrypt.
You can however use the fact that the private key in the SSH Agent is the only one who can generate a signature for something, and use that signature as a symmetric key.
I made sshcrypt to do this.