Is this RSA/AES combination good?
All of the weaknesses in your protocol can be summed up as "use SSL" or even "use SSL, dammit !".
In more details:
- All the protocol is of course vulnerable to impersonation, specifically the double impersonation that is also known as Man-in-the-Middle attack.
- Similarly, if any of potential attackers that can eavesdrop on the line decides to do a modification of the data in transit, then he can, and your client and server will be none the wiser.
- Experience shows that saying "encrypt all the data with that key" and then doing it correctly is awfully complex. SSL itself took almost 15 years to do that, and many implementations are still not up to it. Padding oracles, predictable IV, MAC verification timing, verified closure, protection against resequencing and replay of packets...
As an overall assessment: don't do that.
No this protocol isn't safe.
As already mentioned in the comments:
Don't roll your own crypto. Chances are you're getting it wrong. Especially if you admit that you don't really know what you're doing.
First, you seem to have standard problems for which there are already standard protocols, which have nice properties and security proofs. The protocol for the data transmission should be TLS v1.2 (or newer). You should use a library (like NSS, GnuTLS or OpenSSL for this). For password authentication the best choice you have is to use SRP or TLS-SRP. However if you really can't afford SRP (which should be your first choice) you can still follow this paper.
So far for the improvements.
Now for the flaws of your protocol:
- It's vulnerable to Man-in-the-middle attacks. If an attacker sits in-between the device and the server, he can fish the public key and replace it with his own. In response he'd know the AES key and could decrypt all the data.
- You don't mention integrity / authenticity of the data. It sounds like you want to use ECB mode for AES (which you really don't want to use), you'd be a lot better off with AES-GCM for the bulk-encryption. You also don't mention how RSA is to be used, RSA-OAEP should be your choice (and not textbook RSA). However I assume you'd have done this with your implementation.
- You put a really large workload on the device. The device needs to generate a new private RSA key with every connection, which is a highly computing-intense task. You should rather consider using ECDH key exchange.
- For the verification of the passwords to work, you either need the server to store the passwords in plaintext or to hash them himself, which may enable DoS attacks on the server. And in case a man-in-the-middle attack takes place, the plain passwords will be revealed to an attacker, which may cause serious harm to your users (as they may re-use their passwords across sites...)
Device generates private RSA key, and sends the public key to server.
Server generates unique to user AES key and uses the RSA public key to encrypt it and send it back to device.
Device gets the AES key. Uses it to encrypt password and username and sends it to the server.
Anonymous key exchange.
There is no authentication. Your key exchange is encrypted, but not authenticated. This is bad.
If there is an active Man-in-the-Middle, then you have no way of detecting this. This active MitM can just pretend to the client that he is the server. And pretend to the server, that he is the client.
-> Don't do it. @SEJPM is right with his comment. Use established crypto systems. Like TLS with certificates or TLS with SRP.