Using RSA with 3DES instead of plain 3DES. Does it make sense?

They said encryption of the whole data would cost more and is expensive (in term of performance).

This is correct. RSA, an asymmetric key algorithm needs a lot more computing power/time to encrypt/decrypt data compared to a symmetric key algorithm like DES/3DES.

Asymmetric key algorithms are usually used as a means to communicate a key for use in another symmetric key algorithm for data transfer.

For more information, see: http://en.wikipedia.org/wiki/Hybrid_cryptosystem

Basic example taken from wikipedia:

To encrypt a message addressed to Alice in a hybrid cryptosystem, Bob does the following:

1) Obtains Alice's public key.

2) Generates a fresh symmetric key for the data encapsulation scheme.

3) Encrypts the message under the data encapsulation scheme, using the symmetric key just generated.

4) Encrypt the symmetric key under the key encapsulation scheme, using Alice's public key. Send both of these encryptions to Alice.

To decrypt this hybrid ciphertext, Alice does the following:

1) Uses her private key to decrypt the symmetric key contained in the key encapsulation segment.

2) Uses this symmetric key to decrypt the message contained in the data encapsulation segment.

Perhaps as an improvement, your company would consider using AES instead of 3DES?

Terry Chia gave a fine answer about how and why you combine RSA with a symmetric key encryption scheme and I agree you should do that.

However, one other thing you said in your question stood out to me so I wanted to address that. You said "It decrypts data packets with a public key that was given(entered) by someone in to the machine(where the decryption happened)." I'm not sure what you meant by that, but I'll answer as if you mean something that changes what the "right" answer is.

One of the toughest problems with all the encryption systems is keeping the keys secure. The best solution, used in high-security applications, is to store the keys in a specialized encryption/decryption unit that has electronic and physical security such that the key cannot be extracted from the device. Either the key is generated within the device from a cryptographically secure random number generator or it is programmed into the device in a secure way. After that, all encryption and decryption that needs to be done with that key is done by the device itself and the security of the key is then managed by physically securing the device.

So if you are using a device like that to manage the RSA public and private keys, then that dictates a lot of how you handle other aspects of the overall encryption system. And if that device can also manage 3DES keys and encryption, then that changes things, too. Plus you may be in a regulatory and/or systems integration environment that dictates the use of 3DES in certain ways. All of these factors need to be taken into account in designing the solution, so please fill us in if any of them apply in such a way as to make Terry's solution somewhat inapplicable to your situation.

In general, though, a hybrid crytposystem using RSA to exchange a 256 bit AES key that is newly generated for each communication by a cryptographically secure random number generator and then encrypting the rest of the communication using AES with that key would be an excellent solution. Please do take care to make sure that your key generation mechanism is cryptographically secure, though. Many, many key generation schemes have been broken, generally due to weaknesses in the random number generators. You need to use one that has been audited and certified for cryptographic use.