Secure application data transmission via SMS channel

The TextSecure Messenger (now called Signal) provided encrypted sms based on a public key encryption scheme. However, they dropped the feature a year ago. According to the blogpost, the reasons for dropping the support were the impossibility to do this properly with iPhones and the bad security of SMS in general.

This is of course only one example - but it shows one possible approach to encrypt the content and protect the integrity of SMS messages.

I can't help you with the details of the cryptography involved - but maybe it's worth looking into the documentation and the sourcecode of Textsecure / Signal / Silence (Textsecure fork).

There is no secure data transmission over SMS. The data is exposed to a large variety of information disclosure threats from and through the telco, from tools which sync your SMS to email (such as Exchange and Google Voice).

As Lukas says, it is super-hard to encrypt your data and then send it via SMS. If you're going to build an app, then why use SMS as a transfer layer? If you're not going to build an app, then SMS is insecure.