How to properly encrypt a communication channel between a client and a server (without SSL)?

This is actually simpler than the others have suggested I think. Here is what I propose:

1. Generate a public/private key pair using RSA. You can do this on any unix machine using openssl: openssl genrsa -out rsa.private 2048
2. Distribute the public key as hardcoded in your client.
3. When the client logs in, the client generates a private shared key (also using openssl, or another well known encryption library) and encrypts it using the public key.
4. all data between client and server should be encrypted with the new shared key for the length of the session.

There is a lot of other stuff you can do to harden it, such as limiting sesison length to one or a few hours before re-generating keys, having the server sign message digests, etc. A full understanding of encryption will help you understand what those are and why, but for a basic implementation (for medium to low value data) this should be sufficient.

Before coding, read this: http://www.schneier.com/book-applied.html

This question is too broad to fit on StackExchange unfortunately. The length and breadth of cryptography is rarely traversed by a single individual with any degree of success without formal education in mathematics, computer science, and a smattering of engineering.

This book will help you learn common concepts, common mistakes, fark-ups to learn from, and examples to follow.