Can I define the letter case of a Domain Name?

No, it will not affect the final letter case format of your domain whatever letter case you choose while registering it.

The Domain Name System (DNS) names is case insensitive and you can not manipulate this by any mean when you register it or even define it to search engines for "good looking" purposes.

All cases (upper and lower) will be accepted when you type it on your browser or your FTP client software or your email client software or your links.

You can read all the Case Insensitivity Clarification for the DNS standards on Internet Engineering Task Force website

Whilst DNS itself is case-insensitive, at least one search engine allows you to declare preferred case in search results: Yandex.

The Case of the site name tool lets you set preferred capitalisation in certain cases. This can be accessed in Yandex.Webmaster under Appearance in search results > URL letter case.

I've set this for a few of my sites and it's quite nice to see.

Well technically you can register your domain in any case. But for the record, no matter how you type your domain in the browser it will transform to small case. This is because DNS is not case sensitive.

If it makes sense... www.YAhOO.com should be no different to www.yahoo.com or www.yAhOo.com... You can play around your browser's address bar and see what happen.

This link should be able to help you understand the concept further.