Can I use a private key as a public key and vice versa?

But say I want the server to encrypt data using the private key and decrypt it using the public key, as a way of distributing data that can be verified to have come from the right server.

You can do this - this is, at a very simplistic level, how RSA signing works (note, simplistic - there is a bit more to it).

Rather than modify the code to allow this, can I simply publish the private key and keep the public key secret? Does this affect the security of the system?

You don't need to publish the private key at all - RSA is a trapdoor permutation which means:

• If you encrypt with a public key, you can decrypt with the private key.
• If you encrypt with a private key, you can decrypt with a public key.

Thus, RSA supports doing both signing and encryption relying on the end user having only the public key.

In your case, if the client wishes to verify data came from the server, you apply the second case of RSA and decrypt the signature data using the public key you already have.

Furthermore, because it is a permutation, you shouldn't need to modify your code at all. Both keys should work using the same function. I would expect any decent crypto library would have APIs for verifying signatures according to the varying standards that exist - one of these would probably be a good bet.

RSA Labs provide a nice explanation of this.

If you want to extend this between servers, or verify client communication - generate keys for each entity and swap the public ones. The process can then be used at both ends.

Theoretically speaking, e and d are interchangeable (which is why RSA works)(one must be designated secret and kept secret) but p and q must always be kept secret as these allow you to derive d from e and vice versa. However, you need to be extremely careful in your understanding of the private key - does your software store p/q in the private key? If so, you can't publish it as is. Also, when I say interchangeable - once you publish one of that pair (e or d along with your modulus n) you must guard the other with your life. Practically speaking as Graeme linked to in the comments e is often chosed as a small/fixed value. My comment on e/d being interchangeable clearly does not apply when e is easily determined. Doing this sort of thing therefore has the potential for confusion and mis-implementation. Use a third-party library/don't start publishing private keys.

Yes you can encrypt using a private key and decrypt using a public key, do NOT hand out your private key (private keys can generate public keys, and your entire encryption system is now useless), I'm assuming you want to do something like this:

Server encrypts data, sends it to clients.

Clients have public key, and can decrypt data from the server.

Under this circumstance, it would make sense, if the server is doing both the encryption and decryption, I have no clue why you'd want an asymmetric crypto implementation.

The rule always applies though: private key stays private and doesn't leave the server.

But say I want the server to encrypt data using the private key and decrypt it using the public key:

The server can encrypt data using its private key and the receiver can decrypt using the servers public keys, this is possible. However, this scenario is used to digitally sign a document and as you say its "a way of distributing data that can be verified to have come from the right server", as only the corresponding key (public key of the server) could be used to decrypt the data. The public key of the server is verified by the corresponding digital certificate generated and digitally signed by the server.

can I simply publish the private key and keep the public key secret?

You should never publish the private key, as the name suggests.

Does this affect the security of the system?

Yes, also this system is not viable, for example, if you are using Diffie-Hellman (one of the most widely used key exchange protocol), the server and all the clients have to share public values, that are used to calculate session keys. So if you as a user publish your private key, it's very easy to calculate public key, which is not possible or very difficult the other way round.