IPSec for LAN traffic: Basic considerations?

• Is there good cross-platform support? It must work on Linux, MacOS X and Windows clients, Linux servers, and it shouldn't require expensive network hardware.

I don't really have much experience with this, as I mainly have Linux systems, but I did get it mostly working on a Windows 2000 machine (this was some time ago). It had a problem that the IPsec failed to renegotiate a new session key after some number of bytes had been transferred (this is supposed to happen automatically), so the connection went down after a while, and I could never be bothered to dig into it further. It probably works much better nowadays.

• Can I enable IPSec for an entire machine (so there can be no other traffic incoming/outgoing), or for a network interface, or is it determined by firewall settings for individual ports/...?

How it works is (or, rather, how I managed to get it working), you define that a machine foo must use only IPsec to machines bar, baz, and yow. Any traffic from and to these machines is now secure and as trustworthy as those machines are. Any other traffic is not IPsec and works normally.

• Can I easily ban non-IPSec IP packets? And also "Mallory's evil" IPSec traffic that is signed by some key, but not ours? My ideal conception is to make it impossible to have any such IP traffic on the LAN.

IPsec traffic is only allowed for those IPsec "policies" that you define, so any random machine cannot send IPsec packet - there must exist an IPsec policy matching those packets.

• For LAN-internal traffic: I would choose "ESP with authentication (no AH)", AES-256, in "Transport mode". Is this a reasonable decision?

Yep. There is talk about abandoning AH completely because it's redundant - you can use ESP with NULL encryption with the same effect.

• For LAN-Internet traffic: How would it work with the internet gateway? Would I use
  • "Tunnel mode" to create an IPSec tunnel from each machine to the gateway? Or could I also use

I would choose this option. As it is I don't control the gateway myself, and the traffic will be unencrypted outside my network anyway, so I don't really see a pressing need.

Internet traffic to hosts which does not use IPsec must be seen as possibly being intercepted - there's little point in encrypting on the local LAN when your ISP or your ISP's ISP can listen to the same packets unencrypted.

• "Transport mode" to the gateway? The reason I ask is, that the gateway would have to be able to decrypt packages coming from the LAN, so it will need the keys to do that. Is that possible, if the destination address isn't the gateway's address? Or would I have to use a proxy in this case?

As I understand it, that does not work - you would need a proxy.

• Is there anything else I should consider?

See if you can use something sensible like OpenPGP keys instead of X.509 certificates. I use X.509 since that was the only thing supported by the IPsec keying daemon I first used, and I haven't had the energy to look into redoing it all. But I should, and I will, someday.

P.S. Me and an associate held a lecture on IPsec in 2007, it may be of help to clarify some concepts.