Direct ethernet link between two servers
Solution 1:
There's no reason why you technically can't do this.
I'd probably do something similar, under the circumstances, actually. From a purely linux point of view, it's really easy, just give the connection an IP address with a /30 bitmask, giving you 2 IP addresses, then it's a simple Point-to-Point link.
If you wanted to grow the network, you could get a 10GE switch, and then have a seperate VLAN for traffic between servers. There's some very shiny gear in the Force10 range of switches that can do line-rate 10GE switching, with enormous buffers.
Solution 2:
I cant comment a Linux point of view but I will just use my knowledge and ask some more questions.
Are you that dependant on low latency and that you need to keep these servers in sync? Are they both running a database or something? 10GbE should suffice most needs of 2 servers to keep them in sync. I would soon rather spend the money on a decent switch in the middle of the 2 instead of going the route you are looking at.
You could with a decent switch tag these ports to prioritise the traffic and even QoS the traffic that needs to be real time.
My thoughts.
Solution 3:
I have actually done this between two laptops. Most modern LAN adapters have auto negotiation between them so you can use a regular lan cable.
Set static IP addresses that are not in the same range as any other subnet you are using - for example, if my systems are on a 192.168.x.x subnet, i use a 10.0.0.x subnet between them. Otherwise, it should just work
Solution 4:
Security vs. Performance vs. money.
If the back channel traffic is high and money is low, use a direct connection. It's done everyday and in many real world situations can have better performance than an already overloaded switch.
If the back channel traffic is low and security is medium or low, bond NICs to increase overall Internet throughput - two connections from each server to the internet, multi-home NICS to "isolate" replication traffic (Separate IP spaces makes it easier to firewall, audit, do packet trace diagnostics, etc.).
If security is high and plenty of money use a switch. Easier to expand. Easier to diagnose problems.
In the given scenario a switch purchase would not be warranted. Utilizing an existing switch with VLAN segmentation would possible make sense. Although, I can't see any reason to plug into the switch unless the servers are Co-lo'd, i.e., not physically accessible. It's a waste of two switch ports unless packet capture/debugging is active.
Solution 5:
There is no advantage of using such setup. the switches today are lightening fast, so you never face any visible latency due to switch. and scalability would be a big issue for you as well. Also, there would be problem of setup of routing as well, as you will have to maintain TWO Separate networks instead of just one.