Correct way to provide a direct network link between 2 devices as well as connecting both to a switch
What is the correct way to configure the PC and Server, so that the loop doesn't cause networking issues? Meaning, traffic to/from other devices can only go via the switch as normal and neither device is seen as a bridge or route to the other?
Just connect the two using a cable and configure both ports to have IP addresses. This link should use a completely separate subnet from your main LAN, e.g. if the 1G switch carries 192.168.1.0/24 then the dedicated 10G link should be 192.168.2.0/24 (or even better, a /30).
Bridging will not occur unless you deliberately set up a bridge. Neither Windows nor FreeBSD (nor any other OS) enable it automatically.
Routing will not occur unless you deliberately set up packet forwarding. Neither Windows nor FreeBSD enable it automatically.
(Additionally, IPv4 uses explicitly configured routing tables, so even if the PC or the server did have routing enabled, routing still wouldn't occur because other hosts would have no way of knowing that they can use them as gateways. IPv6 would need a little more care to ensure that the machines do not send out Router Advertisements, but that's also disabled-by-default on both Windows and FreeBSD.)
Of course, if you don't want the PC to use the server as a gateway, then do not fill in the "Default gateway" field when configuring IP addresses on your 10Gbit NIC...
If in future I upgrade the switch to a fully managed switch that can handle access controls, such as Netgear ProSAFE, or I start to use VLANs on it, does the "correct answer" change?
No.
The main issue is that the network is DNS based. That's important because IP.addresses don't show up in Windows file explorer, only discovered host names do
This discovery isn't DNS based. It might be using LLMNR & WS-Discovery, or the older NetBIOS (if SMBv1 is allowed), or mDNS & DNS-SD (but Windows doesn't actually support that yet), or even LDAP (Active Directory), but it won't be using ordinary DNS.
An IP address can be used for a SMB file server and its shares, but it will have to be manually entered every time a file share is accessed, and many programs undortunately don't have scope to enter a manual path, its expected that all browseable devices will appear in a populated file explorer navigation tree, or descend from one of a list of discovered devices.
If you're otherwise okay with manually entering paths, then you can map those paths to a drive letter using "Map Network Drive" or net use. They will permanently show up under "This PC" subtree.
(And the 10G IPs will need to be static, as there's no DNS on that link/subnet).
DNS does not handle address allocation. I think you mean "DHCP"?
Yes, normally such point to point links don't have DHCP, but on the other hand, there is nothing preventing you from running a DHCP daemon on the server and allowing at least the PC to acquire its address dynamically. (But do it with care – do not advertise the 'Routers' DHCP option.)
or else the server's 10G NIC must bridge to the 1G NIC (gives one unique IP for the server, but breakes everything else!)
Bridging is not really a good solution even with STP to avoid loops. There is no way to tell STP to use a specific path only for accessing certain MAC addresses – either the link is active or it's not.
(There's a reason why they don't call switches "Ethernet routers".)
I guess the newer TRILL or 802.1aq "shortest path bridging" protocols would indeed work here (as they do route at MAC level), but it looks like it will still be years until they begin showing up in affordable consumer switches. (The PC and the server would need to participate too, and for Windows that's even more unlikely.) Until then it's STP all-or-nothing.
If it is then a solution might be: LAN subnet 10.0.0.0/8, server 1G IP allocated by DNS to say 10.0.0.1 server 10G static 10.0.0.2
The two links really need to be on different subnets, otherwise you'll need more things to override on the PC...
The problem with this approach is that when a host has multiple interfaces belonging to the same subnet, it will usually not attempt to guess which one to use for each individual destination IP address. It will just always prefer one interface for the whole subnet. Unless you manually add more-specific routes to override it per destination.
(Windows might actually do guess, though – I haven't checked recently. But I do believe it only applies this guessing to designated "link-local" address ranges.)
the PC hosts file is manually hacked to override DNS and use the 10G IP for the server and routing is set up that ensures the servers 1G IP isn't reachable from the PC
Aron's suggested route does not make the 1G IP address unreachable. Rather, it makes the 1G IP address reachable through the 10G link.
So really you don't need both – you only need one or the other.
Too much manual overrides set on individual machines. I don't know if that's "normal".
If you go with Aron's suggestion, there is only a single thing you need to override: the routing for the 1G addresses of both computers.
(It's unfortunately "normal" for hosts to not participate in IP routing protocols – Windows does not speak OSPF for allowing route costs to be determined automatically, and neither does your LAN router I'd suspect. Windows does speak RIPv2 but I'm not sure if that would be useful in this situation, or indeed if it wouldn't be more work than a static route override.)
Same IPs as above, but configure DNS to return both IPs to all devices. Non-PC LAN devices will fail to find 10.0.0.2 and eventually decide that 10.0.0.1 is the IP used. The PC gets firewalled so it drops all packets to 10.0.0.1, and thus decides that 10.0.0.2 is the only working IP (it may need a first hop hint). Workable as well?
This may work – again, the addresses need to be on different subnets though.
And if you're fine with manual firewall rules, then you should be fine with manual routes as well – they're the same amount of "overrides" and are a slightly better solution. (And aren't they literally the "first hop hint" that you speak of?)
- Question: No bridging or routing will occur by default. You have to actively enable it.
- Question: No.
Easiest configuration: Use a different subnet for the 10 GbE link. /30 will be enough. Some devices even work on a /31 subnet. Use a subnet with addresses that do not have to be accessible through your 1 GbE connection. With different subnets, no packets will randomly go the one or the other way.
As you will probably have DNS on your 1 GbE connection, the easiest way to prevent ambiguities would be to not use names for your 10 GbE subnet. As it is point-to-point anyway, just use IP addresses to configure services that should use the 10 GbE link.