Separate Network Traffic on Two Network Interfaces

I'd love to know more about this topic to refine the configuration to be the best that it can be, but here's what I have so far. Even without enabling ARP filtering on all network interfaces (net.ipv4.conf.all.arp_filter = 0), as mentioned by @spuk, traffic seems to be completely separated in this configuration.

The file, /etc/iproute2/rt_tables, is the same in EL 6.x and DEB 7/8, at least. This is the file that creates a named routing table for static routes.

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
252 mgmt

Above, the number of the named, static route, 252, is essentially arbitrary; or, each static route gets its own unique number between 1 and 252.

The file, /etc/network/interfaces in DEB 7/8, at least:

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
  iface lo inet loopback

# The production network interface
# The 'gateway' directive is the default route.
# Were eth0 configured via DHCP, the default route would also be here.
auto eth0
allow-hotplug eth0
iface eth0 inet static
  address 10.10.10.140
  netmask 255.255.255.0
  gateway 10.10.10.1

# The management network interface
# The 'gateway' directive cannot be used again because there can be
# one, and only one, default route. Instead, the 'post-up' directives
# use the `mgmt` static route.
auto eth1
allow-hotplug eth1
iface eth1 inet static
  address 192.168.100.140
  netmask 255.255.255.0
  post-up ip route add 192.168.100.0/24 dev eth1 src 192.168.100.140 table mgmt
  post-up ip route add default via 192.168.100.1 dev eth1 table mgmt
  post-up ip rule add from 192.168.100.140/32 table mgmt
  post-up ip rule add to 192.168.100.140/32 table mgmt

The result of ip route show on Debian:

default via 10.10.10.1 dev eth0
10.10.10.0/24 dev eth0  proto kernel  scope link  src 10.10.10.140
192.168.100.0/24 dev eth1  proto kernel  scope link  src 192.168.100.140

The EL 6.x /etc/sysconfig/network file:

NETWORKING=yes
HOSTNAME=localhost.localdomain
GATEWAY=10.10.10.1

Above, GATEWAY is the default route. Below, were BOOTPROTOCOL set to DHCP, the default route would be acquired from DHCP.

THE EL 6.x /etc/sysconfig/network-scripts/ifcfg-eth0 file, without "HWADDR" and "UUID":

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTOCOL=none
IPADDR=10.10.10.140
NETMASK=255.255.255.0
NETWORK=10.10.10.0
BROADCAST=10.10.10.255

THE EL 6.x /etc/sysconfig/network-scripts/ifcfg-eth1 file, without "HWADDR" and "UUID":

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTOCOL=none
IPADDR=192.168.100.140
NETMASK=255.255.255.0
NETWORK=192.168.100.0
BROADCAST=192.168.100.255

The EL 6.x /etc/sysconfig/network-scripts/route-eth1 file:

192.168.100.0/24 dev eth1 table mgmt
default via 192.168.100.1 dev eth1 table mgmt

The EL 6.x /etc/sysconfig/network-scripts/rule-eth1 file:

from 192.168.100.0/24 lookup mgmt

The result of ip route show on EL 6.x:

192.168.100.0/24 dev eth1  proto kernel  scope link  src 192.168.100.160
10.10.10.0/24 dev eth0  proto kernel  scope link  src 10.10.10.160
default via 10.10.10.1 dev eth0

Update for RHEL8

This method described above works with RHEL 6 & RHEL 7 as well as the derivatives, but for RHEL 8 and derivatives, one must first install network-scripts to use the method described above.

dnf install network-scripts

The installation produces a warning that network-scripts will be removed in one of the next major releases of RHEL and that NetworkManager provides ifup/ifdown scripts as well.

Update for Ubuntu 20.04 LTS

Creating a named routing table is ok, but not required with netplan, which will not use the name anyway. Nonetheless the number of the named routing table from the rt_tables file can be used for netplan. Corresponding NICs are enps03 (eth0) and enp0s8 (eth1).

network:
  version: 2
  ethernets:
    enp0s3:
      addresses:
      - 10.10.10.140/24
      dhcp4: false
      dhcp6: false
      gateway4: 10.10.10.1
      nameservers:
        addresses:
        - 1.2.3.4
        - 1.2.3.5
        search:
        - your-search-domain-name.com
    enp0s8:
      dhcp4: false
      dhcp6: false
      addresses:
      - 192.168.100.140/24
      routes:
      - to: 192.168.100.0/24
        via: 192.168.100.1
        table: 252
      routing-policy:
      - from: 192.168.100.0/24
        table: 252

This results in the following routes from ip r s.

default via 10.10.10.1 dev enp0s3 proto static
10.10.10.0/24 dev enp0s3 proto kernel scope link src 10.10.10.140
192.168.100.0/24 dev enp0s8 proto kernel scope link src 192.168.100.140

I haven't read throughly all your post (sorry, can't really spend the time right now), but I believe it may be related to the way Linux implements the IP host model:

... The IPv4 implementation in Linux defaults to the weak host model. ...

From that same page:

... If the IP stack is implemented with a weak host model, it accepts any locally destined packet regardless of the network interface on which the packet was received. ...

That is, in Linux, by default, the IP addresses "belong to the host", not strictly "to the interface". You can change that behavior via the arp_filter, rp_filter, arp_announce, arp_ignore sysctls (got from LVS: The ARP Problem, seen here). Also, see ip-sysctl.txt.