Reply on same interface as incoming?

echo 200 isp2 >> /etc/iproute2/rt_tables
ip rule add from <interface_IP> table isp2 prio 1
ip route add default via <gateway_IP> dev <interface> table isp2

The above doesn't require any packet marking with ipfilter. It works because the outgoing (reply) packets will have the IP address that was originally used to connect to the 2nd interface as the source (from) address on the outgoing packet.

The following commands create an alternate routing table via eth1 for packets that have the mark 1 (except packets to localhost). The ip command is from the iproute2 suite (Ubuntu: iproute Install iproute http://bit.ly/software-small, iproute-doc Install iproute-doc http://bit.ly/software-small).

ip rule add fwmark 1 table 1
ip route add 127.0.0.0/0 table 1 dev lo
ip route add 0.0.0.0/0 table 1 dev eth1

The other half of the job is recognizing packets that must get the mark 1; then use iptables -t mangle -A OUTPUT … -j MARK --set-mark 1 on these packets to have them routed through routing table 1. I think the following should do it (replace 1.2.3.4 by the address of the non-default-route interface):

iptables -t mangle -A OUTPUT -m conntrack --ctorigdst 1.2.3.4 -j MARK --set-mark 1

I'm not sure if that's enough, maybe another rule is needed on the incoming packets to tell the conntrack module to track them.

I had issues with the locally generated packets with the solution suggested by Peter, I've found that the following corrects that:

echo 200 isp2 >> /etc/iproute2/rt_tables
ip rule add from <interface_IP> table isp2 priority 900
ip rule add from dev <interface> table isp2 priority 1000
ip route add default via <gateway_IP> dev <interface> table isp2
ip route add <interface_prefix> dev <interface> proto static scope link src <interface_IP> table isp2

NOTE: You may run into syntax issues with the 4th line above. In such cases the syntax for the 4th command may be this now:

ip rule add iif <interface> table isp2 priority 1000