After doing a normal configuration of just eth0, I came back around and added config for eth1. With just eth0 up, the route table was:
# ip route show
192.168.0.96/27 dev eth0 proto kernel scope link src 192.168.0.126
default via 192.168.0.97 dev eth0 metric 100
But once I brought up eth1, the order of the default route statements determined which interface was always used. As shown below, it happens to choose the eth1 route to the 192.168.1.65 gateway.
# ip route show
192.168.0.96/27 dev eth0 proto kernel scope link src 192.168.0.126
192.168.1.64/27 dev eth1 proto kernel scope link src 192.168.1.93
default via 192.168.1.65 dev eth1 metric 100
default via 192.168.0.97 dev eth0 metric 100
only one gateway statement
The first problem can be the extra 'via 192.168.1.65' default route. It appears there if the eth1 definition in /etc/network/interfaces has a "gateway 192.168.1.65" statement. So remove any extra gateway statement, and bounce the interface:
# ifdown eth1
# ifup eth1
# ip route show
192.168.0.96/27 dev eth0 proto kernel scope link src 192.168.0.126
192.168.1.64/27 dev eth1 proto kernel scope link src 192.168.1.93
default via 192.168.0.97 dev eth0 metric 100
setup new routing table
Create a new, seperate routing table containing a default route appropriate for all traffic flowing out of eth1. The table number here is not important; 101 is simply not the main routing table. Do this with a 'post-up' command on the eth1 configuration in /etc/network/interfaces. Add only one post-up on eth1; Do not add it to any of the eth1: sub-interfaces.
post-up ip route add default via 192.168.1.65 dev eth1 table 101
Bounce eth1. The main routing table is unchanged, and table 101 will contain the via 192.168.1.65 default route if eth1 is up.
# ifdown eth1
# ip route show
192.168.0.96/27 dev eth0 proto kernel scope link src 192.168.0.126
default via 192.168.0.97 dev eth0 metric 100
# ip route show table 101 (ie, table is empty, no output)
# ifup eth1
# ip route show
192.168.0.96/27 dev eth0 proto kernel scope link src 192.168.0.126
192.168.1.64/27 dev eth1 proto kernel scope link src 192.168.1.93
default via 192.168.0.97 dev eth0 metric 100
# ip route show table 101
default via 192.168.1.65 dev eth1
new routing rule
Add a routing rule to use table 101 to select a default route for packets which should go out eth1.
# ip rule add from 192.168.1.64/27 lookup 101
# ip rule show
0: from all lookup local
32765: from 192.168.1.64/27 lookup 101
32766: from all lookup main
32767: from all lookup default
Add the rule to the /etc/network/interfaces file as well:
post-up ip rule add from 192.168.1.64/27 lookup 101
Now make sure to add cleanup to remove the route and rule when the interface goes down:
post-down ip rule del from 192.168.1.64/27
post-down ip route del default via 192.168.1.65 table 101
[EDIT for ubuntu 16.04+]
Like indicated here and from test I've made from this help, ip route2 have changed his structure of commands.
For making work you will have to adapt just a bit to do in the order how the man ip points.
up ip route add default table 101 dev eth1 via 192.168.1.65
up ip rule add from 192.168.1.64/27 lookup 101
down ip rule del from 192.168.1.64/27
down ip route del default table 101 via 192.168.1.65
Or you will end up after a ifdown - ifup command with a error message @ifdown command(standard message to say that the peripherals is not correctly configured), and @ifup the absence of a route in table 101.
