Restoring lost network routes after loss of network

A routing table will make your route permanent (to avoid adding it again/manually after a switch failover); First, create a named routing table. As an example, we could use "mgmt".

echo '200 mgmt' >> /etc/iproute2/rt_tables

Just for an extended detail about the solution, above, the kernel supports many routing tables and refers to these by unique integers numbered 0-255. A name, mgmt, is also defined for the table. Below, a look at a default /etc/iproute2/rt_tables follows, showing that some numbers are reserved. The choice in this answer of 200 is arbitrary; one might use any number that is not already in use, 1-252.

# reserved values
255     local
0       unspec

Second, edit your post-up rule (under /etc/network/interfaces) like this

  post-up ip route add 10.1.0.0/24 dev eth0.101 table mgmt
  post-up ip route add default via 10.1.2.1 dev eth0.101 table mgmt
  post-up ip rule add from 10.1.0.0/24 table mgmt
  post-up ip rule add to 10.1.0.0/24 table mgmt

Alternatively an other solution could be a background bash script checking for the route existence and adding it back if it's missing, the script could check the result of ip route add 10.1.0.0/24 via 10.1.2.1 dev eth0.101 the script could be setup in a loop or a cron

ip route add 10.1.0.0/24 via 10.1.2.1 dev eth0.101
if [ $? -eq 0 ]; then
    echo "Route added again"
    sleep 10;
    command-to-call-the-script-again
else
    echo "Route exists"
    sleep 10;
    command-to-call-the-script-again
fi

Source: what is the best way to add a permanent route?

Your description is a bit vague with regard to the setup in place: how does the "switch failover" situation affects your server's net device ? loss of link ? mere interruption of packet forwarding ? some kind of explicit notification such as an LACP might do ? or what else ? Also, which interfaces have you got in auto and allow-hotplug ? the physical device ? or the vlan interface ? or both ?

The thing is, the described behavior does not really make sense according to the described setup alone. For instance, the mere loss of link usually does not suffice to delete routes, nor to set interfaces down, unless there is some additional setup in place that does so explicitly.

I would also say that this additional yet unmentioned setup cannot be a standard ifplugd or netplugd, which act on link events (such as cable plugged/unplugged), because these typically invoke ifup and ifdown on those events, and thus your post-up command would be executed when the link would come back up, if their configuration is consistent with your vlan interface.

Note that the ifupdown's allow-hotplug setting relates to device hotplug, not cable hotplug, meaning that it acts basically only at boot-time (i.e. when the kernel detects the device presence for the first time) and/or if your net device is e.g. an USB dongle and you plug/unplug it to/from a USB port of your server.

I would therefore suggest that there is some custom service, running on your server, that possibly detects loss of networking through some probe, maybe a periodic ping or a TCP connection established on purpose, that deletes your route as soon as it detects the loss of networking, but does not add it back when that probe connection comes back alive.

To answer your specific question:

How can I configure an interface to restore routes to networks that where lost, but, to quote the old song, have now been found?

It depends on what it means to your application having "lost network connection" and thus what it means having it back. If the interface link state is enough then you can really just rely on either ifplugd or netplugd or equivalent (whichever you prefer or best fits your requirements): your post-up command is enough if associated to an "ifupdown" vlan interface that is simply always existent on top of its master device or that is configured to follows its master's fate.