Understanding Network Infrastructure · Part 3 · How does traffic reach other networks and the Internet?
How does static routing detect remote failures and switch to a backup?
Using next-hop resolution and tracking to control static-route installation, failover, and failback
Configuring a static route does not guarantee that it can forward packets. The next hop must resolve to a usable outgoing interface and neighbor before the route can carry traffic. Even with the local link up, a remote destination may be unreachable. We follow how floating static routes and tracking work together to detect a primary-path failure, switch to a backup, and restore the primary—and what happens to existing sessions along the way.
A static route is configured. Why might it be missing from the routing table?↑
Even with 0.0.0.0/0 via 192.0.2.1 configured, the router must resolve the next hop, 192.0.2.1, before it can use the route. That address must be on a directly connected network or reachable through a recursive lookup using another route.
An Ethernet interface being up and the next hop resolving through ARP do not prove that the final Internet destination is reachable. The hop between the customer-premises equipment (CPE) and the ISP may still work even when the provider's upstream network is down, leaving the default route installed.
A route in the configuration is not the same as an active RIB route. If the next hop cannot be resolved in the current VRF, or its connected interface is down, it may remain a candidate rather than a usable route. For the same prefix, a more-preferred dynamic route may also leave the static route on standby. Compare the configuration, RIB, and FIB side by side to see its current role.
Resolving next hop 192.0.2.1 with ARP shows that the first ISP router is reachable at L2. Beyond that router, an upstream BGP failure or Internet outage does not change the installation conditions for a local static route. Packets continue to leave through the active interface and encounter a blackhole. Understanding the limited detection rule—if the link stays up, the route can remain installed—explains why a remote probe or dynamic routing may be needed.
An outgoing interface reporting up does not prove that a destination beyond it is reachable. The Ethernet switch or carrier-facing link may remain up while a distant peer is down, so the static route can remain selected. Record route installation, next-hop adjacency, and end-to-end reachability as separate observations.
Sources for this section IETF RFC 1812
On Ethernet, naming only an output interface may not be enough: why the next-hop IP matters↑
On a point-to-point link, specifying only the outgoing interface often identifies the other endpoint unambiguously. On a multi-access network such as Ethernet, however, omitting the next hop and configuring each destination prefix with only an outgoing interface may cause the device to treat destinations as on-link, attempt ARP for each one, or rely on proxy ARP.
On multi-access networks, specify the actual next-hop address and check both its neighbor resolution and the route used to reach it. This makes the forwarding intent explicit. Installation rules and recursive lookup behavior vary by vendor and operating system.
A serial point-to-point link has only one peer, so the outgoing interface identifies where to send the packet. On Ethernet, one interface may have several neighbors. Configuring only 203.0.113.0/24 via Ethernet1 may cause the device to use ARP for each destination as if it were directly connected. More destinations mean more ARP entries and broadcasts. Without proxy ARP from an upstream router, forwarding fails.
Specifying the next-hop IP lets the router resolve just that address and send the neighbor traffic for the prefix. A fully specified static route may require both an interface and a next hop, or be used to constrain recursive lookup, depending on the platform. Similar-looking CLI commands can behave differently across operating systems. Check the route output for the resolved interface and adjacency to verify the result.
Some situations call for both an interface and a next hop in a fully specified route; check the platform's syntax and behavior. The key question is whether the FIB can map the destination prefix to an egress and adjacency. Do not review just the configuration line. Also check the next-hop IP's connected prefix, ARP/neighbor state, and the actual destination MAC.
Sources for this section IETF RFC 1812 · Cisco Static Route Next Hop
The primary route is still present. How does a floating static route wait in reserve?↑
The primary and backup routes point to the same destination prefix. Give the backup route a lower preference than the primary route so it is not normally selected.
Many platforms treat a higher administrative distance as less preferred, but others use different preference systems. Check the device documentation to see which direction the numbers run. When the primary route leaves the RIB, the lower-preference candidate can take over—hence the name floating static route.
The crucial question is not whether a backup exists, but when the primary route is removed. It may be withdrawn when the primary interface goes down or its next hop cannot be resolved. Without separate detection, a distant failure may leave those installation conditions unchanged.
If the primary route is OSPF-learned 0.0.0.0/0 and the static backup is also /0, those two candidates compete by preference. But if the backup is 0.0.0.0/0 and the primary is the more-specific 203.0.113.0/24, traffic matching that /24 still uses the primary while it remains available. A failover design must account for the actual competing prefix set, not just “one circuit.”
A floating route's next hop must also be resolvable when the route is used as a backup. If the backup circuit is completely disabled and the recursive route is unavailable, route installation may be delayed after the primary is withdrawn. If ARP information is missing, forwarding may be delayed even after the route is installed. Conversely, if the primary route stays installed whenever its interface is up, the backup may never take over. Test the primary's withdrawal trigger separately from the backup's readiness.
A backup route is not enough on its own. Its next hop must be reachable, the far end must have a return route, and the firewall, NAT, and circuit capacity must support the required traffic. Decide whether to return immediately when the primary recovers or wait for stability; otherwise, a flapping path can cause repeated switches.
Sources for this section IETF RFC 1812 · Cisco Static Route Next Hop
When a probe detects a remote link failure, tracking changes the route↑
Some devices periodically test ICMP echo, TCP connections, or DNS responses and feed the probe results into a tracking object. A static route tied to that object can be removed after consecutive failures and reinstalled after recovery. Feature names include IP SLA, RPM, and Track.
Choose a probe target that represents the path you need to protect, not just the ISP next hop. Testing only one public IP risks mistaking a server outage for a circuit failure. Combine targets, failure counts, timeouts, and failback delays to distinguish brief loss from an outage. BFD is a separate protocol for rapid path-failure detection between peers, not a general-purpose probe for arbitrary application health.
Verify that test packets actually leave over the primary circuit. Specifying a source interface does not necessarily pin the outgoing path. A probe that succeeds through the backup can falsely indicate primary-path recovery, so check target-specific routes and policy too. An ISP gateway, a remote Internet IP, and a business service test different portions of the path. Choose targets and decision criteria around the service being protected and the possibility that an external target itself may fail.
For example, a probe sent every 3 seconds could mark the path as failed after three consecutive failures and recovered after five consecutive successes, with a 30-second hold-down period. Choose the numbers for the environment and platform capabilities; using different failure and recovery thresholds provides hysteresis that reduces flapping. Document whether the probe packets' QoS, VRF, NAT, and return path match real user traffic or deliberately differ from it. That context determines what a successful probe actually demonstrates.
Maintenance or ICMP rate limiting at one target can make an entire circuit look unavailable. Compare using multiple targets and conditions with alternatives such as provider BFD or a routing protocol. Set detection intervals, consecutive-failure counts, switchover delays, and recovery-stability periods to suit service requirements and control-plane load. Check that the probe's own route does not create a circular dependency with failover.
Sources for this section IETF RFC 5880 · IETF RFC 5881 · Cisco ISP Failover with IP SLA
What to look for in the diagram Follow the IP SLA probe from an ICMP timeout for target 198.51.100.1 through the two steps: withdrawing the AD 1 primary static route and selecting the eligible AD 10 floating static route. A gateway reply alone does not verify the remote business service, so distinguish the detection, switchover, and service-recovery checks listed below.
↔ Scroll horizontally to explore the diagram.
A backup route takes over, but existing TCP sessions may still drop↑
After failover, new packets leave through the backup interface, but existing TCP sessions may still break. If the source public IP changes, or the NAT mapping and firewall state exist only on another device, the connection's address-and-port tuple may change or packets may be dropped at the replacement device. Either can interrupt communication with the remote server.
Immediate failback as soon as the primary circuit recovers can cause brief route instability. Use a tracking hold-down or recovery delay, and design stateful device synchronization together with both traffic directions. A successful route change and an uninterrupted existing session are separate outcomes.
Suppose ISP-A uses source NAT address 198.51.100.8, while ISP-B uses 203.0.113.8. If packets in an existing TCP flow are translated to the new public address after failover, the server sees a different address-and-port tuple that no longer matches the existing connection. The client may receive no response or lose the connection and have to reconnect. This does not mean the client necessarily receives a response with a different address or port. Even if the NAT address remains the same, an existing connection can break if the replacement firewall lacks session state or the return path still points to the old device.
An SLA that says only “route switchover in 3 seconds” is incomplete. Record separately how soon new sessions work and whether existing sessions survive. Voice and video over UDP, long-lived database connections, and VPN tunnels respond differently to loss. Even with state synchronization, test TCP sequence numbers, timers, and application reconnection end to end. Failback changes the path and address again, so decide whether it can happen automatically during business hours or requires operator approval.
Do not stop a failover test at ping. Measure new connections separately from long-lived ones. Record the externally visible source tuple, return path, NAT/firewall state synchronization, TCP retransmissions, and application recovery time. Restoring new connections within seconds and preserving existing sessions without interruption require different device state and circuit designs.
Sources for this section IETF RFC 3022 · IETF RFC 9293
Before testing, define failure and failback criteria in a table↑
Scenario: During an upstream outage at ISP-A, the backup route is not selected, or the two paths keep alternating.
- Check the primary static route's exact installation conditions and recursive next-hop resolution.
- Test the link, ARP, ISP next hop, and remote destination separately to locate the reachable boundary.
- Check the probe's source interface and return path against the intended circuit.
- Record failure counts, timeouts, hold-down, and failback delay, then test the response to brief packet loss.
- For failover, collect before-and-after RIB, FIB, NAT translation table, and firewall state with aligned timestamps.
- Check new and existing sessions separately, and document the expected scope of interruption in the runbook.
Use test-table rows for a physical link going down, an unresponsive ISP next hop, upstream Internet loss, high packet loss, one failed probe target, and a failed tracking process. Use columns for the detection signal, primary route removal time, backup installation, NAT/firewall state, and results for new and existing flows. Document both normal behavior and failures the design intentionally does not detect.
For each scenario, align RIB/FIB change times, probe logs, interface counters, and packet captures. During recovery, as the primary alternates between up and down, check the hold-down and observe sessions as traffic leaves the backup path. Restore the configuration after testing and verify the original route preference and probe source settings. The completed table provides a baseline for comparing expected and observed behavior in a later outage.
Use a common clock to record detection, RIB/FIB changes, the first successful packet, and full recovery in every test. Use those results to tune timers and the runbook. “Automatic failover” then becomes a reproducible operating condition, not just a configuration description.
Sources for this section IETF RFC 5880 · Cisco ISP Failover with IP SLA
FAQ↑
Does BFD also check Internet service health?
Will existing TCP sessions survive the switch to the backup?
Glossary↑
Sources and verification
- IETF RFC 1812 — IPv4 router route and forwarding requirements
- IETF RFC 5880 — Bidirectional Forwarding Detection fundamentals
- IETF RFC 5881 — IPv4 and IPv6 single-hop use of BFD
- Cisco Static Route Next Hop — vendor examples of next-hop specification and floating static routes
- Cisco ISP Failover with IP SLA — IP SLA and tracking for static-route failover
- IETF RFC 3022 — NAT/NAPT mappings that may change when the path changes
- IETF RFC 9293 — TCP connection endpoints and state
Static-route preference values, failure-tracking commands, probe types, and failback conditions vary by device. In an actual deployment, check link state, remote-destination reachability, and existing-connection continuity separately.