Understanding Network Infrastructure · Part 5 · Connecting the enterprise network to branches, clouds, and the Internet
Two ISPs, one destination: which route does BGP choose?
From a TCP session through UPDATEs, policies, and path attributes to installing and advertising a prefix
When two ISPs advertise the same prefix, BGP does not simply pick the route that looks closest. It applies policy to the routes received from peers, compares attributes such as LOCAL_PREF, AS_PATH, and NEXT_HOP, and passes the selected route through to the RIB and FIB. This article separates the roles of eBGP, iBGP, route reflectors, and RPKI, then follows a route to find where it stopped: receipt, policy, selection, or advertisement.
BGP peers: what happens between establishing a TCP connection and exchanging routes?↑
BGP peers use TCP port 179 to establish sessions and exchange OPEN messages to verify the version and AS number, then determine the hold time and capabilities to use. Once the session is established, KEEPALIVE messages check liveness, and UPDATE messages advertise or withdraw routes. A serious error can trigger a NOTIFICATION and close the session.
A TCP connection does not mean the desired routes have been exchanged. A peer may be Established with no prefixes, or an UPDATE may have arrived only to be rejected by policy. Check session state and route state separately.
If the TCP three-way handshake fails, the peers never reach the BGP OPEN exchange. Start with transport prerequisites: the source IP and route, TCP/179 ACLs, multihop TTL settings, and authentication mismatches. Even after TCP connects, if the AS number received in OPEN differs from the expected value or the peer lacks a capability required for the session, a NOTIFICATION may be sent and the session may close. However, RFC 5492 prohibits terminating a session merely because an unrecognized or unsupported capability was received. Repeated transitions through Idle, Connect, Active, OpenSent, OpenConfirm, or Established help locate the failing stage.
Established means the control channel is alive. A session can remain Established even if the peer sends no NLRI or our import policy rejects everything. Support for Route Refresh and Graceful Restart also affects how routes are handled during policy changes and restarts. Check session uptime, prefix counts for Received and Accepted, and the last NOTIFICATION together. Those details show what 'BGP is up' means for actual route exchange.
If a peer will not come up, check both ends: source interface, update source, eBGP multihop/TTL, ACLs, firewalls, AS numbers, and authentication. TCP Established alone does not prove successful BGP capability negotiation. Align state-transition logs with SYN/SYN-ACK, OPEN, and NOTIFICATION timestamps to distinguish transport failures from rejected BGP parameters.
Sources for this section IETF RFC 4271
BGP UPDATE messages carry prefixes and the attributes used to evaluate routes↑
An UPDATE carries reachable prefixes and path attributes. AS_PATH records the sequence of ASes traversed by the advertisement; finding your own AS in it is a basic way to detect a loop. NEXT_HOP identifies the next point to which packets should be sent for that route.
BGP does not measure link bandwidth or physical latency to find the shortest path. A shorter AS_PATH is often preferred at one stage, but earlier local-policy decisions can change the result. The exact decision process also includes implementation-specific extensions.
An UPDATE can carry newly reachable NLRI and the path attributes shared by those routes, and use withdrawn routes to identify prefixes that are no longer available. MP-BGP represents reachability for each address family in separate attributes. Even while the same TCP session remains connected, differences in activation or policy between the IPv4 unicast and IPv6 unicast address families can leave prefixes visible in only one of them.
AS_PATH supports loop prevention and route selection. NEXT_HOP must be resolved through the routing table before forwarding; the data plane uses the resulting forwarding information. ORIGIN, MED, LOCAL_PREF, and communities have different scopes and meanings. A community labels routes for policy. An operator-defined community has its intended effect only when the receiver applies a corresponding policy. However, some values have standardized advertisement restrictions, such as NO_EXPORT in RFC 1997. Confirm the actual handling with the other operator. Read the prefix and all its attributes as one route object to understand the advertisement's intent.
Attributes are not all carried the same way across every session. LOCAL_PREF generally applies within an AS and is not advertised to external eBGP peers. NEXT_HOP handling and community propagation also depend on policy. When comparing routes, save more than the prefix: capture the received attributes, changes made by import policy, and the final Best indication.
Sources for this section IETF RFC 4271
What to look for in the diagram Trace the TCP port 179 connection, the BGP state machine, OPEN capability negotiation, and route processing through Adj-RIB-In, import policy, Loc-RIB, the RIB/FIB, export policy, and Adj-RIB-Out. Read the states from Idle to Established separately from the policy and route-processing boxes below.
↔ Scroll horizontally to explore the diagram.
Our AS chooses exits by local policy before distance↑
Within an AS, raising LOCAL_PREF can favor a particular exit; import policy can also classify routes by community and prefix. To influence traffic coming from an external AS, you can use AS_PATH prepending, MED, or more-specific prefix advertisements. How the other AS treats that information remains its policy decision.
Do not assume a universal order such as 'LOCAL_PREF, then always AS_PATH.' Check the platform's best-path documentation and the actual route attributes. Vendor-specific attributes such as Weight may come first, and multipath has its own conditions. The key is to distinguish attributes that select your internal exit from hints you send to neighbors.
For ISP-A's routes, a higher LOCAL_PREF can generally make our AS's internal routers prefer ISP-A as the exit. The value is distributed through iBGP, not passed unchanged to eBGP neighbors. In contrast, AS_PATH prepending and MED try to influence neighbors' choices for inbound traffic to us. Their own LOCAL_PREF policy can override these signals. That is why our control over outbound and inbound traffic engineering differs.
Policy matches prefix lists, AS_PATH, communities, and RPKI status to change attributes or reject a route. One broad match can also affect the default route or customer prefixes, so check candidate route counts and expected changes before applying it. Vendor features and settings extend the best-path process; inspect the device's selection reason rather than relying on a memorized sequence. Document the goal, match scope, and expected winner with example prefixes.
Policies use prefix lists, AS_PATH and community conditions, and route maps or policy statements to permit, reject, or modify values. Explain which candidate passed through which policy and lost on which attribute—not simply that 'BGP chose the shorter path.' Equal-cost multipath use and the maximum number of paths also depend on configuration.
Sources for this section IETF RFC 4271 · IETF RFC 7454
What to look for On the left, LOCAL_PREF selects an exit from our network. On the right, advertising a longer AS_PATH attempts to influence the remote network's choice. MED also conveys route-selection information to another network, but the configuration shown here is AS_PATH prepending. In both cases, check the policy's scope and the remote network's policy to understand which route will actually be selected.
↔ Scroll horizontally to explore the diagram.
External BGP routes need a way to reach internal routers↑
Between different ASes, eBGP exchanges external reachability. Within one AS, iBGP carries the BGP routes needed for selection to internal routers. By default, a route learned from one iBGP peer is not readvertised to another iBGP peer. Large networks may therefore use route reflectors instead of a full mesh.
A route reflector reduces session count but changes route visibility and selection. Not every client sees every alternative directly. Verify that the intended exit is actually selected, including the effects of IGP cost and NEXT_HOP reachability.
When an edge router passes an ISP's NEXT_HOP into iBGP, that address must be reachable through an IGP, static routing, or a directly connected route for packets to leave. Depending on the design, next-hop self lets internal routers use the edge itself as the next hop. If a BGP route is Best but not installed in the FIB, or traffic takes the wrong exit, check NEXT_HOP reachability and IGP cost together.
A route reflector learns from one client and reflects those routes to other clients, reducing the need for a full mesh. Each client may rely on routes selected and exposed by the reflector rather than comparing all external alternatives itself. This can cause path hiding. Consider cluster design, extensions such as Add-Path, and IGP topology. Test the tradeoff between fewer sessions and visibility of optimal paths against actual failure scenarios and traffic paths.
Receiving a prefix through iBGP is not enough: the routing table must contain a usable route to its NEXT_HOP before the received route can be used for forwarding. IGP, static, and directly connected routes can all satisfy this condition. The design may require the edge router to use next-hop-self, making itself the next hop, or to advertise the external link's addresses into the IGP. Compare the BGP table, RIB, next-hop route, and FIB to connect control-plane information with the actual data-plane exit.
Sources for this section IETF RFC 4271 · IETF RFC 4456
RPKI checks whether a prefix may be advertised by its origin AS↑
An RPKI Route Origin Authorization is a signed statement specifying which AS may originate advertisements for a prefix and up to what length. Routers can classify received BGP routes as Valid, Invalid, or NotFound and use that status in local policy.
This check does not authenticate the entire AS_PATH. It checks the relationship between an origin AS and a prefix. How to treat Invalid routes is an operational-policy decision, including validation-data availability and exceptions. RPKI does not replace prefix filters, max-prefix limits, or per-peer import and export policies.
A received route's prefix is Valid when at least one validated ROA record covers it and permits its origin AS and prefix length. It is Invalid when a covering validated ROA record exists but no such record permits that AS and length, or NotFound when there is no covering validated ROA record. A more-specific prefix that exceeds a covering ROA's maximum length is Invalid only if no other validated record permits it. Check the actual advertised lengths when creating ROAs. The resulting status comes from comparing the received route with validated records and serves as input to router policy.
If a router loses contact with its validation cache, retention of earlier results, treatment of stale data, and fail-open or fail-closed behavior depend on the implementation and operating policy. Before automatically rejecting Invalid routes, plan how to correct errors affecting customer and your own prefixes in their ROAs. Origin validation does not stop every route leak, every false AS_PATH, or every hijack. Keep prefix filters, max-prefix limits, and peer-contract checks in place.
Validation does not prove that the entire AS_PATH is the actual traffic path, nor does it fully establish prefix ownership or service safety. Decide how import policy will handle each validation state, and plan for the effects of validator failures and stale cache data. When a route is missing, check the source ROA, validation time, advertised prefix length, and the policy action actually taken.
Sources for this section IETF RFC 6811
Missing a route? Trace each stage from receipt to advertisement↑
Scenario: From ISP-A, we appear to have received a prefix, but traffic still leaves through ISP-B.
- Check the peer's TCP and BGP states, remote AS, capabilities, and last reset reason.
- Distinguish actually receiving the prefix from accepting it through import policy.
- Compare each candidate's NEXT_HOP, LOCAL_PREF, AS_PATH, MED, communities, and RPKI status side by side.
- Check whether the best path was installed in the RIB and FIB, and whether a more-specific prefix exists.
- Look up the route to NEXT_HOP in the routing table and the actual outgoing interface.
- Verify what our router advertised to each peer after export policy.
Follow a prefix through four stages. First, confirm receipt of its UPDATE from the peer. Then check acceptance by import policy, selection as the best path and installation in the RIB/FIB, and finally advertisement to other peers after export policy. At each stage, inspect the affected route's attributes and rejection reason, not just total prefix counts. Consult platform documentation for Route Refresh and policy-tracing support without soft reconfiguration.
To test traffic, recursively resolve the best path's NEXT_HOP, then check actual egress counters and external traces. Other ASes control return-path selection through their policies, so inspect our advertised routes, looking glasses, and flow telemetry separately. Keep before-and-after route snapshots and the expected number of affected prefixes; check safeguards such as max-prefix limits. Before resetting a session, determine which other prefixes may be disrupted and prepare a rollback plan.
A route can be Received, Accepted by policy, Best and Installed, or Advertised to another peer; these are different stages. If the device does not retain pre-policy routes, viewing them may require Route Refresh or another observation method. Review operational impact and platform behavior before proceeding.
Sources for this section IETF RFC 8212 · IETF RFC 7454
Glossary↑
Sources and verification
- IETF RFC 4271 — BGP-4 messages, path attributes, and the decision process
- IETF RFC 4456 — BGP Route Reflection
- IETF RFC 8212 — Default rejection when eBGP import or export policy is absent
- IETF RFC 6811 — BGP Prefix Origin Validation
- IETF RFC 7454 — BGP operational and security recommendations
- IETF RFC 5492 — Capabilities Advertisement with BGP-4
- IETF RFC 1997 — BGP Communities Attribute
- IETF RFC 8642 — Policy Behavior for Well-Known BGP Communities
The detailed best-path order, vendor-specific attributes, multipath requirements, and handling of RPKI status vary by implementation. This article distinguishes RFC behavior from the role of local policy.