Understanding Network Infrastructure · Part 4 · How do we use these principles to design a real network?
How should switches across multiple wiring closets connect?
With several wiring closets, deciding where to group switches starts with the paths and resources those closets will share. In this example, the access layer handles routing, and two distribution devices provide upstream connectivity. After drawing the normal topology, we check the paths and capacity left after one device fails, using the same demand assumptions.
Back to contentsMap wiring closets and service paths first↑
Three wiring closets may give you three groups of switches, but the closet count does not determine the architecture. To define the uplinks' roles, first identify which users in each closet communicate with which services.
Call the three closets' access devices A1, A2, and A3. Users connect to their local access device; shared servers and external connectivity sit in the service segment beyond D1 and D2. Record addressing and demand separately for traffic that stays within one access segment and traffic that goes upstream to shared services.
D1 and D2 both aggregate access segments and forward traffic between segments. The access layer has no separate core above it in this design. This is a two-tier architecture, also called a collapsed core because it combines distribution and core roles. It does not mean there are only two devices.
Show actual closet locations and fiber routes on the diagram. A1 may have a line to both D1 and D2, yet both fibers could be cut in the same conduit. If service-segment connections share a device or power source, document that common failure as another constraint.
Check whether all services are in one location. A local server in a closet creates different uplink demand from a shared server elsewhere. Record which traffic is included in D1/D2 link calculations alongside service locations. Then, when a server moves, update capacity planning as part of the same change.
For the rest of this example, each access device performs routing. L2 access is another option, but mixing both approaches in one drawing can obscure the VLAN boundaries. We are applying one design selected in the L2/L3 boundary comparison to the building's physical layout.
Place the three closets in one topology as shown below. A1, A2, and A3 each have L3 connections to D1 and D2, with no separate core above them. Use this diagram to distinguish traffic that stays in a closet from traffic that goes upstream to shared services.
↔ Scroll horizontally to explore the diagram.
Distinguish endpoint gateways from upstream routing paths ↑
A1's business endpoints use a gateway on A1 itself. Their VLAN ends in A1's access segment; the links from A1 to D1 and D2 are separate L3 connections. This design does not extend the endpoints' VLAN unchanged to both distribution devices.
To reach a shared server, PC-A first sends a frame to A1's gateway MAC address. A1 selects a route for the destination IP address and forwards toward D1 or D2. Each router then builds a new frame for the next hop on its outgoing link.
Whether both paths can carry traffic simultaneously depends on the routing design and installed routes. Two connected cables do not guarantee an exactly equal split. Even with equal-cost routes, the device's forwarding behavior and traffic mix affect how flows are distributed.
Here, D1 and D2 are independent L3 neighbors. Unless a specific technology makes them operate as one logical system, links to these separate devices cannot be combined into one EtherChannel. This example uses no link aggregation—just two routed paths.
If ECMP uses both paths, observe which flows are reassigned when one disappears. A change in the path count can change where a flow is forwarded. Test new connections separately from existing sessions to see whether users must reconnect after failover.
Draw the server's return path to PC-A on the same diagram. Sending a request through D1 does not guarantee that its reply also uses D1. If stateful security devices sit in between, verify that their configuration supports the resulting asymmetry so both directions can pass.
A1, A2, and A3 each have separate L3 links to D1 and D2. Locate the endpoints' gateways and the point where upstream paths to the servers split.
↔ Scroll horizontally to explore the diagram.
Calculate access and aggregate bottlenecks separately ↑
Each access device has one 10 Gbps link to D1 and one to D2. Assume each access device sends 8 Gbps upstream. This is not a field measurement; it is a fixed teaching assumption for comparing normal and failure conditions.
Normally, each access device has 20 Gbps of nominal uplink speed. Those link speeds can be added only if both routing paths can actually be used. A single flow does not thereby reach 20 Gbps. Path distribution and brief traffic concentrations need separate measurement.
If D1 fails, each access device has one 10 Gbps link left to D2. With the same 8 Gbps demand, the simple utilization ratio is 80%. Protocol overhead and additional traffic also consume capacity, so verify the actual headroom separately.
If demand from all three access devices uses the same service link on D2, the total is 24 Gbps. With an assumed 40 Gbps service link, the simple ratio is 60%. Spare aggregate capacity does not prevent an individual access device's 10 Gbps link from becoming the first constraint. That is why both calculations matter.
A 20 Gbps service link, by contrast, cannot carry the full 24 Gbps demand simultaneously. A connection that looks sufficient normally can become inadequate when a failure concentrates demand on one side. Carry the same direction, observation segment, and demand assumptions from the uplink-capacity calculation into this building diagram.
After D1 fails, calculate capacity separately for each access uplink and D2's service link. An access link can congest before the aggregate service link runs out of capacity.
↔ Scroll horizontally to explore the diagram.
Remove D1 and retrace communication in both directions↑
For failure analysis, do more than gray out D1. Remove its links and the functions it provided, then trace a complete path from PC-A to the server and from the server back, using only surviving devices.
Once A1 determines that the path through D1 is unusable, it must forward over an available path through D2. Detection, route selection, and forwarding updates do not happen instantaneously. Observe any lost packets and application retries separately, using the criteria in redundancy and recovery time.
D2 may have a route toward the server, but communication still fails if the server side has no return path to A1's business subnet. A default route does not prove that replies to PC-A's subnet take the correct path. Record forwarding state in both directions before and after the failure.
Here, the endpoints' gateway stays on A1. D1's failure alone does not mean the endpoints need a different gateway device. If A1 fails instead, endpoints connected only to A1 lose access even while D2 is healthy. This design can accommodate a D1 failure, but it cannot preserve those endpoints' access when A1 fails.
Planned maintenance and sudden failure may remove a path in different ways. Record a test that moves traffic before maintenance separately from a test that shuts a device down without warning. The failure method must be known to compare failover time and packet loss under equivalent conditions.
Test D1's return as well. As its routes are reintroduced, check traffic distribution, existing sessions, and whether new policies or stale configuration enter the network with it. Even if D2 is operating normally, verify the repaired D1's state before reconnecting it.
Hand over diagrams, addressing, policy, and tests as one consistent design↑
A two-tier handover needs operational documents as well as a device diagram. Physical and logical diagrams, address plans, service policies, and normal and failure test results must use the same device names and connections, so operators can reconstruct the actual network.
The physical diagram should show closets, devices, ports, cable routes, and power dependencies. The logical diagram should show each access device's gateway and subnet, its two L3 connections, and routes toward the servers. Mark where every VLAN ends so operators can identify the L2/L3 boundary.
Configuration documentation should explain the design before presenting a full command listing. Why is the gateway at access? If D1 disappears, how much demand moves to each surviving link? Which traffic is allowed or blocked? These decisions must remain understandable even if device syntax changes later.
Record how D1's failure and return were triggered, the two-way communication observed, actual measurement times, surviving load, and results that differed from expectations. Mark untested services as unverified, not healthy. Keep teaching calculations separate from actual measurements.
Use port labels and device identifiers that operators can read on site. They must be able to match the logical diagram's A1 to a physical device in a specific closet. Provide a mapping if the names differ, and require labels and documents to be updated together after a cable break or equipment replacement.
Using this topology, the next article on switch and router redundancy examines which failures are detected and which device takes over. An alternate path may exist while the application still waits for detection, failover, or reconnection. After separating those steps, we consider when several distribution blocks need a core between them.
Glossary↑
This is a list of key terms used in this article; their definitions are available in the full glossary.
- capacity planningcapacity planning
- Collapsed Corecollapsed core
- distribution blockdistribution block
- L2 AccessLayer 2 access
- L2/L3 boundaryLayer 2 / Layer 3 boundary
- Routed Accessrouted access
- Two-Tiertwo-tier design
- serverserver
- MAC addressMAC address
- VLANVLAN
- Network Accessnetwork access
- Core Networkcore network
- Distributiondistribution layer
- endpointendpoint
- gatewaygateway
- IPIP
- packetpacket
- L3Layer 3
- ECMPECMP
- Next Hopnext hop
- routingrouting
- Port Aggregationlink aggregation
- switchswitch
- device portdevice port
Scope and references
The technical principles draw on the official documents below. Equipment layouts and loads identified as examples are illustrative, not field measurements or guaranteed product performance.
- Campus LAN and Wireless LAN Solution Design Guide — The basis for campus layer roles and design options. We use its principles for separating layers; check each product's documentation for current performance and supported combinations.
- Cloud Campus LAN Design Guide — Additional context for comparing campus roles and layouts. The example does not require a particular management product.
- High Availability Campus Network Design — Routed Access Layer — Access-layer gateways and L3 boundaries, compared with traditional L2 access. We do not quote device performance figures.
- Campus Network for High Availability Design Guide — A reference for campus failure paths, recovery, and design validation. Loads in the text are assumptions used to explain the principles.
- Cisco Validated — Guidance on validated design, deployment, and operations resources and their scope. This does not mean Cisco tested our examples.
- Troubleshooting Overview — A troubleshooting process for collecting symptoms and facts, testing hypotheses, and recording changes and results. The failure causes in our examples are assumptions.
Before applying this to a live network, check support conditions for your hardware and software, and test the behavior your site needs.