How do you test whether a network can withstand failures?

Define the failure and the evidence of recovery↑

A requirement to be stable during failures does not define a test's start or finish. Specify which fault, which service, and which conditions must be met. Turn the word 'stable' into criteria that a real test can check.

For this teaching example, suppose a business request must complete again within 3 seconds after one of A1's uplinks fails. The 3 seconds is an example test criterion, not guaranteed Cisco performance or a suitable target for every application. Service owners and operators must agree on the real value.

Define when a business request starts and completes. A successful new connection, a surviving existing session, and a processed request with a response are different tests. For services such as payments or storage, where duplicate processing matters, review retry outcomes with the application team too.

Keep failure criteria separate from stop conditions. Exceeding the recovery target means the test failed; an unexpected service outage or lost management access may require stopping immediately and restoring the previous state. If a stop condition occurs, prevent further impact rather than continuing just to observe the cause.

Set priorities for conflicting goals before the test. A bypass that reduces loss but violates security policy is not a valid alternative. Record both the business operations that must continue and the traffic that must remain blocked, so restoring connectivity also preserves required protection.

Document and hold constant the devices, paths, configuration, and test environment associated with the requirement. If software or policy changes later, reassess whether the old result still applies. What was validated is a design's behavior under specific conditions, not every configuration with the same product name.

Record normal round-trip paths and load first↑

Before introducing a fault, establish an operational baseline so you can judge what changed. Instead of merely writing 'normal,' record which endpoint uses which service, through which path, and with what response. An unclear definition of normal also makes the failure-test comparison unreliable.

Hold PC-A's address, gateway, and name-resolution results constant, along with the destination service's address and port. Repeat the same request consistently and record normal completion times and errors. Separating name resolution, connection establishment, authentication, and the business response makes the failed stage easier to identify.

Record the forwarding path and utilization in each direction. A route-table entry is not proof that actual packets leave through that interface. Use counters or approved packet observation where needed, minimizing sensitive content according to the collection purpose.

Align measurement time references too. To compare logs from different devices, you need their clock synchronization status and error. Subtracting timestamps from unsynchronized logs as if the difference were detection or recovery time can give you the wrong event order.

Link baseline configuration to measurement files. If you cannot tell which settings produced a result, reproducing it is difficult. Preserve timestamps, environment identifiers, and related changes, and protect sensitive operational information according to handover recipients and storage permissions.

A test with almost no traffic may differ from peak-hour behavior. Specify test demand and background traffic, keeping them comparable between normal and failed states. If production load could not be reproduced, record that limitation and the load actually tested.

Marking observation points on the layout makes the baseline easier to compare. At PC-A, observe actual business responses; at A1, D1, and D2, observe forwarding paths and utilization. Use the same locations and conditions when later testing the indicated uplink.

List lost functions and remaining paths and capacity for each failure↑

Build the test list around the failures the design is meant to withstand, not an order for unplugging cables. Separate faults such as one uplink, one distribution device, or one power feed, and list the functions and paths each fault removes.

When testing one failed A1 uplink, verify traffic in both directions over the remaining link. When taking D1 out of service, test the loss of every path and function it provided. Passing a single-link test does not validate a whole-device failure.

Shared failures, such as common power or the same conduit, are separate cases. Without authorization and a safe environment to interrupt that scope, drawings or a limited simulation can provide some evidence, but unmeasured aspects remain. Distinguish design review from results measured under an actual induced fault.

For each test, list affected endpoints and services, remaining paths and capacity, and management access. Business traffic may survive while management is lost, or a management page may load while user services fail. Record these in separate columns so recovery access is evaluated too.

Production testing requires prior approval, a maintenance window, impact notification, named contacts, and a way back. Applying this article's diagrams to a live test requires a site-specific plan and authorization first. Begin in an isolated validation environment, then decide what can safely extend to production conditions.

The uplink, all of D1, common power, and return to service are separate tests. For each one, decide how to check bidirectional traffic, remaining capacity, and management access.

Measure path changes and business recovery separately↑

Stable routing logs after a fault are useful evidence, but they are not the end of business recovery. Record route selection, forwarding-state installation, actual packet round trips, and application completion separately to explain where additional time was needed.

Use the actual fault as the test's time reference. A button click and the link going down may occur at different times, so state your observation point. Record the last success before the fault and the first success afterward, showing the limits imposed by probe spacing.

For example, probes sent every 200 ms cannot reveal the exact instant of a change between two probes. You can observe when the first successful reply arrives, but not the exact instant recovery finished inside the device. Include probe round-trip time and timestamping method in the interpretation.

An ICMP reply can return while a business port remains blocked or authentication, name resolution, or connection-state handling still fails. Exercise the real business request under consistent conditions and record the failed stage and response separately. Keep test traffic within approved scope, without arbitrarily changing business data.

Distinguish a faulty test setup from an actual service failure. If the probing endpoint loses access in the same failure domain, its result alone cannot determine the destination service's state. Record the test endpoint's location and path, and compare an independent observation point where needed.

Even a passing result needs its observations and conditions. Record which service recovered from which fault, how long it took, the probe resolution, and any unobserved segments. Report network and application recovery times separately so readers can see what still had to happen after packet forwarding resumed.

The timeline shows results from probes sent every 200 ms. Because the exact recovery instant between probes is unknown, record both the measurement interval and the first observed success.

Test return to service and rollback, then hand over unverified items↑

After service recovers on the alternate path, reconnect the device or link taken out of service. Returning to the original configuration can redistribute traffic, synchronize state, and change policy. Successful failover does not replace a separate return-to-service test.

First verify that the returning device's software and configuration match the test baseline. An established neighbor relationship does not mean every service is ready. After traffic moves to the new path, observe existing sessions, new requests, load, and errors again.

If return to service has an unexpected impact, roll back according to the predefined stop conditions. Document which settings and connections must return to which state, who will act, and who will verify the result. Prepare separate access in case the management path also disappears.

Keep passed, failed, not run, and conditionally verified results separate in the final table. Do not mark a whole-device failure as passed based only on a single-link test. Connect failures to design changes and retesting, and hand unrun tests and their remaining risks to a named owner.

For each unverified item, record why it remains open and what is needed next. Missing equipment calls for a different follow-up from a test whose risk was not approved. Replace blank cells with the required environment and responsible person so the next operator knows what to test.

Validation results become a reference during real incidents. Operators need ready access to normal paths, permitted alternatives, remaining capacity, and rollback conditions. During incident triage and recovery, use these records to determine the scope of impact and the recovery order. Include security-policy test results so restoration does not reopen prohibited traffic.