When the network fails, what should you check and restore first?

Distinguish device alarms from service outages without delaying response↑

When a device raises an alarm, first check the service impact. Can users access what they need? Has quality fallen outside agreed limits? Has the capacity to withstand another fault disappeared? Distinguishing these states lets you communicate impact and urgency accurately.

If D1 fails but D2 keeps business running, report lost redundancy separately from a current service outage. Continued service does not justify postponing repair. With less protection against another fault or overload on the remaining path, set a risk-based repair deadline and monitor that path more closely.

Conversely, every device can be powered on while repeated request failures or excessive delay constitute a service outage. A connected link does not guarantee successful business operations. First collect actual results showing which users and requests are affected.

At intake, record onset and current times, affected users, locations and services, reproduction conditions, recent changes, and a seemingly healthy comparison. Mark the cause as unknown if it has not been established. Passing a guess along as a confirmed cause can make the next investigator overlook other clues.

When redundancy is lost, explicitly state the protection remaining against another failure. A report that business still works can understate the need for urgent maintenance. Share remaining paths and headroom, expected repair time, and the impact of a further fault so service owners can assess the risk.

Do not wait for severity classification to prevent urgent harm. If impact is spreading or safety and data-loss risks are high, apply approved emergency procedures and keep recording events. Assign technical action and communication leads so different people do not unknowingly make conflicting changes on the same device.

Compare working endpoints with affected endpoints to narrow the problem↑

To narrow the failure domain, record both what fails and what still works. 'The whole office is down' is only a starting point. Compare conditions such as wired versus wireless, the same VLAN versus another VLAN, and internal versus external services.

If only PC-A fails under given port conditions while another endpoint works under the same conditions, start by examining the affected endpoint and its access settings. Several failures under the same access device suggest something shared by those endpoints, such as a link or policy. These comparisons narrow the investigation; they do not immediately prove a particular device is faulty.

Keep conditions comparable when testing access by address versus by name. Replacing a name with an address in HTTPS can change the server name, certificate, and virtual-host behavior. If you changed more than name resolution, those two results alone do not establish a DNS problem.

Start with low-impact observations: existing monitoring and logs, interface status, and read-only route and policy checks. Reboots and broad resets can destroy evidence along with working state. Do not repeatedly use them as the first action without understanding the cause.

Record when and where logs and captures were collected. If a packet appeared at A1, did it also appear at D1 at the same time, in which direction, and for which request? Gather evidence comparable to the operational baseline, choose a plausible cause, and test something that distinguishes that hypothesis before making unnecessary configuration changes.

Place PC-A's failed path beside PC-B's working path to the same destination. Separate their individual access segments from shared segments to decide what to compare first. Then check device state and measurements in those segments before determining the cause.

Contain an ongoing compromise before prioritizing availability↑

When malicious activity or a compromise is spreading, preventing further harm takes priority over keeping services running. Immediately contain confirmed infected or compromised systems and segments. If the scope is uncertain or the compromise is spreading across segments, widen containment, including disconnecting the entire affected network from other networks. Cutting Internet access alone may be insufficient if lateral paths remain. For life-safety systems, follow preplanned safety procedures and alternate operations; service importance must not justify leaving a compromised path open.

Containment is not the same as powering equipment off. Under incident command, disconnect network access and safely preserve needed evidence such as logs and memory. Do not delay urgent containment to collect evidence. Use a trusted, separate communication channel instead of a compromised one. Before reconnecting, address the cause and entry path and remove remaining threats. Verify the restored system and complete required account and credential actions, then reconnect in stages with increased monitoring. Restored connectivity alone does not close a security incident.

Set recovery order by business importance and service dependencies↑

Do not set recovery order solely by the number of inconvenienced users. First prevent further harm and safety risks, then secure a safe way for administrators to reach devices and work. Next, consider business criticality together with the functions each service needs first.

If an application depends on authentication and name resolution, first check that authentication and name resolution work. Repeated application restarts will not help while required authentication and name resolution remain unavailable. Conversely, if name resolution works but path policy blocks traffic, changing DNS first will not solve the problem.

DHCP is not the first recovery target in every outage either. Current traffic from an endpoint with a valid address can be affected at a different time from new access or lease renewal. Identify which DHCP function the interrupted business operation needs, and record the scope and timing of future impact too.

For equally important services, consider actions that restore more essential work with fewer changes. But a temporary bypass can cause new harm if it removes a security boundary or overloads the remaining path. Obtain approval for expected benefit, new risks, stop conditions, and rollback as one plan.

Recovery priority and root-cause investigation do not always follow the same order. If a trusted previous configuration permits minimal recovery, service can resume while evidence-based investigation continues. Clearly separate what has been restored from what remains unexplained, so a temporary recovery is not mistaken for a final fix.

After preventing further harm and securing management access, follow the interrupted service's dependencies to set recovery order. Check whether DNS or DHCP is actually required at the affected stage.

Send the same request from failing PC-A and working PC-B↑

In this hypothetical case, PC-A is 10.60.10.21/24, PC-B is 10.60.40.21/24, and the destination is App-S's TCP 443 service at 10.60.20.10. The two endpoints are in different business VLANs and send the same application request. Begin by recording that it fails from PC-A and succeeds from PC-B.

PC-B's success is evidence against the entire server being down. However, the server could reject PC-A's address, or an intermediate policy could differ, so it does not prove the server is healthy under every condition. Compare both paths with the same request at the same time.

Check PC-A's gateway reachability and route to the destination, then follow request packets at approved observation points. Here, a request leaving A1 appears at D1's ingress but not its egress toward the server. That narrows route and policy checks to the interval between D1's input and output. Also check whether the capture could simply have missed the packet.

Suppose comparison of D1's recent changes and policy order reveals a VLAN10 request blocked by an incorrect ACL deny rule. Check that the rule's counter increases when reproducing the same request, and compare the configuration with the working baseline. Do not conclude from one counter alone: confirm that paths, packet captures, and policy records point to the same problem.

Limit tests of traffic that guests must not access to approved test endpoints and destinations. Do not validate policy by broadly probing unrelated addresses or services. Test the intended permissions and denials within authorized scope; an outage does not authorize you to expand targets or privileges.

Do not remove the entire ACL. Limit the change to restoring the incorrect rule and ordering to the approved original policy. Then verify PC-A's request and reply, PC-B's previously working traffic, and guest traffic that must remain blocked. Confirming that only intended traffic is restored shows that business recovery preserved existing denials.

Compare the same request from PC-A and PC-B at D1's ingress and egress. After locating the block on PC-A's request, restore only the incorrect ACL and retest existing permitted and prohibited traffic.

Test one hypothesis with one minimal change, and record the result↑

Before acting, briefly record observed facts, your hypothesis, the change domain, and expected results. State which log or capture evidence justifies the change. Changing several settings at once makes it hard to tell what caused the problem and which changes were unnecessary, even if connectivity returns.

Preserve the current configuration and required state, and confirm change authority and who can roll back. Keep recording what changed and when, even in an emergency. Uncoordinated changes by another operator can alter the conditions you are comparing mid-investigation.

Here, restore the one verified ACL difference to the working baseline and repeat the same request. Confirm passage at the expected point and a returning response, while observing errors and load. If the expected result does not occur, revisit the hypothesis and rollback conditions before changing more settings.

If you choose a temporary bypass, record remaining capacity, security restrictions, and its expiration. Separately assess whether it can carry the full normal demand over time. Assign an owner and exit criteria for reversal or a formal change so a forgotten workaround does not become the normal configuration.

In recovery from a security compromise, an older configuration is not automatically safe. Verify that recovery uses a trusted baseline and backup from before the compromise, and complete necessary remediation and account or credential actions. Rolling back an ordinary connectivity fault and reconnecting a compromised system cannot use the same approval criteria.

Packet forwarding, business recovery, and incident closure are different↑

One successful ping does not finish recovery verification. Distinguish restored packet round trips, completed user work, and resolution of causes and recurrence risks. Record each stage's owner and evidence separately to state accurately what has recovered.

For network recovery, check the expected paths in both directions, policy, errors, and remaining load. For business recovery, test real requests from affected users and locations. Connectivity from a test endpoint does not verify other VLANs, wireless segments, or new access; check those separately.

Check existing and new sessions, required name resolution, and authentication against service needs. A seemingly normal recovery can fail later as load builds, leases renew, or routes return. Choose observation duration and follow-up checkpoints according to the incident's impact and the service's behavior.

At closure, record the timeline, actions, impact, direct cause, contributing conditions, and remaining risk. If the cause is still unknown, name the follow-up investigator and open questions. Go beyond assigning blame: specify configuration and process changes and retests that can prevent recurrence.

Investigate differences between business-owner confirmation and technical observations. A test request can succeed while a particular user's permissions or save operation still fails. Separate verified from unverified functions so partial success is not reported as full recovery.

Update design documents and the baseline with the paths, policies, and failure domain confirmed during the incident. Add missing tests to the validation list. Revisit security placement to see whether access controls and detection actually did their jobs, and improve containment points and recovery procedures for the next incident. The closing article follows a web request again to bring communication principles, design, and operations together.

We distinguish resumed packet forwarding, restored user operations, and completed cause analysis and recurrence prevention. Compare what has been verified at each stage with the work still open.