What do you do if your network goes down and you're the only engineer on duty?

Time is of the utmost importance, and our primary goal is to minimize any potential downtime. Upon confirming an issue, I will follow the established escalation communication matrix and promptly notify all node owners. Simultaneously, I will initiate isolation techniques using the Network Management System (NMS) to scrutinize for alarms, node failures, potential cyberattacks, recent modifications, and other relevant factors until the root cause is identified. This will provide me with an indication of the outage severity. I will take immediate action by providing a solution or workaround if the issue falls within my expertise domain. If not, I will provide remote access (if allowed) or on-site support until the appropriate personnel arrive.

See if the failover mechanism of switching network to other Service Provider is Auto-activated, if not, I will use manual intervention and ensure the service is restored and hardware/servers are up Then I shall contact the vendor to lodge a priority complaint and share the preliminary logs and analysis asking them to get involved and post us with regular updates. Thereafter I will follow the below Incident Management steps : 1. Assess the Situation: 2. Check Hardware 3. Review Logs 4. Isolate the Problem 5. Troubleshoot Methodically 6. Communicate: Keep stakeholders informed about the situation and provide regular updates 7. Seek Help if Needed: 8. Implement Temporary Workarounds 9. Document Everything 10. Perform Post-Mortem Analysis

Check the following: 1) recent configuration changes or planned works 2) recent physical site access logs 3) power outages 4) fiber outages 5) anomaly events (SW updates, External systems outages, Gaming or Streaming events ) This is covering 90% of the network outages.

, I'd swiftly assess the outage's scope, check configurations and hardware health, and investigate environmental factors or errors in logs. Employing temporary fixes if necessary, I'd communicate updates to stakeholders and escalate if unable to resolve independently, all while maintaining thorough documentation for future reference.

Well first and foremost your duty is to make sure you are NOT the only person on the NetEng team that knows about this. Once you have notified the CoC (Chain of Command) that there is a potentially serious issue you can work the problem. In order of priority. 1.) How bad is it? Total outage, partial outage, serious degredation of service(s). 2.) What applications and users are currently being impacted? 3.) Check of NetMon systems. What do your tools tell you is going on? 4.) Checking logs and device current status. 5.) Failure recovery. Can this be recovered within the timeframe agreed before BCP or DR steps kick in? 6.) Restoration of normal operations. 7.) Post mortem. 8.) Mitigation process/project including ITIL docu updates, etc.

1. Quickly assess the scope: Determine the extent of the outage 2. Check monitoring tools: Review monitoring tools, logs, and alerts 3. Perform basic troubleshooting: Run through a mental or written checklist of common issues 4. Isolate the problem: Use tools like ping, traceroute, 5. Take corrective action: Implement a fix or workaround based on your diagnosis. 6. Communicate with stakeholders: Keep management, colleagues, and affected users informed 7. Document the incident: Log the issue, actions taken, and resolution 8. Escalate if needed: If you're unable to resolve the issue, don't hesitate to reach out 9. Review and improve: After the incident, conduct a post-mortem analysis to identify areas for improvement - implement changes

Identifier le périmètre (géographique, services, personnes (VIP), ...) impacté par la panne Vérifications de base : alimentation électrique des équipements, branchements physiques, ...

One of the easier ways to figure out if there's an outage is just to Google outage in your area from your internet service provider. You can do this through your phone. Which always has a internet connection.

In the event of a network failure, I get an overview of the network failure, which floors, buildings, regions. I find out whether the failure is in my own network area or at the WAN router and whether there is a power failure. I try to reach someone on site.

Once you identify the root cause of the problem, implement fixes or workarounds to restore network connectivity as quickly as possible. network monitoring tools to gather more information about the outage. Check for any alerts or notifications that might indicate the cause of the problem. Begin troubleshooting the issue by checking network devices, cables, and connections. Look for any obvious signs of failure or misconfiguration. Keep stakeholders informed about the situation. Provide updates on your progress and any estimated time for resolution. If necessary, escalate the issue to higher-level support or management.

, I'd swiftly assess the outage's scope, check configurations and hardware health, and investigate environmental factors or errors in logs. Employing temporary fixes if necessary, I'd communicate updates to stakeholders and escalate if unable to resolve independently, all while maintaining thorough documentation for future reference.

Using the network diagram, I search for the nearest offline network device, connect to its online neighbor and examine the interface to the offline device and its own logging. I look at log files from offline network devices.

When the network is managed by multiple vendors, the key point is access to proper, up-to-date documentation. Based on it is sometimes easy to find in which vendor responsibility is the issue probably located. If it is not possible to identify location of the issue based on the documentation, only option is to open multivendor troubleshooting online session and identify possible root cause + it source/location.

If Layer 1 is cool, start at the bottom like running MTRs from source to destination and then reverse to look at the path to see if it's only you or a bigger issue like BGP. Then work your way up. 25 years and the lower levels I have found can be the root cause more often than not. Unless it's a LAN issue, then do pretty much the same.

The best way to find the roor cause of a network failure, is to cleary identify what is the exact symptom the network problem is causing. Generic sympton´s like everything´s is out do not let the people focus on fast finding the cause, since they have to check everything on the network. More precise sympton´s , let us focus on where the problem is. Like only the finance transctions to a specifc bank/card are not going through, wiche is very diferent of finance transactions are not working. Since is only one specifc bank or card, i can focus on that part of the network that provides this services.

Depending on the result of the analysis, various measures are required. The on-site contact may be able to reset the circuit breaker at the distributor, carry out a cold start of devices and check cable connections. It may be necessary to deploy a network technician on site with the required access, who must have suitable replacement optical transceivers and network cables with him, and possibly also a replacement device and measuring device, if known. I inform the responsible Engagement Manager, who can take over the customer information and documentation.

To test the network, you should check to see if the routers are all up with pings to and from each one. Test the firewalls for correct configuration and blocked/allowed IP addresses, and make sure there is internet connectivity. Finally, debug devices and check the logs to be sure there are no more issues.