Fiber Optic Network Restoration
These are guidelines for designers and users of fiber optic communications networks as well as contractors and techs who install, operate and maintain them. What problems can cause transmission loss in a fiber optic network? What problems can be anticipated in planning fiber optic networks? How does the user determine the cause of the problem? How is the best and fastest way to restore communications? You should also download a copy of the NECA/FOA 301 fiber optic installation standard as a reference.
All networks are susceptible to problems that affect communications. A consequence of fiber optic systems’ high bandwidth, long distance capability and security is the extreme dependence of users on the non-stop operation of these systems. They can transmit large amounts of data long distances with immunity from signal degradation and extremely high reliability, so these systems usually carry the most critical data. By critical data, we’re not talking telephone conversations, Internet access or TV signals (although some viewers would question that conclusion during the Super Bowl) we’re talking about utility grid monitoring and control signals, surveillance CCTV systems, traffic control systems, airport monitoring and security, and the like.
If something happens, it’s important to not panic. You should have a plan, components set aside for repair and knowledgeable people on stall or a contractor on call. When you design the network, restoration planning should be part of the design. If you did not do that, now is the time to do it before something happens.
What Can Happen?
Network outages can occur from several problems. While most techs first think the problem is in the cable plant, the network consists of not only that, but also the communications equipment that operates over it. Troubleshooting needs to determine first whether the problem is the communications equipment or the cable plant before one can begin restoration.
Problems with the communications equipment that can cause outages:
<![if !supportLists]>· <![endif]>Problems in electrical power to the equipment
<![if !supportLists]>· <![endif]>Failures in incoming communications links to the communications equipment
<![if !supportLists]>· <![endif]>Failed communications modules in the equipment
<![if !supportLists]>· <![endif]>Connection problems to the cable plant (connectors or patchcords)
Here are some scenarios for cable plant damage that can cause outages:
<![if !supportLists]>· <![endif]>Underground cable damage from construction dig-ups or directional boring.
<![if !supportLists]>· <![endif]>Flooding or ice in winter, in manholes, hand holes, or poorly sealed splice closures.
<![if !supportLists]>· <![endif]>Aerial cable damage from shooting, animals, weather (ice or storms) and vandalism.
<![if !supportLists]>· <![endif]>Damage to utility poles in vehicle accidents affecting aerial cables.
<![if !supportLists]>· <![endif]>Cutting the wrong cable when removing older cables, indoors or out.
<![if !supportLists]>· <![endif]>Breaking patchcords or connectors when doing moves, adds or changes.
<![if !supportLists]>· <![endif]>Getting connectors dirty or not cleaning dirty connectors properly.
<![if !supportLists]>· <![endif]>Misconnections or polarity mismatches – transmitters must connect to receivers.
<![if !supportLists]>· <![endif]>Breaking fibers in splice trays or kinking cable tubes in splice closures.
Underground cable dig-ups
In outside plant (OSP) fiber optic installations, the biggest cause of network failure is likely to be either electronic problems with communications systems (including power outages) or, if it’s in the cable plant, what is usually called “backhoe fade” for buried cables and “target practice” for aerial cables, both of which are self-explanatory.
Aerial cable damage from gunshots and a squirrel. (Photos by S. Casey, City of Albany, GA)
Cables in premises installations are unlikely to be dug up accidentally but are susceptible to damage when any personnel are working around the fiber optic cables in trays or conduit. With the current push by landlords to remove abandoned cables to comply with the NEC, the likelihood of damage is much higher as installers cut out the old cables.
Inside telecom closets and computer rooms, it’s possible to damage cables, patchcords and connectors when equipment is moved or connections changed, especially by untrained personnel. Also likely is getting polarity wrong on duplex fiber links, connecting transmitter to transmitter and receiver to receiver by swapping fibers.
Sooner or later, the end user will likely be faced with restoring the system. By planning for restoration from the beginning of the project, the impact of problems can be minimized. These are general guidelines for restoration planning and execution. Each network is unique, but this can be used as the basis for a complete plan based on any individual system.
Designing For Non-Stop Operation As Well As Restoration
Efficient fiber optic communication network restoration depends on rapidly finding the problem, knowing how to fix it, having the right parts and getting the job done quickly and efficiently. Like any type of emergency, planning ahead will minimize the problems encountered. And in the case of fiber optic network restoration, nothing is more important than having complete, up-to-date documentation on the network.
If possible, design a network with backup options. Many users run dual links, one transmitting data and one “hot back-up” ready to switch over in milliseconds. Electronics must be installed with duplicate links and all power must be backed up with batteries, generators or fuel cells.
Critical systems should add in geographic diversity; two links available running paths that are as widely separated as possible to ensure that if one suffers a failure due to damage to the fiber optic cable plant itself, the other can be switched in immediately. Rings provide a logical way to have route diversity, but simply being able to patch fibers manually to switch over to another fiber/cable is still quicker than repair. Even with backup, a failure requires immediate restoration, as one should never depend on a single link any longer than necessary.
All cables should have spare fibers, especially since fiber is extremely inexpensive compared to installation or restoration costs. Fibers tend to get broken at the ends where terminated or inside splice closures during splicing or re-entry. Having spare fibers makes it easy to simply switch fibers to restore operation. Whenever possible, store extra cable in service loops that can be pulled together for splicing. This can save immense amounts of restoration time for cables installed indoors or pulled in conduit outdoors.
OSP underground cables should be buried sufficiently deep (~1m/3 feet) that it is protected from casual digging and marker tapes that show up on cable locators buried above them. (See OSP Underground Construction in the FOA Guide) Bright colored conduits also help visibility. Cables should be listed in the “Dial 811, Call Before You Dig” database and markers installed where possible.
Marker signs for buried utilities including fiber optic cables
Many users install indoor fiber optic cables inside bright orange “innerduct” to protect it. The additional initial cost of the innerduct may be offset by the simplification of the installation saving worker time. Patch panels where backbone cables are terminated should be enclosed and even locked if possible, to prevent damage by unqualified personnel. Patchcords may be exposed and susceptible to damage, so replacements should be readily available.
Documentation Is Extremely Important
The biggest single help in troubleshooting starts with producing good documentation during the installation and keeping it current. Documentation is the most helpful thing you can have when trying to troubleshoot a fiber network. Include in the documentation:
Start with the manufacturer’s datasheets on every component you use: electronics, cables, connectors, hardware like patch panels, splice closures and even mounting hardware. Along with the data, one should have manufacturer’s “help line” contact information, which will be of immense value during restoration.
Mark Every Component And Keep All Test Data
During installation, mark every fiber in every cable at every connection and keep records using cable plant documentation software or a simple spreadsheet of where very fiber goes. If possible, attach tags identifying the cable as a fiber optic cable and where the ends are terminated. When tested, add loss data taken with an optical loss test set (OLTS) and optical time domain reflectometer (OTDR) data when available. Someone must be in charge of this data, including keeping it up to date if anything changes.
For the electronics, one should have data on the optical power at transmitters and receivers. If that data is not taken during installation and setup, typical data should be available in the equipment manuals.
Mapping The Route
Outside plant cabling should have maps and photos detailing the routing of the cable, with GPS locations if possible – everybody refers to this as GIS data for geographic information systems. For premises cabling, drawings of the building noting all cable runs, again with photos if possible, are needed. One needs lists describing the types of cable on each run, installation hardware and test data for restoration to facilitate identification. Knowing where every cable goes will keep you from blindly searching for the cables when you try to locate problems. Having original test data will make it much easier to find bad cables because you have installation data to compare with.
Restoration Equipment And Supplies
Rapid restoration requires having the right tools and components to repair the cable break. This includes:
Every system needs some basic test equipment for troubleshooting. Every work crew should have a connector inspection microscope with magnification of 100-200X and fixturing for proper connectors to inspect for connector dirt or damage. Video microscopes are recommended. The plastic caps on connectors protect the ends of the connector ferrules but are inadequate to keep the connectors clean, so connectors should be inspected and cleaned before testing or connecting to equipment.
Any optical loss test set (OLTS) should have a power meter to use to test the optical power of signals in the transmission link at the receiver first then the transmitter, to determine if the problem is the cable plant or the transmission equipment. Total failure of all fibers in the cable plant usually means a break or cut in the cable. For short lengths of OSP cables or premises cables, finding the location is often simple if you have a visual fault locator or VFL. The VFL is a bright red laser coupled into the optical fiber for visual tracing or fault location over distances of several kilometers. Coupling the VFL light into the cable allows testing continuity and polarity visually, important since fiber optic transmission systems use infrared light invisible to the human eys and sometimes harmful to the eye if viewed directly. If there is a break or cut, the VFL laser light should be visible at the location the fiber is cut and allow locating the damaged point. VFLs also can be used to find bad connectors at patch panels.
OTDR and VFL used for troubleshooting
For longer cables, an OTDR will be needed. Outside plant networks should use the OTDR to document the cable plant during installation, so during restoration a OTDR test comparison of installation with current traces will usually help find problems. OTDRs generally do not have adequate resolution for short cables, say less than 30-50 meters, so a VFL will be needed. OTDRs can also find non-catastrophic problems, for example when a cable is kinked or stressed, so it only has higher loss, which can also cause network problems. Remember that OTDRs measure fiber length, not cable length, which is usually 1-2% shorter than the fiber due to the excess fiber in the cable, so there will be uncertainty in the location of the fault shown in the OTDR trace.
Cracked fiber in splice tray found with VFL and kinked aerial cable found with OTDR
Every test set should have cleaning supplies available. Every cable end should be cleaned before testing and before reinsertion. Dirty connectors are a big problem in fiber optic networks but are unlikely to be the cause of a sudden system failure. Fiber optic components are designed to keep dirt out after termination, so if a component becomes dirty, it’s usually because it is opened to outside air or contaminated by human intervention, e.g. being touched by a finger. Needless to say, fiber optic components should never be disturbed after installation and/or testing.
Video microscope and cleaning kit
Tools and Supplies For Restoration
Once you find the problem, you have to repair it. Repair requires having the right tools, supplies and trained personnel available. Besides the test equipment needed for troubleshooting, you need tools for splicing and termination, which should include a fusion splicer for outside plant cables.
Components For Restoration
You also need fiber optic components matching those used in the cable plant. It is always recommended that for every installation, a reasonable amount of excess cable and installation hardware left over from the installation be set aside in storage for restoration. Some users store the restoration supplies along with documentation in a sealed container ready for use. Remember that the fiber optic patchcords that connect the electronics to the cable plant can be damaged also but are not considered repairable. Just keep replacements available.
Leftover cable from an installation should be carefully stored for use in restoration
What else besides cables and cable plant hardware should be in a restoration kit? You should have a termination or mechanical splice kit and proper supplies. For splices, you need 1 or 2 splice closures of the proper type with adequate space for a number of splices equal to the fiber count in the cable. All these should be placed in a clearly marked box with a copy of the cable plant documentation and stored in a safe place where those who will eventually need it can find it fast. And you need to know where to find the test equipment you need.
Spares are also needed for the electronics. If many of the same types of equipment are used, several spares should be stocked. Remember that fiber optic network equipment may have a short lifetime due to upgrades to faster speeds, while fiber optic links tend to be used for years or even decades. If spares are not kept available for restoration, replacements may not even be available, making keeping spares even more important.
Any supplies that have a shelf life (termination or splicing adhesives, for example) should be noted and alert systems provided to ensure timely replacement.
Preparing For Restoration
Personnel must be properly trained in fiber optics including how to use the equipment and do the troubleshooting and restoration. And, of course, they must be available on a moments notice. The biggest delay in restoring a fiber optic communications link is often the chaos that ensues while personnel figure out what to do. Having a plan that is known to the responsible personnel is the most important issue.
Besides the tools and test equipment noted, the user needs the same types of equipment used in the installation, for example trenching equipment to be able to dig up cables or bucket trucks to work on aerial cables. Large users will have such equipment available but smaller users may find this is another reason to have arrangements with an installation contractor who will do the restoration work.
Optional Arrangements For Restoration
Major users of fiber optics have restoration plans in place, personnel trained and kits of supplies ready for use. It’s doubtful that most users are ready for such contingencies. Smart users work with their contractors doing fiber installations to create restoration plans, noting the contractor can assist in the planning and must be available on short notice for such work when needed. The end user may find that the cost of having all this equipment is not economic, so relying on a contractor is more cost effective. It may be preferable to keep an inexpensive test set consisting of a VFL and OLTS at each end of the link and having an experienced contractor on call for restoration.
The first step in troubleshooting is to determine if the problem is the fiber optic cable plant or the communications system electronics. A sudden loss of communications may indicate the loss of the link, which can be caused by either the transmitting or receiving electronics or the cable plant. Intermittent problems are much harder to diagnose but can usually be traced by monitoring equipment.
Beginning troubleshooting, review the Restoration Plan and documentation to see what the network designer determined to be the most expeditious method of determining the problems and created a checklist for solving them, plus it will be much faster and easier with documentation covering the routing of the cable plant and information on the system as installed.
Also check to see if anyone has been working on parts of the network who might have caused the problem. Many systems problems are caused by personnel doing moves, adds and changes to the network. They may change patchcords to the wrong equipment, reverse polarity, or even damage patchcords. Technicians working on the cable plant adding drops or new cables in a splice closure may misconnect fibers or damage fibers while doing their work. So before running around troubleshooting, determine if anyone has been doing work on the network. Of course, the network manager should know if any work is being done and has ensured that personnel working on the network are properly qualified.
Look for clues. If all the network links are down, does the electronics have power? If it does, all the links using one cable plant may indicate a catastrophic cable failure. If only one link of many is down, it’s either the electronics, the patchcords or one or two fibers which have been damaged, typical when a worker damages fiber in a splice closure. Remember that many links self-test and if one fiber is bad, it will take the link offline to ensure that no optical power exists in a fiber that could be viewed by a human and potentially damaging their eye.
Always begin by checking the electronics – the communications equipment - to quickly determine if there is a failure there.
<![if !supportLists]>· <![endif]>Check the power and the backup power supplies
<![if !supportLists]>· <![endif]>Link status lights, usually green is good and red is dead
<![if !supportLists]>· <![endif]>Check connectors and patchcords – inspect and clean connectors
<![if !supportLists]>· <![endif]>Check optical power at receivers and transmitters
<![if !supportLists]>· <![endif]>Electronics problems will generally require you swap with spares – you do have spares, don't you?
<![if !supportLists]>· <![endif]>Remember to reset modules – many systems require a reset after replacement to reboot the electronics
Since patchcords are usually open to the outside world, they are one of the first things to check. Visually check for kinks or stress on any patchcord. Use a VFL to see it the patchcords transmit light. If not, test and/or replace them and check the link. If good, check the electronics next.
Note: Patchcords are often a problem. Patchcords purchased on price alone are often of poor quality, even counterfeit, and cause problems. Patchcords should be tested if the performance is questionable.
Like all electronic systems, it’s important to first determine that all electronic devices have proper power available, are turned on and in the correct operating mode. If the equipment has a self-test mode that is used for diagnostics, it should be determined that the failure is not that the test mode has been inadvertently chosen rather than the operating mode.
Operating range of optical power for a link
Troubleshooting electronics is straightforward; start at the receiver and measure the optical power, which may require putting the remote transmitter into a test mode. The power at the receiver is the first piece of data you need. If the power level is correct, the transmitter is transmitting and the fiber in the cable plant is good for that link, so the problem is likely a data problem. If the power is bad, you must test the remote transmitter to see if its power level is correct, in which case the problem is the cable plant, or if transmitter power is low, you have a bad transmitter that needs replacing. If both are OK, but the link communication is bad, you have a data problem and the e fiber optic electronics needs troubleshooting.
Troubleshooting The Cable Plant
<![if !supportLists]>· <![endif]>Nothing helps troubleshooting like documentation
<![if !supportLists]>· <![endif]>Documentation should include location of cable plant features including manholes, pedestals, splice points, service loops
<![if !supportLists]>· <![endif]>GIS data preferred
<![if !supportLists]>· <![endif]>Cable and fiber IDs and test data from end to end to compare with tests during restoration
<![if !supportLists]>· <![endif]>Splice closure diagrams if needed to resplice fibers
Begin by visual inspection of all sections of the cable that are available for viewing. Send someone to follow the route of the cable plant and look for any signs of physical damage, especially evidence of construction near the cable or kinking, bundling, or other possible stress at the ends. If any suspicious areas are noted, clear the faults and test the loss of the cable plant. If the cable plant has been damaged by a dig-up, underground cable will usually show the damage at street level. Also check for directional drilling or even microtrenching along the cable route. Even microtrenched cables can be damaged if someone else tries microtrenching without knowing where other cables are near the surface.
If the power testing says the cable is bad, try testing a premises cable plant with the VFL for continuity if it’s short or an OTDR for longer cables. If one fiber is bad, test to see if others in the cable are bad. If all are bad, a cable failure is indicated. If only a few fibers are the problem, the issue may be damage at termination or splice points and one should be able to switch to spare fibers. But only a few fibers may be damaged is a cable is dug-up or kinked, so you should still locate the fault. Whenever personnel re-enter splice closures or change patch cables, the possibilities of making mistakes that cause problems are raised, so make sure only trained personnel enter hardware.
If the cable is bad, the location of the problem must be found. Often the quickest way to find the problem is simply looking along the cable route for damage or workers who may have caused the problem. Driving along the route of OSP cables often finds the problem before test equipment can be located and set up. On short cables, VFL tracing may find the problem. On long cables, an OTDR must be used to locate the problem. Be wary of using OTDRs on short cables as “ghosts” in the trace can confuse the location of real problems.
When you start OTDR testing:
<![if !supportLists]>· <![endif]>Start testing the fiber on the link that is down
<![if !supportLists]>· <![endif]>Look for the end, compare to the expected length from documentation to see if you have a break
<![if !supportLists]>· <![endif]>If you locate a break in the fiber, test other fibers in same cable to see if there is a cable problem
<![if !supportLists]>· <![endif]>Locate faults from nearby features like splice points – short distances on OTDRs will have less uncertainty
<![if !supportLists]>· <![endif]>OTDR data is not accurate: ~ +/-2%, but you can shoot from both ends on long runs and average
<![if !supportLists]>· <![endif]>Aerial problems may be hard to spot, use OTDR averaging to get best estimate of location and use GIS data to locate area then carefully inspect visually
Here is a link to a more detailed troubleshooting guide.
Faulty electronics can be replaced quickly if spares are available. If several units of similar types are in use, having a spare or two is easy to justify. If no spares are available, using replacements from less-critical links may get the more important system operational quicker. Remember that fiber optic network equipment undergoes fast evolution, usually to faster speeds, while fiber optic links tend to be used for years or even decades. If spares are not kept available for restoration, replacements may not even be available, causing major disruption while the network is redesigned around newer equipment.
Restoration sites may not be in convenient locations, as midday in the desert or a big city at night.
If the problem is a cable cut or damage, sometimes it may be more expeditious to do a temporary repair, getting only enough fibers operational to restore communications, allowing more time for a complete, neat, permanent repair. For many years, that role was assigned to simple mechanical splices and a few meters of cable. However, if the repair tech has a fusion splicer, the easier to install and more reliable fusion splice is recommended.
Short cables, especially in a premises application but often outdoors too, may be easier to replace than repair. Patchcords should be replaced from spares, but it may be a good idea to check them for continuity or loss – and inspect and clean them - before replacing the defective ones.
Cable cuts or breaks generally can be spliced or re-terminated. In the outside plant, the splices will usually be fusion splices just like the initial installation and sealed in a new splice closure. Sometimes in premises cabling, temporary mechanical splices placed in an enclosure can used, the broken fibers can be terminated with connectors or a new cable can be pulled, whichever is more efficient.
Splicing with and without adequate slack from service loops.
One big problem is pulling the two cable ends close enough to allow splicing them together. If you have service loops with extra cable, you may be able to release sufficient length of cable to pull the ends together and join the cables with one splice closure. You need quite a bit of cable, including cutting off the damaged cable ends sufficiently long to ensure no damage was done the fibers at that point plus about 2 meters of cable on each end to strip the cable, splice the fibers and place them in a splice closure, plus sufficient cable to pull together for splicing, which is the reason service loops are recommended for all cables. If the cable ends are too short, you have to splice in a new section of cable, which hopefully has been kept from the leftovers after the initial installation, requiring two separate splices and splice closures. On singlemode cable, 10 to 100 m of cable is necessary to prevent modal problems with transmission.
Once the cable plant is repaired, it should be tested to confirm the repair. If the repair is temporary, arrangements for protection of the components should be made immediately until permanent repairs are possible. If the repair is permanent, all components should be returned to the original state immediately.
Once the restoration is complete, it is important to prepare for the next time a problem arises, including updating documentation and replenishing supplies.
The documentation should be updated to reflect changes in the cable plant after repair. This includes any new components, new splices or splice closures, or especially any fibers that are no longer serviceable. New test data (loss measurements and OTDR traces) should be recorded, preferably compared to pre-restoration data for comparison. Inventories of restoration supplies which are left over and or are reusable should be updated. Required supplies should noted and ordered and inventories updated when received.
Updating and Replenishing Restoration Supplies
As soon as possible, all supplies used in a restoration should be replaced with proper supplies and components. Inventories should be updated to reflect new quantities. While awaiting replenishment, restoration kits should be marked as incomplete.
A final word, no, a warning, is needed. Fiber optic cable plants do not generally need maintenance! Inspecting and/or testing them often involves bringing the network down, exposes components to contamination and is more likely to cause damage than provide useful information or prevent failures. Install fiber optic cable plants, preferably have patch panels behind locking doors and leave them alone, unless a failure occurs.
There are hundreds of pages of technical information that can be used as references for the planning, design and installation of a fiber optic network. These documents can be downloaded from the FOA Guide Table of Contents.
In addition to those, we recommend these FOA textbooks available from Amazon.com:
The FOA Reference Guide to Fiber Optics
The FOA Reference Guide to Fiber Optic Testing
The FOA Reference Guide to Fiber Optic Network Design
The FOA Reference Guide to Premises Cabling
The FOA Reference Guide to Outside Plant Fiber Optics
FOA Online Reference Guide, FOA website, www.thefoa.org
NECA/FOA-301 Standard For Installing And Testing Fiber Optic Cables (NECA/FOA-301), Download free from FOA website
There are 3 FOA lectures on fiber optic restoration.
Lecture 51 covers the types of damage to a fiber optic network that will require restoration.
Lecture 52 covers planning for restoration - how to prepare for restoring a cable plant after damage occurs.
Lecture 53 covers how to troubleshoot problems and make repairs as fast as possible.
Note: This information is provided by The Fiber Optic Association, Inc. as a benefit to those interested in designing, manufacturing, selling, installing or using fiber optic communications systems or networks. It is intended to be used as an overview and guideline and in no way should be considered to be complete or comprehensive. These guidelines are strictly the opinion of the FOA, and the reader is expected to use
them as a basis for creating their own documentation, specifications, etc. The FOA assumes no liability for their use.
The Fiber Optic Association, the international professional society of fiber optics, has available on its website, www.foa.org, guides for end users on fiber optic network design and installation. The FOA also has a website offering free online self-study programs, www.fiberu.org.
( C ) 2019-20 The Fiber Optic Association, Inc.
The Fiber Optic Association, Inc.
1-760-451-3655 Fax 1-781-207-2421
Email: firstname.lastname@example.org http://www.foa.org