NEW HANOVER COUNTY -- Strong thunderstorms wreaked havoc on the Port City this weekend, cutting connections to some vital emergency services including 911.
New Hanover County's 911 Center lost six of its emergency telephone lines Saturday.
Telecommunications Director Brenda Hewlett says those lines are still trying to get fixed Monday.
Operators at the 911 call center say some telephone lines were out of service for more than 12 hours after lightning struck one of their towers.
Call center administrators say some of the computers' motherboards were fried and technicians had to wait until Monday morning to bring in new equipment.
In the meantime callers were asked to contact non-emergency numbers if they needed help. But despite the loss of several lines operators say each call that did come through was answered.
Hewlett said, "No one should have seen a difference in the way we handled our call volume -- just the units on the streets and how we had to dispatch them. We didn't lose power... We were still covered."
Hewlett says technicians have been working all day to restore damaged equipment.
The downed emergency lines should be back up and running by later Monday evening.
Telecommunications say in case of emergency you should be able to dial 911 without any problems. But if you can't through to an operator you're urged to call a non-emergency line at (910) 452-6120.
redundancy
i work for the county and can tell you that if its not within our budget we cant do it. do you want your taxes to go up so we can do this? what other recourse do we have? the county commissioners have the difficult job of distributing the money in the budget as best they can. if anyone out there wants to step into their shoes, there will be a couple of vacancies coming up this fall. maybe you can do better. also, anyone who was watching tv this weekend would have seen on the local channels a crawl that was continually on screen that gave the information on the lightning strike as well as the phone number to call. there is enough blame to go around-the county did the best it could to remedy the situation.
Two kinds of redundancy....
cptaco mentions that he/she works for the County. You talk about taxes going up in the process of trying to prevent this from occurring again. In my previous post, I mentioned a number of steps that can and should be taken to prevent lightning damage. None of what I talked about should be overly expensive to implement. Considering the fact that that the EOC is new, you're probably almost there. Just connecting a ground wire to something doesn’t mean it’s protected against the fast rise-time currents associated with lighting. There’s a wealth of information available on the subject.
With regard to the word "redundancy" that keeps being repeated:
There's two kinds of redundancy -
1. Geographical redundancy - AKA: Geographical diversity. This is where you would basically have two EOCs. Real expensive, and probably not necessary if the primary EOC is properly protected - weather, security, etc.
2. System redundancy - This is where a single failure doesn’t cause a serious issue. This is a common technique used all around us. Something you see every day, and don't even think about is the two traffic signals you always see facing you even on a two lane road. One red light fails, and you still know to stop. This kind of redundancy is already present at the EOC. Single telecom circuit failures cause no issue. Single power supply failures within the many critical systems there cause nothing more than alarms. So, they are already at a reasonable level of redundancy. Even the entire radio console system can fail, and they could still communicate to the field using portable radios, if necessary. All of this is considered redundancy.
What still needs to be looked at to prevent this from repeatedly occurring are the items covered in the documents I mentioned in my previous post. These items are neither difficult, nor expensive to implement. I’m not talking about duplicating equipment, buying different hardware, or anything else that could even remotely be considered drastic, or out of reach of the county. Every time this happens, it costs the county money in terms of destroyed equipment, contractors, and county manpower - let alone the liability issues. The precedent has been set that there are problems with the facility.
The county needs to talk with their peers in areas that have even more lightning than we do, and find out who their experts are. The recommendations may be surprisingly simple.
Can't afford it?
Perhaps if the county commissioners weren't giving away money like drunken sailors, they'd have more to invest in our infrastructure? I'd sure rather have a bit more redundancy at the Emergency Communications Center than really nice soccer fields.
telcom redundancy
With all due respect if you have worked telecommunication for a long time in Wilmington you would have known that at the old location there were two seperate legs, (One on each end of the cave taking it's path to the 911 center accross the street...)Ogranted you are correct that even a small line surge can blow out dozend of cards in a c.o.
However with correct clustering and two seperate legs run to a building, and in the case of yesterday... a generator that was properly monitored, tested and checked you would not have to use second site mirroring to achieve the goal. The only time you need dual sites is for cases of high dependancy/high availability in the event that an event of catestrophic devistation occurs at the first site facility. But in that... you are very correct. How can you place a price tag on the safety of tax payers? You would think that we are worth the extra bit of $$$.. Certainly the city of Wilmington could use some redundancy as well... Oooo.. Free co-location.. An IT strategy decades old. But that would make way too much sense...
I work on the wireless network
I've lost DS3s simply because the mux card at the CO failed and its hot standby partner didn't come on line. That's with no lightning within a hundred miles!
I guess I'm just trying to say regardless of what system we're talking about, a catastrophic failure is eventually going to happen. The Space Shuttle occasionally blows up. Elevators are idiot-proof, but they occasionally still fall and kill people. The 100% guaranteed redundant computers in the Eastern Airlines L-1011 that crashed into the Everglades failed to detect that both of them were programmed for standby/backup mode when it came to monitoring altitude....and neither alerted the pilot as he flew it into the ground.
The absolute, 100% fail-safe system has yet to be invented. That's why I don't understand some people are immediately jumping all over the PSAP, screaming negligence.
I'm sure that they will find things they could do better and can make improvements. If their generator isn't being auto-tested weekly, they're very foolish. But having seen what lightning can do, especially when it comes to arc-overs, I'd be reluctant to criticize anyone without clear evidence that they were totally unprepared and just didn't care.
New Hanover EOC lighting damaged..... again!
4411 talked about having contracts for the rapid replacement of failed equipment, and ITPRO spoke of redundancy. Both of these are valid points, and should certainly be requirements for a facility as critical as a county emergency dispatch center.
There are however much larger issues there that must be addressed. The fact that lightning was able to cause ANY issue at this facility, let alone causing damage that has still not been repaired 48 hours later should be enough to show there are underlying issues that must be resolved.
The statement commonly heard is along the lines of “nothing can protect against a direct lightning strike”. That is patently untrue. Look around outside, and you will see scores of radio towers - many of them for are for cellular service - but many are related to broadcast - Radio stations, TV stations, etc. These towers are subjected to direct lightning strikes on a relatively frequent basis. The vast majority of lightning strikes to these towers cause no issues whatsoever. The reason is that the correct combination of lightning protection techniques were implemented at these sites.
From what I understand, this is the second time that there was serious damage induced by lightning at the New Hanover EOC since last year. It’s been claimed that “lightning struck one of their towers”. That may or not be the case. It could have struck the building, the tower, the parking lot, or a tree out back. If all power and communications lines entering the section of the building that is considered to be the “911 center” were protected using proper techniques, there should have been no effect on equipment from lightning striking a communications tower some 300 feet away that just happens to be on the same site as the 911 center.
It appears that an attempt has been made at this facility to gain some of the protections offered by following NFPA 780 and/or UL96A( lightning protection system standards), but two issues stand out to me as a result of a simple drive past the building that indicate that work needs to be done. (aside from the repeated damage to the facility, Progress Energy replacing damaged transformer(again!), and a generator service crew having to convince a standby generator to operate)
1. There are antennas on the roof of the main building in the “lightning protected area” that are above the level of the air terminals (lightning rods).
2. The site - New Hanover County EOC, or any derivative of the name - is not listed as having a UL inspected and approved lightning protection system. (Try looking for yourself - http://www.ul.com/lightning/ .. find certified sites... Hint: There are quite a few in Florida) The county is pretty big on having things done by it’s residents inspected and approved. They have neither the qualifications nor the authority to self inspect with respect to lightning protection systems.
The other techniques that MUST be implemented to prevent recurring damage at this facility are detailed in ANSI / IEEE 607 (Commercial Building Grounding and Bonding Requirements for Telecommunications) They may have attempted some of these, but small mistakes can result in big problems.
They really need to get someone in there to fix the issues, and make some concrete rules about how to do things. Try talking to the EOC in Tampa, St. Pete, or Orlando FL. and find out who their experts are, and see what their experiences are being that they are in the lightning capital of the US. Two times in the two years the EOC's been open is a little ridiculous.
The carefully crafted words "But despite the loss of several lines operators say each call that did come through was answered.." shouldn't provide much comfort. Those six trunks that were not working were installed to handle possible 911 traffic. Can anyone say "busy signal"???
Don't I wish!
I'm an engineer for a cellular company, and while the radio equipment is fairly robust, we lose millions of dollars every year in telco interface and DC power distribution due to lightning strikes. It was the same at the previous carrier I worked for.
You can only boiler-plate so much. I'm not saying that everything at the county PSAP was hunky-dorey, and in an outage like this you always conduct a post-morten to find out what happened and how it can be prevented..... but I know all too well what lightning can do to a cellsite or MSC!
It all depends on where the lightning struck
You can build in 100% redundancy in all your operational circuitry, but if you sustain a direct hit, all bets are off. That's especially true if you have only one service point from Progress Energy, which is the norm. If you have a common AC service serving your primary and back-up equipment, an AC surge can take it all out. And NO surge protection is 100% reliable against a direct strike.
I work in telecommunications and have seen lightning take out every redundant circuit you have, sometimes causing destruction that requires quite a bit of time to repair. Thankfully, such mega-strikes are rare.
Bottom line, if you want 100% redundancy, guaranteed, then you're looking at a mirror image facility that's totally off premises, served by a different distribution leg and different telco central office.....and even then, I wouldn't rule out losing both facilities in one of those one-in-a-million situations.
Why doesn't the county 911 have 4 hour replacement ?
As an IT Director I pay extra for 24x7 by a four hour maximum response time hardware replacement from IBM and Dell. I find it hard to imagine this service is not available for the county and is only available for small business.
Local Goverment
Everybody needs to realize this is ran by the local government! Which means they run on a budget which means that everything goes to the lowest bidder. As cheap as possible.
And 24/7 coverage! Thats almost funny! They see that as a waste of money. Something they put money in and possibly wont get anything out of. IT takes money out of there budget.
We as the public just got lucky this time BUT it wont change anything. It will never change until someone dies as a result of there actions.
And how many people of the public knew to call the other number?!? Never heard about it and didnt even know there was another number.
Such poor mismanagement!
Why is there no redundancy?
Being in technology for fifteen years I can tell you it is pretty sad the county 911 doesn't have the redundancy in place to facilitate and or prevent an outage such as this. It is definitly available.........
Talk about minimal IT effort............