ProSoundWeb Community

Sound Reinforcement - Forums for Live Sound Professionals - Your Displayed Name Must Be Your Real Full Name To Post In The Live Sound Forums => AC Power and Grounding => Topic started by: Adam Ellsworth on June 03, 2014, 07:42:19 PM

Title: Recording/monitoring power surges?
Post by: Adam Ellsworth on June 03, 2014, 07:42:19 PM
Hello!

I've been working in a brand-new venue for about 6 months and in the last couple weeks I've had a few suspicious equipment failures, including a few "affordable" moving light fixtures, a laptop power supply, and the last straw being a nice Ashly 70v amp... none of which were on surge protection. Rack gear I'm less worried about but things like moving lights seem impractical to put on a surge protector (unless it's not unusual to install whole-distro surge suppression similar to a whole-house protector?)

Either way, I'm mainly curious if anyone has any experience trying to confirm/document power issues as opposed to simply deploying protection. I realize in the grand scheme of things my options are probably limited even if I go to the power company and say "Hey - we spiked to 150 volts at 3am one day".

A Google has found devices like this for around $500
https://www.powermonitors.com/product/detail/eagle-120

... as well as fancy test gear that starts around $12,000 and measures harmonics and a host of things I won't understand.

Just curious if anybody's played with power data logging, I feel like there must be a group of people on The Internet who go love that stuff.
Title: Re: Recording/monitoring power surges?
Post by: Adam Ellsworth on June 03, 2014, 07:47:49 PM
I should add... most of the time this gear is NOT connected to power. My distro has voltage meters on each phase and I've never observed any unusual voltages - I have a high confidence in the building's power because it's new, well-planned, and my feed is barely 25 feet from the building main disconnect... but that doesn't eliminate transient spikes coming from outside. (Is "transient spike" redundant and repetitive?)
Title: Re: Recording/monitoring power surges?
Post by: Stephen Swaffer on June 03, 2014, 08:26:13 PM
How new is the venue?  "Depending" on a lot of issues-mainly the fact that electricians are human at best-it is possible for connections to work loose through thermal stress.  A few months ago I replaced a name brand breaker that melted down due to a faulty factory connection-the breaker had been in service less than a year .  It happens-thermal imaging and retorquing connections are reasonable preventative maintenance.  Personally, I would start with having an electrician double check connections.  Depending on the size of the venue that might be the least costly option.

Do you have a new "neighbor" that might be introducing noise to the power line?  Especially on a shared transformer or transformer bank?  If that is a possibility, I would ask if the POCO would run some tests-I have done this before .  They usually have the equipment, as well as the expertise to interpret what they see.  Making the measurement is only half the battle-you have to understand what it really means. 

If you can get your hands on a DMM that has the ability to record a high/low voltage , it might pay to leave that connected whenever equipment is powered up a few times.   
Title: Re: Recording/monitoring power surges?
Post by: Tim McCulloch on June 03, 2014, 09:55:32 PM
Hello!

I've been working in a brand-new venue for about 6 months and in the last couple weeks I've had a few suspicious equipment failures, including a few "affordable" moving light fixtures, a laptop power supply, and the last straw being a nice Ashly 70v amp... none of which were on surge protection. Rack gear I'm less worried about but things like moving lights seem impractical to put on a surge protector (unless it's not unusual to install whole-distro surge suppression similar to a whole-house protector?)

Either way, I'm mainly curious if anyone has any experience trying to confirm/document power issues as opposed to simply deploying protection. I realize in the grand scheme of things my options are probably limited even if I go to the power company and say "Hey - we spiked to 150 volts at 3am one day".

A Google has found devices like this for around $500
https://www.powermonitors.com/product/detail/eagle-120

... as well as fancy test gear that starts around $12,000 and measures harmonics and a host of things I won't understand.

Just curious if anybody's played with power data logging, I feel like there must be a group of people on The Internet who go love that stuff.

"Suspicious" equipment failures will be more relevant once documented to be power-related.  Things get wonky when under-voltage, too, but it usually doesn't do permanent damage.

Of the failed items you mentioned, I suspect the laptop PSU and affordable moving lights to have universal (95-250v 50/60Hz) power designs.  If you have a failing neutral connection the furthest the voltage could swing to any phase leg is 240v or 208v (single phase/3 phase), within the capability of the device.  If they were 120v only, my bet is off...

That said, has the venue had other equipment fail?  Cash registers?  Server ordering stations?  Ticketing?  Internal LAN network equipment?  HVAC control?  If so, Stephen's suggestion to involve the power company should be immediately considered.
Title: Re: Recording/monitoring power surges?
Post by: Mike Sokol on June 03, 2014, 10:58:25 PM
Tim, on a side note about your signature. I do know my early Genesis...

Chewing through your wimpy dreams
They eat without a sound,
Digesting England by the pound.

Title: Re: Recording/monitoring power surges?
Post by: Mike Sokol on June 04, 2014, 11:26:01 AM
That said, has the venue had other equipment fail?  Cash registers?  Server ordering stations?  Ticketing?  Internal LAN network equipment?  HVAC control?  If so, Stephen's suggestion to involve the power company should be immediately considered.

Also be aware that I've seen a few "new" installations with the ground and neutral swapped at the receptacles. That would not be found by most casual testing, nor would it affect most stand-alone gear. But cross connect audio gear with XLR cables and plug them into G-N reversed outlets, and some awfully large "ground" currents will flow during peak system draw. I've postulated before that this might be able to do some gear damage, and I've actually trashed a few power amps in active speakers experimenting with steady-state ground loop currents over 3 amperes.

Don't know if that's your problem, but I'm just throwing it out there for discussion. Has anyone on the forum seen any swapped G-N connections? If so, how did you find them? And was any gear damaged by it?
Title: Re: Recording/monitoring power surges?
Post by: Tim McCulloch on June 04, 2014, 11:39:13 AM
Also be aware that I've seen a few "new" installations with the ground and neutral swapped at the receptacles. That would not be found by most casual testing, nor would it affect most stand-alone gear. But cross connect audio gear with XLR cables and plug them into G-N reversed outlets, and some awfully large "ground" currents will flow during peak system draw. I've postulated before that this might be able to do some gear damage, and I've actually trashed a few power amps in active speakers experimenting with steady-state ground loop currents over 3 amperes.

Don't know if that's your problem, but I'm just throwing it out there for discussion. Has anyone on the forum seen any swapped G-N connections? If so, how did you find them? And was any gear damaged by it?

True, all of this.  Adam, is everything that failed powered from your distro or are they plugged into Edison outlets that are part of the venue service, or both?
Title: Re: Recording/monitoring power surges?
Post by: Kevin Graf on June 04, 2014, 02:40:23 PM
For line monitoring, you might look into the TED system.

http://www.theenergydetective.com/home

********************************

An EE writes about the TED system.

http://www.beta-a2.com/engry_b.html

*********************************
Title: Re: Recording/monitoring power surges?
Post by: Adam Ellsworth on June 05, 2014, 02:49:43 AM

Thanks everyone for various thoughts/suggestions... I think my biggest takeaway here is that it's not unreasonable to escalate this to the power company and see if they'll investigate it. Even if I waited 3 weeks to catch a problem myself, they'd still probably do their own monitoring first to confirm it.

Adam, is everything that failed powered from your distro or are they plugged into Edison outlets that are part of the venue service, or both?

Most was on my distro which is tied into a dedicated 100amp disconnect, attached to the building switchgear... if neutral/ground were reversed I almost think it would be irrelevant. (Not "correct" by any definition, but they're effectively bonded at the other end... Still, I feel like this comment might start a flame war.) Also the problems happened during times where overall load was trivial, so that probably eliminates insufficient neutral. I usually check the meters anytime we're hitting things pretty hard and never noticed voltage issues. We were just programming some LED lighting one evening and noticed two lights went out and the laptop wasn't charging anymore...

I'm not aware of issues elsewhere in the building but all network components are on surge/UPS... most devices including phones are POE, and the POS terminals are probably pretty rugged. We've had a LOT of trouble with light bulbs burning out in the event hall but they're fancy halogens and the prevailing theory is loud music vibration is killing them. I'm not sold.

The Ashly amp going out is the most troubling. It's one of a pair that share a standard dedicated outlet. One is fine and the other goes into over-current protection on all 8 channels even if nothing's connected; even channels that have NEVER had a load attached. Could just be a bad egg. The amp brags about multiple levels of protection, but specs say 120VAC +/- 10%, which isn't a ton of wiggle room. (Also troubling is I've left voice mails for Ashly on Saturday and Wednesday and haven't heard back... must be vacation week?)

So anyway, I'll talk to the owners about getting Xcel involved. Thanks!
Title: Re: Recording/monitoring power surges?
Post by: Stephen Swaffer on June 05, 2014, 07:44:43 AM

Also the problems happened during times where overall load was trivial, so that probably eliminates insufficient neutral. I usually check the meters anytime we're hitting things pretty hard and never noticed voltage issues. We were just programming some LED lighting one evening and noticed two lights went out and the laptop wasn't charging anymore...


Ah-the REST of the story :).  Evening typically means a lighter load on the grid as well-which might allow transients to be worse or voltages to creep up.  I once dealt with a situation where we were running 460 V  VF drives on "480" that actually ran 485-490 most of the time,  460 + 10%=504 so not any wiggle room left.  I got called on several Monday mornings to fix a fault-the fault was always overvoltage. We ran 3 shifts and the issue only happened over the weekend with nothing else running.  In our case the fix was decidedly low tech-throw the knife switch when production was done for the week-the same thing I did on Monday morning to "fix" the problem, just felt avoiding the over voltage was smarter in the long run.

Just make sure Xcel knows the whole story-when troubleshooting you can very rarely have too much info.   
Title: Re: Recording/monitoring power surges?
Post by: Mike Sokol on June 05, 2014, 07:46:59 AM
Most was on my distro which is tied into a dedicated 100amp disconnect, attached to the building switchgear... if neutral/ground were reversed I almost think it would be irrelevant. (Not "correct" by any definition, but they're effectively bonded at the other end... Still, I feel like this comment might start a flame war.)

You're correct that is EVERYTHING is powered from a reversed G-N connection that there should be no damage. But the point that was made that if you cross connect gear with XLR cables that have been plugged into correct and incorrectly wired G-N, then large currents can flow into circuit boards during peak loads. Been there, seen that, smoked the gear. Don't worry about flame wars here. I'm not going to allow it. We're all learning and you would not believe the things I've seen built by electricians and engineers who should know better.
Quote

I'm not aware of issues elsewhere in the building but all network components are on surge/UPS... most devices including phones are POE, and the POS terminals are probably pretty rugged. We've had a LOT of trouble with light bulbs burning out in the event hall but they're fancy halogens and the prevailing theory is loud music vibration is killing them. I'm not sold.
 

Surge protectors are overrated solutions since they really only clamp voltage spikes and rely on a solid ground to work at all. So a high impedance ground can actually channel a spike from nearby lightning or collapsing magnetic field opening up right into the ground of your gear. And I don't buy into the loud music killing halogens either. That could be the smoking gun.
Quote

The Ashly amp going out is the most troubling. It's one of a pair that share a standard dedicated outlet. One is fine and the other goes into over-current protection on all 8 channels even if nothing's connected; even channels that have NEVER had a load attached. Could just be a bad egg. The amp brags about multiple levels of protection, but specs say 120VAC +/- 10%, which isn't a ton of wiggle room. (Also troubling is I've left voice mails for Ashly on Saturday and Wednesday and haven't heard back... must be vacation week?)


Sadly, most service centers are swap shops, so even if they do the repair they probably won't do a component level diagnostics. They'll just swap out the entire circuit board which won't give you any solid info on what the failure mechanism might be.

I do suggest you borrow or purchase some kind of circuit analyzer such as a Amprobe INSP-3 or SureTest Analyzer. They can provide a lot of information about ground impedance, voltage droop under loads, etc... I find myself doing this more and more for troubleshooting.
Title: Re: Recording/monitoring power surges?
Post by: Adam Ellsworth on October 06, 2014, 04:00:42 PM
I know this is an old thread but I thought I'd post a follow-up for the Google archives...

The building owner contacted the power company and they (unknown to me) attached a monitor for 18 days (!) and then sent us a dozen graphs that their engineer had reviewed. (Peak/Min/Avg for Voltage, frequency, and current for all three phases plus neutral.... I was glad to see detail.) There were a couple minor events, all brief sags, and the engineer had noted causes for each (one was severe storms, another was a phase outage "elsewhere on feeder") Ultimately they determined everything was within normal but I did feel like they took it seriously at least. It was actually interesting to look at a couple weeks of activity.... there are predictable voltage curves at various times every day and week to week. Phase 1 and 3 stayed between 119 and 125 for the entire period; On day 6, Phase 2 became much more exaggerated (between daytime/nighttime) and barely squeaked in between 115 and 126 for exactly 7 days - and then was perfectly normal again. Maybe it was just a hot week.

BTW since then I programmed my computer's UPS Battery to write voltage to a spreadsheet - all I have is voltage but my residential power is FAR more stable, usually varying only a couple degrees in a 24-hour period.

We've had no other equipment failures but a number of light bulbs burned out one night in the restaurant just a couple weeks ago. I'm surprised there's not a whole-building surge unit like I've seen other places, and they don't seem extremely expensive, so I might recommend they pursue that.

.... so there you go.
Title: Re: Recording/monitoring power surges?
Post by: Scott Holtzman on October 06, 2014, 04:26:45 PM
Adam, Something still smells wrong.  That testing was done on the building entrance.

The gear that was damaged, especially the laptop charger.  Two questions

1 - Was it a two prong or a three prong plug?
2 - Was the charger wide range (like 90-250V) or just 110Vac?




Sent from my To Be Filled By O.E.M. using Tapatalk