ProSoundWeb Community

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 2 [All]   Go Down

Author Topic: dsp failure  (Read 7603 times)

Craig Hauber

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 580
    • http://www.csaproductionsinc.com
dsp failure
« on: September 05, 2009, 12:15:21 pm »

These things have been around long enough now and they are definitely an "all your eggs in one basket" type device.
Just had a first generation Audia fail in a large college type auditorium and have several BSS omnidrives with burnt-out display backlights and failed front-panel buttons (they are still holding their settings so I don't touch them) And I have many nightclubs with Ashly protea 24.24's that don't open up until too late in an evening for me to get there in time to fix if something fails.

How do you handle this type of thing?  You just can't say screw'em when someone calls in a panic.  
Is there a method to switch to backup unit that can be semi-automated?  

What plan do you all have for handling this.

In the old days stuff would slowly start to fail or give warning signs for months -and even when something died it may have only effected a pair of speakers out of many.  When these DSP's quit it's very sudden and the entire facility goes silent.  
Logged
Craig Hauber
CSA Productions Inc.
Ventura-Santa Barbara CA
www.csaproductionsinc.com

Ivan Beaver

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 9010
Re: dsp failure
« Reply #1 on: September 05, 2009, 12:47:16 pm »

We don't supply backup DSP's as standard-but could if the customer wanted to pay for it.

Since there is no way of telling what could go wrong, the most failsafe way is to have two identical DSP's and a switchover system consisting of relays that are in the normally closed state.

You switch all the inputs and outputs to a new DSP with a switch that engages the DSP's.

You want a complete switchover.

Simply paralleling signal paths could have a problem if the particular input or output devices fail and short the line-causing the line to be "dead"-even if a good device is hooked up to it.
Logged
For every complicated question-there is a simple- easy to understand WRONG answer.

Can I have some more talent in the monitors--PLEASE?

Ivan Beaver
dB Audio & Video Inc.
Danley Sound Labs

Craig Hauber

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 580
    • http://www.csaproductionsinc.com
Re: dsp failure and thoughts on redundant system design
« Reply #2 on: September 05, 2009, 03:16:29 pm »

That was one thing I was pondering.  Yes I can easily supply a second identical DSP, but how does a customer switch it over?  Moving 30 phoenix connectors probably isn't going to happen (knowing my customers!) So I was thinking of the parallel operation but with relays to remove the dead unit from the chain in case of loading issues on the lines.

Some other thoughts:

Do you leave the backup dsp on and running or fully disconnected from the mains until needed?  -saves it from mains power related failure but requires someone to plug it in.
-and yes, I've seen dsp's do really strange things due to block-wide power spikes and surges while the amps didn't seem to mind -especially in large construction areas like these new mall -almost-a-city -type of complexes I've done system installs in.

Do you run the primary dsp 24-7-365?  -I tend to.  -and to solve the problem above I've started using UPS's -just for the dsp as some analog gear doesn't like the square-wave output of most units.  But this then makes you wonder how good a UPS is since it contains lead-acid batteries.  Should they be cycled occasionally?  Do they risk bursting and causing other severe damage in a rack, what about power outages -does the UPS shut down gracefully when it's batteries die un-attended or do they brown out?  How about when power is restored? they might cause a worse spike/bounceback than not having it at all.

Relays,- what are the contact life on these things if they aren't used in a warm dry dusty rack for several years then suddenly called on to engage.  Anybody have a good signal-grade relay that they recommend?  (there's 1000's to choose from in the Mouser catalog!)

I know it's impossible to have a perfect system but when your selling an entire 2nd DSP to your client as "insurance" I would hate for the system to remain dead when the emergency plan is activated.

-And yes, I know amps could die too but they tend to be more robust and there's multiples of them that the odds against complete failure are better.  (something that would take them all out would hurt other systems in the place too such as the video rig, architectural dimmer controller or the POS system)
I also stagger channels on amps so that not one amp is doing an entire band (so if 1 dies they don't lose all the highs with only subs remaining)

Sorry for the rambling post, just putting thoughts down as they pop into my head.  I find this stuff very interesting.

Logged
Craig Hauber
CSA Productions Inc.
Ventura-Santa Barbara CA
www.csaproductionsinc.com

Ivan Beaver

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 9010
Re: dsp failure and thoughts on redundant system design
« Reply #3 on: September 05, 2009, 05:57:14 pm »

I would leave the 2nd DSP not exactly powered up.  Plug its AC into say a Lowell RPC 20 AC line switcher that is switched on when you switch the switch to swap the DSP's.  It will take a little bit for the DSP to power up.  If that time is not acceptable, then you will have to leave it on.

Now how the customer is going to figure out the DSP is bad and not a cable somewhere or a switch on the console is a totally different matter Laughing .

If you use sealed relays or reed switches the contacts will remain fine-but you will pay a price- literally Shocked

I have probably installed 800 or so DSP's of all types.  If you take the Behringer fying bacon issue out of the count, I think we have only had 2 or 3 DSP failures.  And those have all been discontinued models-(maybe there is a reason they are discontinued Rolling Eyes )

It all boils down to how much redundancy you want.  There are all kinds of other places that can go bad also-what about your console for example?
Logged
For every complicated question-there is a simple- easy to understand WRONG answer.

Can I have some more talent in the monitors--PLEASE?

Ivan Beaver
dB Audio & Video Inc.
Danley Sound Labs

Chad Johnson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 285
Re: dsp failure and thoughts on redundant system design
« Reply #4 on: September 05, 2009, 06:39:57 pm »

I've never spec'd a redundant DSP system.

I suggest using units with no displays for starters, but that's not always practical. Unless the job or client requires it, I always spec the same brand.

Coincidentally I had a DSP fail in April. I was able to run one of my own over and drop it in and load the files without a hitch.

I always have one of those DSP units in my own rack and I always keep the system files on disc in a file as well as on my programming tablet. Its not hard for me to pop in a replacement unit and load the files in very short notice.
Logged
Digital consoles are so 2006. The real future is in steering bass arrays.

Brad Weber

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 2477
Re: dsp failure and thoughts on redundant system design
« Reply #5 on: September 06, 2009, 08:15:02 am »

Craig Hauber wrote on Sat, 05 September 2009 15:16

Do you run the primary dsp 24-7-365?  -I tend to.  -and to solve the problem above I've started using UPS's -just for the dsp as some analog gear doesn't like the square-wave output of most units.  But this then makes you wonder how good a UPS is since it contains lead-acid batteries.  Should they be cycled occasionally?  Do they risk bursting and causing other severe damage in a rack, what about power outages -does the UPS shut down gracefully when it's batteries die un-attended or do they brown out?  How about when power is restored? they might cause a worse spike/bounceback than not having it at all.

I typically use online, sine wave output UPS units in front of SurgeX devices for the DSPs, network switches, etc. in a system.  If there is a control system, the UPS is tied into that via serial or network communication so that an orderly shutdown can be initiated if the UPS runs down too much.  By putting the UPS first it does put that at risk during a surge but you can shut down the system via the SurgeX and still be fully protected.
Logged
Brad Weber
muse Audio Video

Don Boone

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 782
Re: dsp failure and thoughts on redundant system design
« Reply #6 on: September 07, 2009, 03:44:33 pm »

Brad is there a reason for putting the UPS upstream of the SurgeX?
Seems like the SurgeX should come first.

Don
Logged

Ivan Beaver

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 9010
Re: dsp failure and thoughts on redundant system design
« Reply #7 on: September 07, 2009, 04:43:53 pm »

Don Boone wrote on Mon, 07 September 2009 15:44

Brad is there a reason for putting the UPS upstream of the SurgeX?
Seems like the SurgeX should come first.

Don

Actually there should be a SurgeX before and after the UPS.

Having it after would "catch" something if the UPS went belly up and did weird stuff.
Logged
For every complicated question-there is a simple- easy to understand WRONG answer.

Can I have some more talent in the monitors--PLEASE?

Ivan Beaver
dB Audio & Video Inc.
Danley Sound Labs

Craig Hauber

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 580
    • http://www.csaproductionsinc.com
Re: dsp failure and thoughts on redundant system design
« Reply #8 on: September 08, 2009, 12:53:30 am »

I've searched around and they are either really expensive (more than the DSP itself) or really huge (or both)  Any recommendations on brands and models?  
I'm looking for something to keep a DSP afloat long enough to weather on/off surges, spikes and brownouts -not endless uptime (450 to 700VA square-wave units give me enough time but don't like the relay bypass on-offline and as mentioned earlier really despise the square-wave action.
I also would like ones that can handle the production-audio end of my business too (portable and don't complain about road-case and transit vibration)

thanks again

Logged
Craig Hauber
CSA Productions Inc.
Ventura-Santa Barbara CA
www.csaproductionsinc.com

Brad Weber

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 2477
Re: dsp failure and thoughts on redundant system design
« Reply #9 on: September 08, 2009, 11:56:00 am »

Ivan's right that before and after would be ideal and if it is simply an inline surge suppressor and a UPS that are always live then it would make sense to put the surge suppressor first.  However, locating a sequencer or any switched power device before the UPS does not usually work well.  For example, you might want to put a media server, digital mixer, DSP and some external effects devices on a UPS, but want to sequence those on and off.  Or you may want the control system and system DSP off the UPS but with the control system always on and the DSP being switched on and off.  You can't easily do either of those with the UPS after the sequencer or switched power as not only would that require multiple UPS devices, one per sequenced circuit, but when a circuit turn off it would also cause the UPS to engage and run the equipment off the battery until the battery drained rather than turning the equipment off.  Adding to this, now the battery is drained and the next time you turn on the system you have to hope you have no problems until it had sufficiently charged.
Logged
Brad Weber
muse Audio Video

Tim Padrick

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 5008
Re: dsp failure and thoughts on redundant system design
« Reply #10 on: September 13, 2009, 03:56:17 pm »

You could use several of these for switching: http://www.radialeng.com/re-sw8.htm

If enough units were involved, you might get then to build one with a higher channel count, and sans the unnecessary transformers.  (Customer requests were what prompted the development of the SW8 - and many other products I'm sure.)

Craig Hauber

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 580
    • http://www.csaproductionsinc.com
Re: dsp failure and thoughts on redundant system design
« Reply #11 on: September 16, 2009, 04:01:54 pm »

Excellent idea.  I never realized something like that existed and I would have never thought to look at Radial for something like that either!

You could have the primary DSP generate a test-tone to an unused output that could trigger the box.  Tone goes away with a failure causing the switch -it can even trigger an AC relay to power up the standby unit.

You wonder what type of design robustness was considered.  A concert piece isn't designed for 24-7 uptime in a rack forever.  -But on the other hand can probably handle vibration, crappy power and powering up/down all the time better because of it's touring heritage.

Pity on the connectors though, phoenix would be awesome!  It does look like Radial could probably make you some db-type multipins to just bare-wire fanouts to connect this into your DSP's and amps.

Thanks for the idea

Logged
Craig Hauber
CSA Productions Inc.
Ventura-Santa Barbara CA
www.csaproductionsinc.com

Tim Padrick

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 5008
Re: dsp failure and thoughts on redundant system design
« Reply #12 on: September 18, 2009, 10:40:36 am »

Craig Hauber wrote on Wed, 16 September 2009 15:01


Pity on the connectors though, phoenix would be awesome!  It does look like Radial could probably make you some db-type multipins to just bare-wire fanouts to connect this into your DSP's and amps.


We use Horizon for such cables.  Good product, good price, good service.  (Nothing against Radial - quite the contrary.)

Joules Newell

  • Newbie
  • *
  • Offline Offline
  • Posts: 21
    • http://www.pureinstall.co.uk/
Re: dsp failure and thoughts on redundant system design
« Reply #13 on: November 24, 2009, 03:22:43 pm »

BSS Omnidrives all loose the backlight eventually. I've replaced tens of them, but the unit itself nearly always continues to work

Your worse nightmare is the green range of BSS Soundweb. They have a fan failure issue which causes one of the caps in the PSU to dry up. when this goes the 5 volt rail drops and the unit will tend to lock out any external function. Shortly after, it will die completely until the fan and cap are replaced. I must have replaced over 50 soundweb PSU fans.

Usually the cap will fail quite a few months after the fan fails, so if you can spot it with regular inspections you will only need to swap the fan.

BSS have known about this issue for over 4 years now, and did change from bearing fans to mag-lev fans. However I have had mag-lev fans fail too.

Good system design when using more than one DSP will make your system robust. Do things like run LEFT on one DSP and RIGHT on the other. That way with a few XLR link cables the venue staff can get the other side back on (or MONO switch on the back of the amps)

This is one of the areas where a good installer will provide a better designed system and leave the customer less prone to failure.
Logged
I get about a bit!

1999-2001 HSL Productions
2001-2003 Community Loudspeakers UK Tech support
2003-2006 Lynx Lighting
2006-2008 CVA / ESL
2008-today Pure Install
http://www.pureinstall.co.uk/

Martin Thorpe

  • Newbie
  • *
  • Offline Offline
  • Posts: 6
Re: dsp failure and thoughts on redundant system design
« Reply #14 on: December 13, 2009, 09:37:38 pm »

Joules Newell wrote on Wed, 25 November 2009 07:22


This is one of the areas where a good installer will provide a better designed system and leave the customer less prone to failure.


very true!

although in 4-5 years of using a wide range of DSP's, i've only witnessed 2 failures, and i believe both were XTA... well one was a martin dx-1 but same thing i think...

unfortunatly most of our customers are unprepared to pay for a 2nd DSP(s) for failsafe, so this isnt a path i've ever looked at to be truthful

we have a lot of DSPs in nightclubs etc too, so we generally just keep a small rack of analoge equipment (EQ, compressor, crossover) on standby with some patch leads etc just in case, not always the most ideal replacement, but none the less, better than a dead DSP






Logged

Tom Bourke

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1568
    • http://www.cwalv.com
Re: dsp failure
« Reply #15 on: December 30, 2009, 11:11:47 am »

I tend to fall in the keep it simple crowd.  When I read about all the extra required to make a system redundant all I see are more failure points.  Most of the time I see failures in the external wiring or connectors pulled out.
Logged
I have a mild form of Dyslexia that affects my ability to spell.  I do use spell checking to help but it does not always work.  My form of Dyslexia does not affect my reading.  Dyslexics of the world untie! <a href="http://www.cwalv.com" target="_blank">http://www.cwalv.com</a>

Dave Sturzenbecher

  • Full Member
  • ***
  • Offline Offline
  • Posts: 136
Re: dsp failure
« Reply #16 on: January 05, 2010, 11:20:00 am »

Your options here can really open up if you use amps with on-board DSP.   I understand that this only works in new installations, but i know the Crown I-techs (HDs too) have an analog override feature,  where if your digital in goes down, the amp will take the analog input. (This can be fully automatic, or when analog signal is detected it over rides the digital) This analog input can be a direct line off your console.  Remember, in this analog backup mode....you are mainly concerned with getting through the show...  Again, as long as you have DSP in the amps to keep the speakers from frying your ok.  This works real well because your single point of failure is minimized.  As long as your mixer is still up and sending signal, and your lone XLR wire is good, you are making sound. This analog backup line directly from the mixer can simply be daisy chained to every amp. Even if you lose one amp, all the other are still working.

If your amp is not one of the $3000 ones your could rig up a similar setup if you can sum ch1 and ch2.  As long as these two channels are not electrically in parallel you have your DSP hit ch1, and your analog backup line from the console hit ch2. Keep your analog backup line switched off until you lose your DSP. Now this only works if you don't need your amp in stereo mode, and if you lose all your DSP.  You wouldn't want to sum an analog backup signal with a DSP signal that has some latency with it.  I am thinking Crown Xti, Cdi, Dci series here.  

The company I work for makes a unit that does the switching for us. ins and outs for each channel, which is a simple pass through in normal mode.  Then when switched to analog backup mode, a relay switches to a single (analog backup) signal that splits to all outputs.  It is very much a brute force method, but it works.
Logged
Pages: 1 2 [All]   Go Up
 

Page created in 0.115 seconds with 19 queries.