ProSoundWeb Community

Sound Reinforcement - Forums for Live Sound Professionals - Your Displayed Name Must Be Your Real Full Name To Post In The Live Sound Forums => LAB: The Classic Live Audio Board => Topic started by: Robert McMillan Arthur on April 15, 2017, 12:29:15 PM

Title: What happened at Coachella on Friday?
Post by: Robert McMillan Arthur on April 15, 2017, 12:29:15 PM
Apparently, there was failure of FOH at the Radiohead show at Coachella on Friday (April 14th). Anybody know what happened?

https://www.yahoo.com/news/glitch-sound-fails-radiohead-coachella-003301055.html

I'm always interested in "lessons learned" after such things, because they are everyone's nightmare.
Title: Re: What happened at Coachella on Friday?
Post by: Ivan Beaver on April 15, 2017, 01:13:46 PM
Apparently, there was failure of FOH at the Radiohead show at Coachella on Friday (April 14th). Anybody know what happened?

https://www.yahoo.com/news/glitch-sound-fails-radiohead-coachella-003301055.html

I'm always interested in "lessons learned" after such things, because they are everyone's nightmare.
What I read (no idea if it is true or not) was that the console went down.

Title: Re: What happened at Coachella on Friday?
Post by: Milt Hathaway on April 15, 2017, 01:50:36 PM
What I read (no idea if it is true or not) was that the console went down.

At 46 seconds in on this video of the event you can hear what has been described elsewhere as the "bloop!" of the band's Avid S6L having a DSP error and shutting down. 26 seconds later audio is restored. I don't know enough about the S6L to know if this is a valid possibility.

https://www.youtube.com/watch?v=gp6bpQPJRgM
Title: Re: What happened at Coachella on Friday?
Post by: Jean-Pierre Coetzee on April 16, 2017, 12:20:19 AM
If the S6L is anything like the previous consoles there will likely be no audio dropped if there was an error.

Have had many errors on avid desks including a DSP card freaking out and the entire desk becoming unresponsive but audio still passed.
Title: Re: What happened at Coachella on Friday?
Post by: Erik Jerde on April 16, 2017, 09:43:35 AM
Apparently, there was failure of FOH at the Radiohead show at Coachella on Friday (April 14th). Anybody know what happened?

https://www.yahoo.com/news/glitch-sound-fails-radiohead-coachella-003301055.html

I'm always interested in "lessons learned" after such things, because they are everyone's nightmare.

As seen on the facebook.  Apparently the lesson is to pay attention to the warnings from your console.

Title: Re: What happened at Coachella on Friday?
Post by: Debbie Dunkley on April 16, 2017, 04:34:56 PM
As seen on the facebook.  Apparently the lesson is to pay attention to the warnings from your console.

Oops - sounds like someone is in big trouble....
Title: Re: What happened at Coachella on Friday?
Post by: Wayne Smith2 on April 16, 2017, 06:00:01 PM
Just curious, what would be the nature of the warnings?
Error messages perhaps?
Thanks
Title: Re: What happened at Coachella on Friday?
Post by: Scott Holtzman on April 16, 2017, 06:31:01 PM
Just curious, what would be the nature of the warnings?
Error messages perhaps?
Thanks


Like any computer the desk has error log output.  The AVB card connects digital signals to the console.  The errors could have been framing or bit errors on the link or other hardware related things like bus or memory issues.



Title: Re: What happened at Coachella on Friday?
Post by: Ray Aberle on April 16, 2017, 07:42:55 PM
I would HOPE it's in the form of error messages. "Caution, something's whack with AVB Card #Whatever; you should fix it, RTFN."

-Ray
Title: Re: What happened at Coachella on Friday?
Post by: David Sturzenbecher on April 16, 2017, 09:05:32 PM
I would HOPE it's in the form of error messages. "Caution, something's whack with AVB Card #Whatever; you should fix it, RTFN."

-Ray

One would hope.  I mix on an SC-48 with an AVB remote stagebox regularly.   There is absolutely no on screen indication of a primary or secondary link failure.
Title: Re: What happened at Coachella on Friday?
Post by: Robert Lofgren on April 17, 2017, 04:20:47 AM
I read somewhere else that it was due to generator problems because they were supplied three smaller generators instead of two beefier ones, even though the total supply was to be the same, and barfed when the electronics needed more peak power?
Title: Re: What happened at Coachella on Friday?
Post by: Ivan Beaver on April 17, 2017, 07:19:40 AM
I read somewhere else that it was due to generator problems because they were supplied three smaller generators instead of two beefier ones, even though the total supply was to be the same, and barfed when the electronics needed more peak power?
Sensitive things (like consoles) should be on a UPS.
Title: Re: What happened at Coachella on Friday?
Post by: Jeffrey Knorr - JRKLabs.com on April 17, 2017, 11:12:43 AM
Sensitive things (like consoles) should be on a UPS.

I think the UPS thing is over valued... Unless you have a very high quality UPS, you're probably just adding another failure point.  If you have a cheap UPS and it switches to battery power, are you going to make it through your gig?  Most modern equipment with well designed PSUs should be able to handle most situations that you'll run into.

Also where is the line drawn?  What about amps that have DSP built into them--we'd probably not put them on a UPS.  Intelligent lighting?  I'd rather recommend that people meter/monitor their power, make sure you use good generators, and invest in quality AC distribution.

Jeff "Not trying to pick on Ivan" K :)

Title: Re: What happened at Coachella on Friday?
Post by: Brian Adams on April 17, 2017, 11:51:30 AM
Also where is the line drawn?  What about amps that have DSP built into them--we'd probably not put them on a UPS.  Intelligent lighting?  I'd rather recommend that people meter/monitor their power, make sure you use good generators, and invest in quality AC distribution.

My opinion is that anything that anything that is constantly updated throughout the show and takes several minutes to boot up should be on a UPS. Audio consoles, lighting consoles, and computers without an internal battery, for example.

It's not about making it through the gig on battery power, it's about getting through brief interruptions without going down for several minutes while everything reboots.

Amps and DSP usually boot pretty quickly. Most consoles and computers don't. You can't (and shouldn't) put a UPS on everything, but having them on the sensitive components that take a while to boot makes sense to me. If you don't want to run UPS's on your equipment, that's your decision, but you're not going to talk me out of running mine.
Title: Re: What happened at Coachella on Friday?
Post by: Erik Jerde on April 17, 2017, 12:11:13 PM
My opinion is that anything that anything that is constantly updated throughout the show and takes several minutes to boot up should be on a UPS. Audio consoles, lighting consoles, and computers without an internal battery, for example.

It's not about making it through the gig on battery power, it's about getting through brief interruptions without going down for several minutes while everything reboots.

Amps and DSP usually boot pretty quickly. Most consoles and computers don't. You can't (and shouldn't) put a UPS on everything, but having them on the sensitive components that take a while to boot makes sense to me. If you don't want to run UPS's on your equipment, that's your decision, but you're not going to talk me out of running mine.

Agreed.  I put a UPS on anything that a power hiccup is going to force a reboot on.  Digital audio consoles, DSP units, video switches, video routers, etc.

With DSP in amps that blurs the lines.  Ideally you'd be able to mute the amps (and cut the high current draw as a result) whenever a UPS kicked in.  With the right setup you could accomplish this and for an install it might be useful.  However, as someone else said up thread having good power is a better option.  That said, I've experienced unexplained power blips in buildings that should have been golden.  Didn't have UPS in the racks then and it caused a little scrambling.  We've fixed that now.
Title: Re: What happened at Coachella on Friday?
Post by: Ivan Beaver on April 17, 2017, 12:12:24 PM
My opinion is that anything that anything that is constantly updated throughout the show and takes several minutes to boot up should be on a UPS. Audio consoles, lighting consoles, and computers without an internal battery, for example.

It's not about making it through the gig on battery power, it's about getting through brief interruptions without going down for several minutes while everything reboots.

Amps and DSP usually boot pretty quickly. Most consoles and computers don't. You can't (and shouldn't) put a UPS on everything, but having them on the sensitive components that take a while to boot makes sense to me. If you don't want to run UPS's on your equipment, that's your decision, but you're not going to talk me out of running mine.
And if there is a quick dip in voltage "hopefully" the UPS will be able to take care of that.

The cheap ones???  But one that constantly provides power will be fine and keep the console from rebooting or shutting down
Title: Re: What happened at Coachella on Friday?
Post by: Dave Garoutte on April 17, 2017, 12:25:47 PM
I also put the stagebox on UPS power.
I had the stage power go down once and when the stagebox came back on, there was a BIG pop.  I think anything that takes time to reboot should be protected from having to.  Finding the power issue and fixing it can be blamed on the house and you get to be the hero.  But waiting everything boots back up is always on you.
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 17, 2017, 12:45:50 PM
First, I take the blame for the Radiohead console failure.  I wasn't there and have nothing to do with the band but I was told today is my turn to be Designated Scapegoat, so there you have it. /sarc

Humor aside, I'm a big believer in UPS back up for *everything* that has a boot time of more than a few seconds.  Consoles, system DSP, lighting consoles, etc.

WWE has the biggest UPS I've ever seen in portable use (and they have generators and land line power with auto switching).  The UPS powers the primary production truck and the uplink truck for the 10 seconds it takes to accomplish the switch over.  The battery bank and inverter are both about the size of a large commercial refrigerator (and weigh considerably more).

If Vince McMahon can have a UPS, so can I!!
Title: Re: What happened at Coachella on Friday?
Post by: Scott Holtzman on April 17, 2017, 01:06:23 PM
First, I take the blame for the Radiohead console failure.  I wasn't there and have nothing to do with the band but I was told today is my turn to be Designated Scapegoat, so there you have it. /sarc

Humor aside, I'm a big believer in UPS back up for *everything* that has a boot time of more than a few seconds.  Consoles, system DSP, lighting consoles, etc.

WWE has the biggest UPS I've ever seen in portable use (and they have generators and land line power with auto switching).  The UPS powers the primary production truck and the uplink truck for the 10 seconds it takes to accomplish the switch over.  The battery bank and inverter are both about the size of a large commercial refrigerator (and weigh considerably more).

If Vince McMahon can have a UPS, so can I!!


I saw a similar rig at the RNC.  I believe it was a 90kva Liebert.  Small battery bank.  Maybe 30 minutes of run time.  Just enough to swap a genie out or do any other mains maintenance.



Title: Re: What happened at Coachella on Friday?
Post by: TJ (Tom) Cornish on April 17, 2017, 01:18:40 PM

WWE has the biggest UPS I've ever seen in portable use.
That's because the penalty for going down is being thrown in the ring.
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 17, 2017, 01:20:56 PM
That's because the penalty for going down is being thrown in the ring.

LOL.  I'd like to see someone throw either the battery bank or the inverter....  It was sketchy moving them with a 5000 lbs fork truck.
Title: Re: What happened at Coachella on Friday?
Post by: TJ (Tom) Cornish on April 17, 2017, 01:32:11 PM
LOL.  I'd like to see someone throw either the battery bank or the inverter....  It was sketchy moving them with a 5000 lbs fork truck.
At the day job (IT in a multi-story office tower) we had a 50KVA Powerware UPS with built-in 480v -> 120/208v stepdown transformer and two battery cabinets.  Each cabinet was a series string of batteries, providing a string voltage of 540 volts so the inverter didn't need to step up the voltage.  IIRC, the inverter cabinet and each battery string were ~3000lbs each and were right next to each other.  I'm fairly sure we were overweight for the office tower, at least considering the amount of floor under the unit.

Due to virtualization and "the cloud", it has since been replaced with a much smaller multi-string, tall dorm-fridge-sized 12KVA unit with internal batteries.
Title: Re: What happened at Coachella on Friday?
Post by: Steve Litcher on April 17, 2017, 03:04:58 PM
If this had been posted in the Lounge, the response from everyone would've been:

"I can't believe they didn't have a back-up console. Especially if it was spitting out error messages. I would've fixed it at first sign, thrown it in the trash, or had another console or two as a back-up."

(sorry, couldn't resist)

But seriously, it does seem odd that they didn't have some type of contingency plan in the event of an issue. I'd love to know if it was truly generator related or if it was an existing software glitch in the band's console.
Title: Re: What happened at Coachella on Friday?
Post by: David Allred on April 17, 2017, 03:27:13 PM
If this had been posted in the Lounge, the response from everyone would've been:

"I can't believe they didn't have a back-up console. Especially if it was spitting out error messages. I would've fixed it at first sign, thrown it in the trash, or had another console or two as a back-up."

(sorry, couldn't resist)

But seriously, it does seem odd that they didn't have some type of contingency plan in the event of an issue. I'd love to know if it was truly generator related or if it was an existing software glitch in the band's console.

How dare you.
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 17, 2017, 03:32:56 PM
If this had been posted in the Lounge, the response from everyone would've been:

"I can't believe they didn't have a back-up console. Especially if it was spitting out error messages. I would've fixed it at first sign, thrown it in the trash, or had another console or two as a back-up."

(sorry, couldn't resist)

But seriously, it does seem odd that they didn't have some type of contingency plan in the event of an issue. I'd love to know if it was truly generator related or if it was an existing software glitch in the band's console.

We're an Avid shop.  I've been inside a cranky SC48 and replaced faders on the Profile surface.  The error messages you DO get are helpful but seem to pop up right as the failure commences.  It's possible to recover semi-graciously from some errors but full recovery requires a soft reboot.  Errors from remote racks don't seem have early-warning popups.

No matter, though.  If this was genuinely a known condition the SE or BE should have been on the phone with Avid's excellent 24/7 support (they walked me through a hard drive replacement on the phone).  I'd think the hire company would have sent parts or a back up if they'd been asked, but I'm not privy to the business arrangements.
Title: Re: What happened at Coachella on Friday?
Post by: Nitin Sidhu on April 17, 2017, 04:23:34 PM
If this had been posted in the Lounge, the response from everyone would've been:

"I can't believe they didn't have a back-up console. Especially if it was spitting out error messages. I would've fixed it at first sign, thrown it in the trash, or had another console or two as a back-up."

Thats just mean.

My girlfriend told me bout this day before, she read up a uk gossip site or something.
She asked me what could've happened, and I said the Bloop and the Bleep sounds like the console may have gone down.

Her instant reaction was "They weren't carrying a backup?"... Im still cringing from those words..
Title: Re: What happened at Coachella on Friday?
Post by: Dave Pluke on April 17, 2017, 05:12:32 PM
I think the UPS thing is over valued... Unless you have a very high quality UPS, you're probably just adding another failure point.

Might one assume that if one forked out the cash for an S6L, one would also be able to afford something better than a "dime store" UPS?

That said, Eaton makes some nice rack mount UPS's that don't cost a fortune.

Dave
Title: Re: What happened at Coachella on Friday?
Post by: Nathan Riddle on April 17, 2017, 06:21:58 PM
Her instant reaction was "They weren't carrying a backup?"... Im still cringing from those words..

For those of us who are following along at home and don't understand why this is an issue (me).

Can someone explain?
Title: Re: What happened at Coachella on Friday?
Post by: Dave Garoutte on April 17, 2017, 07:31:07 PM
I'm not quite clear how a backup would have helped unless it was fully wired and redundant.
A backup console helps if you go down during setup, but during the show would be a major problem.
Title: Re: What happened at Coachella on Friday?
Post by: Chris Hindle on April 17, 2017, 08:10:55 PM
I'm not quite clear how a backup would have helped unless it was fully wired and redundant.
A backup console helps if you go down during setup, but during the show would be a major problem.
I had a 3K with 2 supplies go down during a show. Looked like the 1st supply blew, and took out the board. 2nd supply powered up, but the board was already toast. (Turned out it was something to do with Phantom Power that shut everything down. Desk was back in service in a week)
In 5 minutes I had my personal 2404 sitting on top of the 3K, another 15 minutes to put a 40 input show on 24, set rough gains, and on with the show.
Promoter was happy once we restarted, Patch Bitch was happy because I did it all from my end, Monitor Dude was happy because I didn't steal his board for FOH.
No, not fully wired and redundant, but in this digital age, with a digital snake, it shouldn't take more than 5 minutes to do what I did in 20. PLUS, You have a copy of the showfile. No fuking around with patches and gains.
Truly plug and go.
And yes, If it takes longer than 5 seconds to boot, it can't be allowed to shut down.
And yes, if the tool you depend on is throwing error messages, why the fuck is it still on the platform ?
Oh well.
Glad I'm not touring any more.
Chris. 
Title: Re: What happened at Coachella on Friday?
Post by: Riley Casey on April 17, 2017, 08:53:40 PM
What's really remarkable is how rare it is that this sort of thing happens, not that someone didn't have an identical spare console on line for instant change over on a festival.  My favorite crash dive call back in the days of large analog consoles was to provide a spare PSU cable. The tour had a top of the line console, dual supplies, shock mounted cases, one console supply cable and one short jumper between the supplies. What gets the most physical abuse twice a day on tour?  The cables.  How many tours carry spare one of a kind cables?  Not enough apparently.  When your mixing console is smart enough to tell you it's unhappy listen to it but the economics of the business don't make carrying a spare, dedicated online $100k mixing console a workable option.

For those of us who are following along at home and don't understand why this is an issue (me).

Can someone explain?
Title: Re: What happened at Coachella on Friday?
Post by: Mike Caldwell on April 17, 2017, 09:27:46 PM
A friend of mine is out on tour as the video tech with a B+ level country act and he says they carry a back up for the FOB and monitor boards. He said they use the same model Digico  at FOH and monitors and the back up system is the same model as well. He wasn't sure of the exact model number.
Title: Re: What happened at Coachella on Friday?
Post by: Michael Storey on April 18, 2017, 12:18:06 AM
What's really remarkable is how rare it is that this sort of thing happens, not that someone didn't have an identical spare console on line for instant change over on a festival.   ....

My sentiments exactly. Admittedly I'm looking up from the depths of the lounge here, but considering the logistics of a show this size, it's impressive these major failures don't happen more often.

But let's put the rumors to rest, Dave already detailed exactly what happened via one ingenuous tweet...
Title: Re: What happened at Coachella on Friday?
Post by: Lyle Williams on April 18, 2017, 07:35:50 AM
It is easy to throw stones with the benefit of hindsight.

Highly reliable digital system just stops.  20 seconds to come up with a plan - umm - restart it!

Then it stops again.  ****.  20 seconds to come up with a new plan - **** - replace it.

They say time flys when you are having fun, but the truth is that time never goes past as fast as when something is down and people are looking at you.



Title: Re: What happened at Coachella on Friday?
Post by: Keith Broughton on April 18, 2017, 07:52:06 AM
For those of us who are following along at home and don't understand why this is an issue (me).

Can someone explain?
If the console had completely failed and the show may have been cancelled or delayed for a long time, a backup console would be useful.
However, a failure to pass audio for even 2 or 3 minutes will not be solved with a back up console.
The rarity of these types of failures and the cost of carrying (or renting) a couple of $30,000 consoles may not be financially viable.
"My car broke down and I can't make it to work on time"
"Well, don't you have a back up?" ;D
Title: Re: What happened at Coachella on Friday?
Post by: David Allred on April 18, 2017, 09:18:25 AM
If the console had completely failed and the show may have been cancelled or delayed for a long time, a backup console would be useful.
However, a failure to pass audio for even 2 or 3 minutes will not be solved with a back up console.
The rarity of these types of failures and the cost of carrying (or renting) a couple of $30,000 consoles may not be financially viable.
"My car broke down and I can't make it to work on time"
"Well, don't you have a back up?" ;D

"A tornado destroyed my house.  Can I stay at your house?"
"Don't you have a back-up?"
Title: Re: What happened at Coachella on Friday?
Post by: Peter Morris on April 18, 2017, 09:30:59 AM
If the console had completely failed and the show may have been cancelled or delayed for a long time, a backup console would be useful.
However, a failure to pass audio for even 2 or 3 minutes will not be solved with a back up console.
The rarity of these types of failures and the cost of carrying (or renting) a couple of $30,000 consoles may not be financially viable.
"My car broke down and I can't make it to work on time"
"Well, don't you have a back up?" ;D

FWIW ... a 6L-32D Control Surface and E6L-192 Engine and Stage 64 Remote I/O has a MSRP of around $90K  :-\
Title: Re: What happened at Coachella on Friday?
Post by: Nathan Riddle on April 18, 2017, 09:31:17 AM
I'm not quite clear how a backup would have helped unless it was fully wired and redundant.
A backup console helps if you go down during setup, but during the show would be a major problem.

Quote
...and others...

Thanks! I just always thought on the larger gigs where they have enough money for the best gear that even with the insanely low failure rates they can afford to have a redundant console with near seamless switchover. I mean those shows where they have UPS's for backup while switching to generator power, etc. That's crazy to have redundancy on the power...why not a console too?

I get having backups on site not hooked up don't help. But having them hooked up and ready to swap (even if it is a few cables?) or even a smaller/different console?

(lounge level rubbish...) I mean I have a backup for my main console that I take on the bigger gigs. I might not be able to hot swap it. But at least I can say if the main console goes down I can still get something going within 15min or so *shrug*
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 18, 2017, 09:42:36 AM
Thanks! I just always thought on the larger gigs where they have enough money for the best gear that even with the insanely low failure rates they can afford to have a redundant console with near seamless switchover. I mean those shows where they have UPS's for backup while switching to generator power, etc. That's crazy to have redundancy on the power...why not a console too?

I get having backups on site not hooked up don't help. But having them hooked up and ready to swap (even if it is a few cables?) or even a smaller/different console?

(lounge level rubbish...) I mean I have a backup for my main console that I take on the bigger gigs. I might not be able to hot swap it. But at least I can say if the main console goes down I can still get something going within 15min or so *shrug*

I have seen tours with 3 consoles - FOH, Monitors, and HOT SPARE, set up on rolling racks ready to be wheeled into position.  All you do is move multi pin cables.  Still takes 5-15 minutes. The brand was not Avid....
Title: Re: What happened at Coachella on Friday?
Post by: Peter Morris on April 18, 2017, 09:59:09 AM
Thanks! I just always thought on the larger gigs where they have enough money for the best gear that even with the insanely low failure rates they can afford to have a redundant console with near seamless switchover. I mean those shows where they have UPS's for backup while switching to generator power, etc. That's crazy to have redundancy on the power...why not a console too?

I get having backups on site not hooked up don't help. But having them hooked up and ready to swap (even if it is a few cables?) or even a smaller/different console?

(lounge level rubbish...) I mean I have a backup for my main console that I take on the bigger gigs. I might not be able to hot swap it. But at least I can say if the main console goes down I can still get something going within 15min or so *shrug*

FWIW consoles such as Digico’s SD7 incorporates dual redundant mix engines and complete power and operating redundancy.  Similarly the Midas ProX has dual-redundant control computers, and three removable power supplies - only two of which are required for operation.

Most mid and large scale consoles have dual redundant snakes and power supplies.  The PSUs are typically interchangeable and hot swappable. 

Allen & Heaths dLive is interesting in that the mix engine is in the stage rack and will continue to mix even if the surface at FOH is removed. In desperation you can still mix with your laptop.
Title: Re: What happened at Coachella on Friday?
Post by: Art Nadelman on April 18, 2017, 10:07:01 AM
Allen & Heaths dLive is interesting in that the mix engine is in the stage rack and will continue to mix even if the surface at FOH is removed. In desperation you can still mix with your laptop.

Unless it's the stage rack that goes down.  Yikes!
Title: Re: What happened at Coachella on Friday?
Post by: TJ (Tom) Cornish on April 18, 2017, 11:44:30 AM
Unless it's the stage rack that goes down.  Yikes!
That's true, but the presumably the mix rack is near stage, so a cable cut or power loss between FOH and stage wouldn't take the show down.  The S-class racks have redundant power supplies, so are at least reasonably well protected from faults.
Title: Re: What happened at Coachella on Friday?
Post by: Stephen Kirby on April 18, 2017, 02:25:11 PM
In this day when everything is connected with Ethernet it would seem to be easier to swap things out than in the analog days.  I don't think anyone at the lounge level carries two X-32s with them but they might have an analog Mackie they could get though the show with.  Similarly I think you can run the D-Live stage box from a GLD.  Maybe not as powerful or as accessible to all the features but you can limp though.
The D-Live architecture does make a lot of sense.  Running everything out to FOH and back is kind of a legacy approach, no longer necessary with current technology.  With the reduced physical size having a standby available becomes more practical and processing costs a lot less than large cabinetry and encoders.  As mentioned, the control surface can be backed up with any number of computer interfaces, albeit with reduced control functionality, but the show will go on.  With network switches, any of this can be switched over quickly and only multi-pins from stage snakes have to be moved at the stage box.
Title: Re: What happened at Coachella on Friday?
Post by: Peter Morris on April 18, 2017, 06:17:06 PM
Unless it's the stage rack that goes down.  Yikes!

Yes but there is more less one less point of show stopping failure. Other designs will stop if you lose the surface or the rack, the dLive for want of a better description will operate in limp mode if the surface or its connection fails.

As Riley said above, its most often the cables and connectors that fail and this design eliminates some of these as a show stoppers.

It doesn't have the dual redundant mix engines like the SD7, only dual hot swappable PSUs, but its not the same price.
Title: Re: What happened at Coachella on Friday?
Post by: Brian Jojade on April 18, 2017, 08:23:01 PM
I don't think anyone at the lounge level carries two X-32s with them but they might have an analog Mackie they could get though the show with.

I do.  Using X32's as stage boxes means that if something goes down, I can re-patch and continue on my way, even if it comes down to iPad mixing as a last resort.
Title: Re: What happened at Coachella on Friday?
Post by: Geert Friedhof on April 19, 2017, 07:44:40 AM
Lounge leveler here.

I can go redundant with a racked X32 core, with separate wifi and DL16, stacked on the DL32. Always have a laptop synced at FOH. I covered everything as far as I can tell.

One thing I once had in the old analog days, and something I have never seen redundant: outboard failures. I once was mixing a band, performing for about 3000 ppls. Desk was a H2000, with nice outboard. Suddenly I had only whisper level music. After about 2-3 min we deduced that the Klark dn360 main eq had decided to commit harakiri. We pulled the inserts, and along we went. Needless to say those were the longest 3 min of my life.
Title: Re: What happened at Coachella on Friday?
Post by: Jim McKeveny on April 19, 2017, 10:02:48 AM
Schadenfreude.

Reluctantly went w/family to a Cirque show in Atlantic City a few years back. House console was on floor, center, ahead of bisecting aisle. Our seats were further back. I spied a Yamaha 4K at mix, w/an Instant Replay laying on unused channels to right.

 Mid-performance all audio goes dead. Onstage performers stop mid-step. Lots of side-eye/WTF? glances onstage, but no panic.

Soundguy starts slamming on Instant Replay with fist and whole audience is treated to appropriate-yet-family-unfriendly FOH cursing. Lots of weird digi noise/digi hunting....then music resumes. Onstage characters scrambled to match what now came through monitors and auditorium. Show finished w/o more incident.

After show I introduced myself to soundguy as a veteran NYC-based soundguy, and  I felt his pain. He told me "IT HAPPENS TWICE A WEEK!"

My young children asked "What did you think of the show?". I replied "For me? THAT was entertainment!"




Title: Re: What happened at Coachella on Friday?
Post by: Stephen Swaffer on April 19, 2017, 12:53:18 PM
No wonder the onstage perfomers took it in stride!
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 19, 2017, 03:06:15 PM
No wonder the onstage perfomers took it in stride!

I think that by the time an act is on a major festival stage they've experienced enough glitches, gremlins and failures that they know to not freak out, let their crew deal with it.  They know that shit happens and getting the show back up and running is already in process.  If the band treats the situation with grace (appropriate to genre) the audience is more likely to, also.

It's been maybe 20 years ago, but I was attending a small festival where stage power was lost so the band grabbed acoustic instruments and went out into the audience area and played there until the power was restored.  I bet they sold a ton of merch that night. :)

Obviously some acts can't do that - the nature of their genre or reliance on electro-centric technology to produce sounds that are non-acoustic in origin - but in smaller situations it can be 'lemonade'.
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 19, 2017, 03:08:28 PM
My sentiments exactly. Admittedly I'm looking up from the depths of the lounge here, but considering the logistics of a show this size, it's impressive these major failures don't happen more often.

But let's put the rumors to rest, Dave already detailed exactly what happened via one ingenuous tweet...

An unrepentant cable chewer, now in doggy time-out.  8)
Title: Re: What happened at Coachella on Friday?
Post by: Dave Garoutte on April 19, 2017, 04:26:11 PM
I love this SRV clip.
https://www.youtube.com/watch?v=wn1pk8Lpelc
How to deal with failure!
Title: Re: What happened at Coachella on Friday?
Post by: Stephen Swaffer on April 19, 2017, 06:32:34 PM
I think that by the time an act is on a major festival stage they've experienced enough glitches, gremlins and failures that they know to not freak out, let their crew deal with it.  They know that shit happens and getting the show back up and running is already in process.  If the band treats the situation with grace (appropriate to genre) the audience is more likely to, also.

It's been maybe 20 years ago, but I was attending a small festival where stage power was lost so the band grabbed acoustic instruments and went out into the audience area and played there until the power was restored.  I bet they sold a ton of merch that night. :)

Obviously some acts can't do that - the nature of their genre or reliance on electro-centric technology to produce sounds that are non-acoustic in origin - but in smaller situations it can be 'lemonade'.

I certainly agree that handling things gracefully puts everyone at ease-and is the best way to have a lapse "forgiven".  I was referring to the "it happens twice a week" comment-if that were me something would be in the dumpster before the next show.
Title: Re: What happened at Coachella on Friday?
Post by: Tim McCulloch on April 19, 2017, 11:03:17 PM
I certainly agree that handling things gracefully puts everyone at ease-and is the best way to have a lapse "forgiven".  I was referring to the "it happens twice a week" comment-if that were me something would be in the dumpster before the next show.

Too many unknowns but if the desk were crashing like this, 2x/wk AFTER consultation with the manufacturer I'd be sending it back for a full on exchange.

I'm not buying that part of the story at face value, though, because I know how touring at this level works.  If Clair or Sound Image or Spectrum sent this desk out and it failed, they'd be sending a replacement via FedEx Custom Critical or arranging for a substitute desk from a regional supplier until the replacement could be received.  That's the kind of support one gets at this level and if you don't, your PM or TM is firing the vendor.
Title: Re: What happened at Coachella on Friday?
Post by: Chris Hindle on April 20, 2017, 08:16:43 AM
Too many unknowns but if the desk were crashing like this, 2x/wk AFTER consultation with the manufacturer I'd be sending it back for a full on exchange.

I'm not buying that part of the story at face value, though, because I know how touring at this level works.  If Clair or Sound Image or Spectrum sent this desk out and it failed, they'd be sending a replacement via FedEx Custom Critical or arranging for a substitute desk from a regional supplier until the replacement could be received.  That's the kind of support one gets at this level and if you don't, your PM or TM is firing the vendor.

Guys, the 2xweek thing was the replay device, not the desk.
But why was the replay thingy still there if it's proving to be a regular show-stopper?

Maybe back to the actual OP question now ?
Chris.
Title: Re: What happened at Coachella on Friday?
Post by: Stephen Swaffer on April 20, 2017, 12:46:21 PM
Well, the replay thingy was giving the only warning it was capable of.  Apparently, the AVB card (at least via hearsay) was giving a gentler warning.  Both were ignored.

Lesson to learn from?

They say doing the same thing over and over expecting a different (better?) result is a form of insanity...

Of course, maybe they had changed something-which sometimes is all you can do with intermittent faults.  Been bit by that too many times myself.
Title: Re: What happened at Coachella on Friday?
Post by: Josh Hana on April 20, 2017, 08:01:26 PM
Avid console error messages are no joke.

I was on a Profile for an indoor arena show and I was getting "CPU over-temp" messages during load in. We're used to outdoor Miami weather and we handle it fine, an A/C arena gave me some concern.

I opened the FOH rack and found a big rubber band "securing" the sticks of RAM in place had snapped and gotten jammed in the CPU fan. Not a great design.

I've had a massive critical show-stopping error on a Midas Pro desk, but it actually presented similar to the "Bloop!" sound, but for a solid 4-5 seconds. I'd argue that's much worse than a cut in audio.

If it's true that they had received this error for the past few days and ignored it, then this deserves every bit of attention they got because that's a pretty big no-no.
Title: Re: What happened at Coachella on Friday?
Post by: John Lackner on April 30, 2017, 07:20:36 PM
What's really remarkable is how rare it is that this sort of thing happens, not that someone didn't have an identical spare console on line for instant change over on a festival. 

Back in the early days of festival touring (the early 70's) the likelihood of a major failure of the sound or lighting equipment was about 50% (you can ask John Meyer about this). Nowadays it's less than 1%. There is still no such thing as a piece of electronic gear that is 100% immune from failure.
Y'all SPOILED!!
Title: Re: What happened at Coachella on Friday?
Post by: dave briar on May 01, 2017, 10:02:19 PM
This lounge guy carries a pair of X32 Racks. Switchover would likely take five+ minutes but hey, it's just the lounge.

Now a good friend of mine is the BE for a national "New-grass" band.  Pro2 at FOH, SC48 for monitors, and a M32 core permanently mounted and wired in their splitter rack on stage. They mainly use the M32 core in lieu of the consoles for fly dates but if the Pro2 bricked it would just take firing up the core and router in the splitter rack and picking up an iPad to continue on -- with additional other "minor" adjustments likely required <wink>!
    ..db
Title: Re: What happened at Coachella on Friday?
Post by: Jim McKeveny on May 03, 2017, 08:05:48 AM
Sometimes, despite best efforts......shit happens.

I recall a Mellencamp show @ "The World's Most Famous Arena" where an MSG house power issue shut down the show. A World Series game had lighting issues. Another one had earthquake issues. D-Day was repeated postponed due to weather.

My point is: Let us not, from afar, cast stones at show personnel. The crew on this show are at the top of our game.
Title: Re: What happened at Coachella on Friday?
Post by: Jay Brett on May 03, 2017, 03:03:33 PM
A show a few weeks ago at The Sprint Center in Kansas City experienced a power failure on the lighting side. The show lost all lighting but still had audio. The headliner handled it perfectly by asking everyone to get out their lighters and they did a few songs acoustically why the crew quickly ran feeder to a secondary power source. Instead of being a major failure, it almost seemed like part of the show.

Title: Re: What happened at Coachella on Friday?
Post by: Stephen Kirby on May 03, 2017, 05:26:20 PM
I love this SRV clip.
https://www.youtube.com/watch?v=wn1pk8Lpelc
How to deal with failure!
Or you can see how the real pro's handle it.
https://youtu.be/MXwBsxTBxGM (https://youtu.be/MXwBsxTBxGM)

I also saw John Scofield change a string on stage and instead of winding a bunch around the peg as BB does (something from his chitlin circuit days when he needed the slack to re-tie a broken ball end and save the string) Sco took his pick to the string hanging off the peg like scissors to a curling ribbon and it curled right up out of the way.  No cutters needed.