ProSoundWeb Community

Please login or register.

Login with username, password and session length
Advanced search  

Pages: 1 2 [All]   Go Down

Author Topic: Network Robustness - Dante  (Read 1427 times)

David Sturzenbecher

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1460
    • Sturz Audio
Network Robustness - Dante
« on: October 04, 2018, 05:18:56 pm »

I just fielded a call from large stadium that their sound system was acting squirrely.  Amplifiers were coming and going on the monitoring page and the audio "wasn't right".

Turns out a dante card in one of the amplifiers had gone bad, and changed itself to switched mode instead of redundant,  connecting the primary and backup networks together.  As soon as I disabled the port connecting the amp to the secondary network everything straightened out.  The card was unable to be seen in Dante Controller, but the amp showed up fine in the amplifier manufactures monitoring software.   The odd thing (or not) is that the amp communication and dante primary are on the same physical port.

Is there any way to prevent this from happening in the future?

Just spit balling here but what about independently Vlanning the primary and secondary networks?   Even though the networks are physically separated (when operating correctly), this should keep them from talking if an amp goes rouge no?

Logged
Audio Systems Design Engineer
Daktronics, Inc.
CTS-D, CTS-I
AES Full Member

Erik Jerde

  • Classic LAB
  • Hero Member
  • *
  • Offline Offline
  • Posts: 586
Re: Network Robustness - Dante
« Reply #1 on: October 04, 2018, 06:22:05 pm »

I just fielded a call from large stadium that their sound system was acting squirrely.  Amplifiers were coming and going on the monitoring page and the audio "wasn't right".

Turns out a dante card in one of the amplifiers had gone bad, and changed itself to switched mode instead of redundant,  connecting the primary and backup networks together.  As soon as I disabled the port connecting the amp to the secondary network everything straightened out.  The card was unable to be seen in Dante Controller, but the amp showed up fine in the amplifier manufactures monitoring software.   The odd thing (or not) is that the amp communication and dante primary are on the same physical port.

Is there any way to prevent this from happening in the future?

Just spit balling here but what about independently Vlanning the primary and secondary networks?   Even though the networks are physically separated (when operating correctly), this should keep them from talking if an amp goes rouge no?

Putting the ports on separate vlans won't protect against the scenario you describe because the amp card going to switch mode will connect the two vlans together.

You could look into port security and configure switch ports to only accept traffic from one mac address.  Then if a switch gets attached to one of those ports (as effectively happened here) it should reject all the traffic coming through that switch that doesn't have the correct mac address.

I'm assuming the problem here was the primary and secondary traffic being on the same network not a more fundamental network issue. 
Logged

Andrew Hollis

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 82
Re: Network Robustness - Dante
« Reply #2 on: October 04, 2018, 07:30:23 pm »

The odd thing (or not) is that the amp communication and dante primary are on the same physical port.

To the outside it is one port, but inside it is switched; Dante and proprietary control would be two different things.

Scott Helmke

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1210
Re: Network Robustness - Dante
« Reply #3 on: October 05, 2018, 01:22:37 pm »

This is a constant worry as firmware updates might reset port behavior, etc. My usual first Dante troubleshooting item is "unplug everything marked secondary".
Logged

Thomas Dameron

  • SR Forums
  • Jr. Member
  • *
  • Offline Offline
  • Posts: 72
Re: Network Robustness - Dante
« Reply #4 on: October 08, 2018, 11:08:27 pm »

This is a constant worry as firmware updates might reset port behavior, etc. My usual first Dante troubleshooting item is "unplug everything marked secondary".

The simple solution is to eliminate the secondary network entirely.  I know that sounds like a JV option, but a few years into constantly working with Dante I've had many problems with the secondary network taking out the primary network for a variety of reasons.  I've had exactly zero examples  of the primary failing and falling over to the secondary. 

I would be curious to hear other peoples' experience on this. 

thomas d.
Logged

Jelmer de Jong

  • Full Member
  • ***
  • Offline Offline
  • Posts: 164
Re: Network Robustness - Dante
« Reply #5 on: October 09, 2018, 03:41:05 pm »

You need spanning tree, a very intelligent version of it.
Logged

Bryan Hargrave

  • Newbie
  • *
  • Offline Offline
  • Posts: 31
  • I'm one jammin' dude.
    • Soundworks of VA, Inc
Re: Network Robustness - Dante
« Reply #6 on: October 09, 2018, 04:51:49 pm »

You need spanning tree, a very intelligent version of it.

According to Audinate you should not use STP as it takes upwards of a minute to resolve a broken connection. Even RSTP takes around 10 seconds. Audinate's redundancy is seamless. Cisco default timing settings for STP work out this way too. So if ditching the secondary as the back up is the answer... go with analog or AES. It will be about a 1 second drop. As always, correct me if I'm wrong... nicely.

I haven't had an instance of failure that was network related so I can't comment on the secondary network mucking things up. Fingers crossed I will be able to say that for a long time to come. Now that I say it, I give myself a week.
« Last Edit: October 09, 2018, 05:05:21 pm by Bryan Hargrave »
Logged
Like us on facebook at Soundworks of VA, Inc.

Aram Piligian

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 70
Re: Network Robustness - Dante
« Reply #7 on: October 09, 2018, 06:04:25 pm »

Using Link Aggregation between crucial segments (FOH to BOH, etc) is more the route to go here.  The failover is much faster than even rapid spanning tree.

I've been leaning towards Thomas' mentality lately too--there's very very few problems I've had with a Dante network that are fixed by having the Secondary network.  And with the prevalence of Ultimo chipped devices which only work on the Primary anyway, I feel like Secondary is only a half safety net. 
Logged

David Sturzenbecher

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1460
    • Sturz Audio
Re: Network Robustness - Dante
« Reply #8 on: October 09, 2018, 06:11:05 pm »

So I am now confused. We replaced the bad amp, and the replacement too will not show up in Dante controller, yet shows up fine in the amp manufactures software.  Tried different cables, different switch ports... no change.   The previous amp was working fine for weeks... and numerous identical amps are currently working fine.   


Sent from my iPhone using Tapatalk Pro
Logged
Audio Systems Design Engineer
Daktronics, Inc.
CTS-D, CTS-I
AES Full Member

Andrew Hollis

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 82
Re: Network Robustness - Dante
« Reply #9 on: October 09, 2018, 07:47:06 pm »

Using Link Aggregation between crucial segments (FOH to BOH, etc) is more the route to go here. The failover is much faster than even rapid spanning tree.

How would Link Aggregation be used to prevent a primary-secondary interconnection fault described by the OP?

Scott Holtzman

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 5028
  • Ghost AV - Avon Lake, OH
    • Ghost Audio Visual Systems, LLC
Re: Network Robustness - Dante
« Reply #10 on: October 09, 2018, 07:52:37 pm »

How would Link Aggregation be used to prevent a primary-secondary interconnection fault described by the OP?
Link aggregation works at the interface level. Bandwidth drops in half but connectivity from a layer 2 perspective is unaffected.

Sent from my VS996 using Tapatalk

Logged
Scott AKA "Skyking" Holtzman
River Delta Audio is now:

Ghost Audio Visual Solutions, LLC
Cleveland OH
www.ghostav.rocks

Erik Jerde

  • Classic LAB
  • Hero Member
  • *
  • Offline Offline
  • Posts: 586
Re: Network Robustness - Dante
« Reply #11 on: October 10, 2018, 01:17:20 am »

Link aggregation works at the interface level. Bandwidth drops in half but connectivity from a layer 2 perspective is unaffected.

Sent from my VS996 using Tapatalk

Still doesn’t fix the OPs problem. 

LAG and STP aren’t the solution to what is essentially a hardware failure here.  The problem is devices have what is effectively a 5 port software configurable  switch in them.  If that switch changes configurations unexpectedly and links two networks that shouldn’t be you’ve got problems.  Because it’s a single link between the two networks STP doesn’t come into play.  Because it doesn’t involve a link failure LAG doesn’t come into play.

The only real solution for this would be a change in hardware design where you have a dedicated daisy-chain port on the primary internal switch and a completely separate port directly attached to the Dante secondary PHY.  It would increase cost enough that barring a lot of high profile failures we’re highly unlikely to ever see it happen.  More likely would be Audinate finding a way to gracefully handle a primary/secondary interconnect in software.
Logged

Aram Piligian

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 70
Re: Network Robustness - Dante
« Reply #12 on: October 10, 2018, 07:05:21 am »

Still doesn’t fix the OPs problem. 

LAG and STP aren’t the solution to what is essentially a hardware failure here. 

Right, I was speaking more broadly to Thomas' point.

Quote
More likely would be Audinate finding a way to gracefully handle a primary/secondary interconnect in software.

That would be cool.  When a device boots, if it sees that the Dante chips on one port are in one subnet and the chips on the other port are on another subnet (but with the same names), disable the secondary until it's resolved or the mode is changed.  It's using mDNS anyway to see what other devices are on the network.
« Last Edit: October 10, 2018, 10:39:51 am by Aram Piligian »
Logged

Ron Bolte

  • Newbie
  • *
  • Offline Offline
  • Posts: 37
  • Mmmmm.. ok...
Re: Network Robustness - Dante
« Reply #13 on: October 10, 2018, 10:14:19 am »

So I am now confused. We replaced the bad amp, and the replacement too will not show up in Dante controller, yet shows up fine in the amp manufactures software.  Tried different cables, different switch ports... no change.   The previous amp was working fine for weeks... and numerous identical amps are currently working fine.   


Sent from my iPhone using Tapatalk Pro

How are the amps getting addresses?  (static, DNS, Self Assigned?) Check that the new amp is properly configured.  If its put itself in a different subnet address than the computer you are running controller, it will show up in some places (amp software), but not in Dante Control
Logged

Phillip Ivan Pietruschka

  • Newbie
  • *
  • Offline Offline
  • Posts: 9
Re: Network Robustness - Dante
« Reply #14 on: October 10, 2018, 05:03:43 pm »

How are the amps getting addresses?  (static, DNS, Self Assigned?) Check that the new amp is properly configured.  If its put itself in a different subnet address than the computer you are running controller, it will show up in some places (amp software), but not in Dante Control

Further to this:

Some devices will have separate IPs (and subnets) for the control and Dante interfaces, and frustratingly, often they are set in different places and with different software.
I have also found that with some of these devices they ‘might’ work albeit with unpredictable or reduced function if both these IPs are set the same by accident or design.

Typically in DC devices that are in another subnet will show up in red on the devices tab. Double clicking will present no control options in the window that opens but will show the devices IP.
Logged

David Sturzenbecher

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1460
    • Sturz Audio
Network Robustness - Dante
« Reply #15 on: October 10, 2018, 10:33:17 pm »

How are the amps getting addresses?  (static, DNS, Self Assigned?) Check that the new amp is properly configured.  If its put itself in a different subnet address than the computer you are running controller, it will show up in some places (amp software), but not in Dante Control

The amp control is static, and the Dante is DHCP.  This is really only because I am in the habit of leaving Dante stuff on DHCP, as the early chipsets didn't support static.   I have two NICs on the computer to simultaneous talk to the two subnets.  Technically, i have four NICs on the computer, an additional NIC for the Dante Secondary, and another for Internet/Remote Access. 

Of course I understand networking basics 101 that the IPs need to be set properly in order to communicate.  The amp control came in at a link local address and that was changed to static to talk to the monitoring software.  The dante config on these amps is done entirely in Dante controller, and the amp wont show up at all there.... So hard to configure it if it doesn't show up.  Ill again state, all the other amplifiers show up fine...so its hard to pin it on the network.      I did have to turn off the port connected to the secondary NIC with this amp as well, as everything went squirrely otherwise. I confirmed through the MAC address tables that with this port turned on, i can see all the MAC addresses associated with the secondary network on the primary network switches.
« Last Edit: October 11, 2018, 06:35:49 am by David Sturzenbecher »
Logged
Audio Systems Design Engineer
Daktronics, Inc.
CTS-D, CTS-I
AES Full Member

Ron Bolte

  • Newbie
  • *
  • Offline Offline
  • Posts: 37
  • Mmmmm.. ok...
Re: Network Robustness - Dante
« Reply #16 on: October 10, 2018, 11:34:41 pm »

...  The dante config on these amps is done entirely in Dante controller, and the amp wont show up at all there.... So hard to configure it if it doesn't show up.  Ill again state, all the other amplifiers show up fine...so its hard to pin it on the network.      I did have to turn off the port connected to the secondary NIC with this amp as well, as everything went squirrely otherwise....
The fact that the secondary connection made everything squirrely says that the config on the Dante side of the amp is set to "daisy chain" from the factory.  Good chance the address is set wrong also.  I would isolate the amp from the network, connect a laptop to it directly, set to self address, that would be your best chance of getting the amp dante chip and the laptop on the same sub net, run controller and you should see the amp and be able to fix the address/reduntant-daisy issue.  Its possible the amp uses a "default" dante address.  either way, the key is getting a computer and the amp on the same subnet to get control of it and fix the issue.
Logged

Bob Leonard

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 6805
  • Boston, MA USA
Re: Network Robustness - Dante
« Reply #17 on: October 11, 2018, 06:32:43 am »

The amp control is static, and the Dante is DHCP.  This is really only because I am in the habit of leaving Dante stuff on DHCP, as the early chipsets didn't support static.   I have two NICs on the computer to simultaneous talk to the two subnets.  Technically, i have four NICs on the computer, an additional NIC for the Dante Primary, and another for Internet/Remote Access. 

Of course I understand networking basics 101 that the IPs need to be set properly in order to communicate.  The amp control came in at a link local address and that was changed to static to talk to the monitoring software.  The dante config on these amps is done entirely in Dante controller, and the amp wont show up at all there.... So hard to configure it if it doesn't show up.  Ill again state, all the other amplifiers show up fine...so its hard to pin it on the network.      I did have to turn off the port connected to the secondary NIC with this amp as well, as everything went squirrely otherwise. I confirmed through the MAC address tables that with this port turned on, i can see all the MAC addresses associated with the secondary network on the primary network switches. 

It would be preferable to use static addresses. DHCP is seldom your friend in networks of this type/usage.
Logged
BOSTON STRONG........
Proud Vietnam Veteran

I did a gig for Otis Elevator once. Like every job, it had it's ups and downs.

David Sturzenbecher

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1460
    • Sturz Audio
Re: Network Robustness - Dante
« Reply #18 on: October 11, 2018, 06:49:54 am »

It would be preferable to use static addresses. DHCP is seldom your friend in networks of this type/usage.

Yes I know. For years (back when Dante was a black art) Audinate’s official recommendation was to just leave IPs set to automatic addressing and have them fall back to link local addressing, unless there was a specific reason otherwise. I never had a specific reason, so it just became habit not to do it for Dante. I have always used static IPs for the control equipment, and just used two NICs in the computer. The setup has worked well in numerous installs, but it only takes one guy to plug in a home router to really mess things up.... that happened last week.


Sent from my iPhone using Tapatalk Pro
Logged
Audio Systems Design Engineer
Daktronics, Inc.
CTS-D, CTS-I
AES Full Member

David Sturzenbecher

  • Hero Member
  • *****
  • Offline Offline
  • Posts: 1460
    • Sturz Audio
Re: Network Robustness - Dante
« Reply #19 on: October 11, 2018, 06:51:46 am »

The fact that the secondary connection made everything squirrely says that the config on the Dante side of the amp is set to "daisy chain" from the factory.  Good chance the address is set wrong also.  I would isolate the amp from the network, connect a laptop to it directly, set to self address, that would be your best chance of getting the amp dante chip and the laptop on the same sub net, run controller and you should see the amp and be able to fix the address/reduntant-daisy issue.  Its possible the amp uses a "default" dante address.  either way, the key is getting a computer and the amp on the same subnet to get control of it and fix the issue.
Thanks


Sent from my iPhone using Tapatalk Pro
Logged
Audio Systems Design Engineer
Daktronics, Inc.
CTS-D, CTS-I
AES Full Member
Pages: 1 2 [All]   Go Up
 


Page created in 0.062 seconds with 21 queries.